Method and device for multi-view video coding based on implicit neural representation

Abstract

Disclosed are a method and device for multi-view video coding based on implicit neural representation according to the present embodiment. According to the present embodiment, an image decoding device decodes parameters of a multi-view INRVC model and an INRVC parameter set (IPS) from a bitstream. Here, the IPS includes a quantization parameter and information about the structure of the multi-view INRVC model. The image decoding device uses the parameters of the multi-view INRVC model and the quantization parameter to reconstruct the multi-view INRVC model so that the multi-view INRVC model accords with the information about the structure of the multi-view INRVC model. The image decoding device acquires the current viewpoint, inputs the current viewpoint into the multi-view INRVC model, and generates a video from the current viewpoint on the basis of a 4D grid and camera parameters.

Description

Method and device for multi-view video coding based on implicit neural network representation

[0001] The present disclosure relates to a multi-view video coding method and device based on implicit neural network representation.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Recently, research on implicit neural network representation models, which represent various data, including images, using neural network structures, has been active. Conventional video representation methods use explicit representations, which represent RGB pixel values ​​at each pixel location. To replace this explicit representation, implicit neural representations are being introduced, which represent functions that generate (r,g,b) values ​​from (x,y) coordinates (pixel locations) in an image using neural networks.

[0004] Among implicit neural representation methods, NeRF (Neural Radiance Fields) is a representative model used for synthesizing 3-dimensional (3D) videos. Using a learning process to reconstruct all pixels within an image into world coordinates, NeRF outputs rendered results from an arbitrary viewpoint. At this time, NeRF uses indices in the world coordinate system and camera parameters required for projection into the world coordinate system as input. Furthermore, NVP (Neural Video Representations) technology generates videos spanning the t-axis, y-axis, and x-axis based on learnable keyframes and sparse positional features. NVP generates an image at time t based on keyframe vectors, which are 2D grid-based embeddings, and positional features, which are 3D grid-based embeddings. Therefore, to enhance the efficiency of multi-view video encoding representations and improve multi-view video quality, a method for efficiently configuring a multi-view video coding device based on implicit neural network representations needs to be considered.

[0005] The present disclosure aims to provide a multi-view video coding method and device that efficiently compresses multi-view video and generates video of an arbitrary viewpoint requested by a decoder side, based on a multi-view implicit neural representation for video coding (Multi-view INRVC) model.

[0006] In addition, the present disclosure aims to provide a multi-view video coding method and device that learns and utilizes a 4D grid-based embedding that adds a multi-view dimension to an existing (x, y, t) 3D (Dimensional) grid-based embedding based on a multi-view INRVC model.

[0007] According to an embodiment of the present disclosure, a method for restoring a video of a current viewpoint, performed by a video decoding device, is provided, comprising: decoding parameters of a multi-view INRVC (Implicit Neural Representation for Video Coding) model and an IPS (INRVC Parameter Set) from a bitstream, wherein the IPS includes quantization parameters and information about a structure of the multi-view INRVC model; restoring the multi-view INRVC model using the parameters of the multi-view INRVC model and the quantization parameters so as to conform to the information about the structure of the multi-view INRVC model, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters; obtaining the current viewpoint; and inputting the current viewpoint into the multi-view INRVC model to generate a video of the current viewpoint based on the 4D grid and the camera parameters.

[0008] According to another embodiment of the present disclosure, a method for encoding videos of multiple views, performed by a video encoding device, is provided, comprising: acquiring videos of the multiple views; configuring a multi-view INRVC (Implicit Neural Representation for Video Coding) model, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters corresponding to videos of the multiple views; training the multi-view INRVC model based on the multi-views, the 4D grid, and the camera parameters; determining a quantization parameter and information about the structure of the multi-view INRVC model based on a result of the training; encoding parameters of the multi-view INRVC model using the quantization parameter; and encoding an IPS (INRVC Parameter Set) including the quantization parameter and information about the structure of the multi-view INRVC model.

[0009] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device comprises: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: acquiring videos of multiple views; configuring a multi-view INRVC (Implicit Neural Representation for Video Coding) model, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters corresponding to videos of the multiple views; performing training of the multi-view INRVC model based on the multi-views, the 4D grid, and the camera parameters; determining a quantization parameter and information about a structure of the multi-view INRVC model based on a result of the training; encoding parameters of the multi-view INRVC model using the quantization parameter; And a method is provided, characterized by including a step of encoding an IPS (INRVC Parameter Set) including information about the quantization parameters and the structure of the multi-view INRVC model.

[0010] As described above, according to the present embodiment, by providing a multi-view video coding method and device that efficiently compresses multi-view video based on a multi-view INRVC model and generates video of an arbitrary viewpoint requested by a decoder side, it is possible to improve multi-view video encoding efficiency and multi-view video quality.

[0011] In addition, according to the present embodiment, by providing a multi-view video coding method and device that learns and utilizes a 4D grid-based embedding that adds a multi-view dimension to a 3D grid-based embedding of (x, y, t) based on a multi-view INRVC model, it is possible to reduce the burden on the network based on bit rate reduction in various contents such as UHD (Ultra High Definition) video, game broadcasting, 360-degree video streaming, VR / AR (Virtual Reality / Augmented Reality) video, online lectures, etc., and to reduce energy consumption for a video playback device.

[0012] FIG. 1 is a block diagram illustrating a multi-view INRVC (Implicit Neural Representation for Video Coding)-based multi-view video encoding device according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram illustrating a multi-view INRVC-based multi-view image decoding device according to one embodiment of the present disclosure.

[0014] FIG. 3 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.

[0015] Figure 4 is an example diagram showing a SISR (Single Image Super Resolution) network.

[0016] Figure 5 is an example diagram showing a residual block used in SISR.

[0017] Figure 6 is an example diagram showing coordinate systems related to extraction of arbitrary points in NeRF (Neural Radiance Fields).

[0018] Figure 7 is an example diagram showing the NeRV (Neural Representations for Videos) model.

[0019] FIG. 8A and FIG. 8B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.

[0020] FIG. 9 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0021] FIG. 10 is an exemplary diagram illustrating a decryption pipeline of a NeRV model according to one embodiment of the present disclosure.

[0022] FIG. 11 is an exemplary diagram illustrating an NVP (Neural Video Representation) model according to one embodiment of the present disclosure.

[0023] FIG. 12 is an exemplary diagram illustrating a multi-view INRVC (Implicit Neural Representation for Video Coding) model according to one embodiment of the present disclosure.

[0024] FIGS. 13A to 13C are exemplary diagrams showing the configuration of a multi-view grid according to one embodiment of the present disclosure.

[0025] FIG. 14 is an exemplary diagram illustrating an INRVC model that learns depth information according to one embodiment of the present disclosure.

[0026] FIG. 15 is an exemplary diagram showing warping of a reference image according to one embodiment of the present disclosure.

[0027] FIG. 16 is a flowchart illustrating a method for a video encoding device to encode videos of multiple views according to an embodiment of the present disclosure.

[0028] FIG. 17 is a flowchart illustrating a method for a video decoding device to restore a video at a current point in time according to an embodiment of the present disclosure.

[0029] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0030] The present embodiment provides a multi-view video coding method and device that efficiently compresses multi-view video and generates video of an arbitrary viewpoint requested by a decoder side, based on a multi-view implicit neural representation for video coding (Multi-view INRVC) model.

[0031] FIG. 1 is a block diagram illustrating a multi-view INRVC-based multi-view video encoding device according to one embodiment of the present disclosure.

[0032] In the example of FIG. 1, a multi-view INRVC-based multi-view video encoding device (hereinafter, referred to as the 'video encoding device') generates (i.e., trains) a multi-view INRVC model to learn to generate a video from multiple views, compresses the trained multi-view INRVC model to generate a bitstream, and transmits the generated bitstream. The video encoding device includes a multi-view INRVC encoding unit (110) including a multi-view INRVC model and a neural network compressor (120). Here, the components included in the video encoding device according to the present disclosure are not necessarily limited thereto. The video encoding device may additionally include a training unit (not shown) for training the multi-view INRVC model, or may be implemented in a form linked with an external training unit.

[0033] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0034] The video encoding device can store the bitstream of encoded video data on a non-transitory recording medium or transmit it to a multi-view INRVC-based video decoding device using a communication network.

[0035] FIG. 2 is a block diagram illustrating a multi-view INRVC-based multi-view image decoding device according to one embodiment of the present disclosure.

[0036] In the example of Fig. 2, a multi-view INRVC-based multi-view video decoding device (hereinafter, referred to as the 'video decoding device') decompresses a bitstream to restore a multi-view INRVC model, and generates a video from multiple views using the restored multi-view INRVC model. The video decoding device includes a neural network decompressor (210) and a multi-view INRVC decoding unit (220) including the restored multi-view INRVC model.

[0037] Similar to the video encoding device illustrated in Fig. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor configured to execute the software functions corresponding to each component.

[0038] Below, before describing the components illustrated in FIGS. 1 and 2, the deep learning-related element technologies used in the present disclosure are described.

[0039] I. CNN(Convolutional Neural Network)

[0040] CNN, a neural network comprised of multiple convolutional layers and pooling layers, is a deep learning technique known to be ideal for image processing. Convolutional layers extract feature maps (also known as "features" or "characteristics") using multiple kernels or filters. The kernel coefficients that make up the filters are parameters determined during the learning process.

[0041] Among the convolutional layers of CNN, the front layer closer to the input extracts feature maps that respond to simple, low-level image features such as lines, points, or surfaces, while the back layer closer to the output extracts feature maps that respond to higher-level features such as textures and object parts.

[0042] FIG. 3 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.

[0043] A convolutional layer generates a feature map from an input image using a convolution operation. The example in Fig. 3 illustrates a kernel (or filter) with a kernel size of 3×3. The kernel size is also referred to as the kernel size or filter size. The kernel has kernel parameters, also called weights. The kernel illustrated in Fig. 3 has a total of nine kernel parameters. The kernel parameters are initially set to arbitrary values, and their values ​​can be updated based on learning.

[0044] The convolution layer performs convolution operations using blocks of the same size as the kernel size in the input image. At this time, the blocks of the same size as the kernel size in the input image are referred to as windows.

[0045] When filtering an input image in raster scan order, the window movement size is called the stride. In the example of Fig. 3, the stride is 1. If the stride is set to 2, the convolution operation is performed by spacing the window by 2 samples, and as a result, the width and height of the feature map become half the width and height of the input image.

[0046] As mentioned above, a single convolutional layer can include multiple filters. The number of filters or kernels is called a channel. In other words, the number of channels is equal to the number of filters. Furthermore, the number of filters determines the dimensionality of the feature map.

[0047] Padding refers to a method of expanding input data by filling the area around it with a specific value before performing a convolution operation. Padding is primarily used to adjust the spatial size of the output data. The padding value can be determined by hyperparameters, but zero-padding is commonly used. Without padding, the spatial size of the output data decreases with each convolutional layer, potentially causing boundary information to disappear. Therefore, padding is used to prevent this problem. Specifically, padding can be used to match the spatial sizes of the output data from the convolutional layer to the input data.

[0048] The deconvolution layer performs the opposite operation to the convolution layer. It generates the desired data image from the input feature map as output.

[0049] The pooling layer performs pooling, a process of subsampling the feature map generated by the convolutional layer. The pooling layer uses a 2×2 window to select samples so that the output is half the width and height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by condensing a 2×2 region into a single sample.

[0050] The opposite concept of a pooling layer is defined as an unpooling layer. Unpooling layers, in contrast to pooling layers, function to expand the dimensionality and are primarily used after deconvolution layers.

[0051] The convolutional encoder-decoder architecture is a network structure composed of pairs of convolutional layers and deconvolutional layers. The convolutional encoder consists of a convolutional layer and a pooling layer, and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also referred to as a latent vector. The convolutional decoder consists of a deconvolutional layer and an unpooling layer, and generates an output image from the feature map or latent vector.

[0052] The inputs and outputs of a convolutional encoder-decoder can be configured in various ways depending on the application and network purpose. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.

[0053] Figure 4 is an example diagram showing an SISR network.

[0054] One example of CNN application is Single Image Super Resolution (SISR). The SISR network generates a high-resolution image from a low-resolution input image. As illustrated in Figure 4, the SISR network may include multiple convolutional layers. Each convolutional layer includes an activation function, such as the Rectified Linear Unit (ReLU). The parameters of the SISR network can be trained so that the resulting SR (Super Resolution) image approximates the Ground Truth (GT).

[0055] SR methods using CNN can improve SR performance by increasing the depth (e.g., increasing the number of convolutional layers). To overcome the overfitting problem that may occur in learning due to the increase in depth, a residual block that can perform skip connection and residual learning can be used in the SISR network. The residual block, as illustrated in Fig. 5, is a block that stores input features x l In addition to the path that applies the convolution operation, it includes a skip path. In addition, the residual block outputs x l+1 When generating, a path or skip path for applying convolution operations can also be selected based on learning efficiency. In the example of Fig. 5, the residual block includes a BN (Batch Normalization) layer.

[0056] For example, Enhanced Deep Residual Networks (EDSR) improves network performance by continuously connecting residual blocks to increase their depth. Another example is Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR), a CNN model based on the Visual Geometry Group (VGG) network. It uses residual learning, a method that adds residual frames to the final output. VDSR adds residual signals to the very end of the network, thereby adding them to the input signal.

[0057] II. Implicit neural representation model

[0058] Recently, research on implicit neural network representation models, which represent various data, including images, using neural network structures, has been active. Conventional video representation methods use an explicit representation method, which assigns RGB pixel values ​​to each pixel location. To replace this explicit representation method, implicit neural network representation models are being introduced, which represent functions that generate (r,g,b) values ​​from (x,y) coordinates (pixel locations) in an image using neural networks. Compared to explicit representation methods, implicit neural network representation models can be used regardless of image resolution.

[0059] II-1. Neural Radiance Fields (NeRF)

[0060] As an example of 3D video coding that utilizes an implicit neural network representation model, there is NeRF (Neural Radiance Fields) technology.

[0061] By learning from images captured from multiple viewpoints of a multi-view video, NeRF synthesizes an image of an object viewed from any arbitrary position and orientation. After receiving a 5-dimensional coordinate consisting of a 3D position (x, y, z) that samples ray information in the direction the camera is looking and a viewing direction (θ, φ), NeRF outputs the RGB values ​​(r, g, b) and volume density σ corresponding to the input coordinates. Here, the 3D position (x, y, z) is the position in the world coordinate system, and θ and φ that constitute the viewing direction can represent the azimuth and altitude, respectively.

[0062] Compared to existing implicit neural network representations, NeRF offers the following improvements to better represent high-resolution, complex scenes. NeRF embeds positional information using positional encoding, and then uses the embedded index as input, as shown in Equation 1.

[0063]

[0064] As shown in the example in Equation 1, each spatial coordinate can be mapped to a 2L-dimensional vector γ(p). Instead of directly inputting the 3D input coordinates, each spatial coordinate can be mapped to a 2L-dimensional vector γ(p) using the sin and cos functions. By utilizing the embedded spatial index in training the NeRF model, the NeRF model can better predict data containing high-frequency variations.

[0065] NeRF is an implicit neural network representation model that uses a multi-layer perceptron (MLP) network containing multiple fully-connected layers (FC). NeRF represents an arbitrary scene by outputting RGB values ​​(r, g, b) and volume density σ corresponding to high-dimensional input coordinates. Then, the color C(r) of an arbitrary ray r passing through the scene can be rendered according to a volume rendering technique based on the RGB values ​​(r, g, b) and volume density σ. Since C(r) is differentiable, it is used for training NeRF together with GT (Ground Truth). At this time, an image that captures world coordinates (x, y, z) along the view direction (θ, φ) is used as GT for each 5-dimensional coordinate.

[0066] Meanwhile, to improve rendering efficiency, NeRF includes two networks, a coarse network and a fine network, instead of using a single MLP. NeRF uses stratified sampling to achieve N c After sampling the positions of the dogs, N c The output of the coarse network is calculated at each location. To generate more samples to apply to the fine network from the output of the coarse network, N c Composite color C based on the coarse network output of the dog c (r) is expressed as in mathematical formula 2.

[0067]

[0068] In mathematical expression 2, c i is N c The color of the coarse network calculated from the dog's location. w iis a value that depends on the distance between the volume and the sampled locations as a corresponding weight. Using inverse transform sampling, N with high weights from the aforementioned composite color f After sampling the location of the dog, NeRF c +N f Calculate the output of the fine network at the location of the dog, and color C based on the output of the calculated fine network. f (r) can be rendered. At this time, NeRF can be trained using the MSE (Mean Square Error) loss function between the color and GT according to the output of the coarse network and the fine network.

[0069] As mentioned above, NeRF outputs a rendered result at an arbitrary point in time by using a learning process to reconstruct all pixels in the image into world coordinates. Therefore, a process is needed to transform all points projected to a point in the image coordinate system (x, y) into world coordinates. Therefore, the input of NeRF includes the camera parameters required for projection into the world coordinate system along with the index in the world coordinate system.

[0070] The process of extracting camera parameters is to convert pixels in an image into a world coordinate system [U, V, W] through a coordinate system [X, Y, Z] with a focal length Z = -1, as shown in the example of Fig. 6. At this time, prior information about the camera is essential, and the prior information, the camera parameters, includes internal parameters and external parameters. The internal camera parameters include the focal length, the principal point on the image plane, and the skew coefficient. According to the internal camera parameters, the object can be projected into the camera coordinate system [X, Y, Z]. In order to convert the information projected into the camera coordinate system into the world coordinate system [U, V, W], translation information about where the camera is located in the world coordinate system and rotation information about where it is looking are required. The external camera parameters include the aforementioned translation information and rotation information. When capturing an image, a matrix [R|t], which is a combination of a rotation matrix R and a translation vector t, is used to calculate the projection from the world coordinate system to the image coordinate system. Therefore, the camera extrinsic parameters can be extracted by calculating the inverse matrix of the combination matrix [R|t].

[0071] Meanwhile, the 5-dimensional coordinates, which are the inputs of NeRF, can be derived from the extracted camera parameters. As described above, the camera parameters can be derived based on the captured 2D images and the positions in the world coordinate system corresponding to the 2D images. When capturing images for GT use, the internal camera parameters can be fixed. When the internal camera parameters are used while being fixed, the 5-dimensional coordinates can be derived from the external camera parameters.

[0072] As another example, in addition to using the camera parameters used when shooting the video, a method such as colmap can also be used to extract feature points from re-view images and reconstruct information about the position and orientation of the camera within the world coordinate system.

[0073] II-1. Implicit Neural Representation for Video Coding (INRVC)

[0074] An example of 2D video coding that utilizes an implicit neural representation model is Implicit Neural Representation for Video Coding (INRVC). INRVC technology generates (r,g,b) values ​​for the (x,y) coordinates, which are pixel locations in an image.

[0075] As an example of INRVC technology, there is the NeRV (Neural Representation for Video) model. As mentioned above, the explicit representation method expresses pixel values ​​on the (x, y, t) grid using the location (x, y) of the pixel in the image and the time index t of the frame. The implicit neural network representation model outputs RGB pixel values ​​from the input indicating the location (x, y, t). However, since training the neural network at all locations of (x, y, t) can significantly increase the computational complexity, as shown in the example of Fig. 7, the NeRV model outputs the RGB image of the entire frame at the time index t using a neural network structure that only takes the time index t as input.

[0076] FIG. 8A and FIG. 8B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.

[0077] To easily provide image output for a time index t input, it may be more effective to use a convolutional layer than to use an MLP network of an existing implicit representation model. The NeRV model can generate output using a structure in which NeRV blocks composed of multiple convolutional layers are stacked. As shown in the example of Fig. 8a, an MLP-based output and a NeRV block-based output can be compared. The NeRV block includes convolutional layers that perform convolution, a pixel shuffle layer, and an activation layer, as shown in the example of Fig. 8b. In the example of Fig. 8b, C represents the number of channels, W and H represent the width and height of the input of the NeRV block, and S represents an upscaling factor.

[0078] Additionally, the NeRV model embeds the time index t into a high-dimensional space as in Equation 3 and then uses the embedded index as input.

[0079]

[0080] As illustrated in Equation 3, the temporal index t can be mapped to a 2L-dimensional vector γ(t). By using the embedded temporal index t to train the NeRV model, the NeRV model can better predict video data containing high-frequency variations.

[0081] As the loss function of the NeRV model, a combination of L1 loss and SSIM (Structural Similarity Index) loss can be used. During training of the NeRV model, the aforementioned loss can be calculated at all pixel locations of the image estimated by the NeRV model based on the input and the ground truth (GT) image (i.e., the original image). The L1 loss is calculated using the absolute value of the difference between pixels in the estimated image and the GT image. The SSIM loss is calculated based on the mean, standard deviation, and correlation of pixels in the estimated image and the GT image. Thereafter, the weights (or parameters) that constitute the NeRV model during the training process can be updated based on the calculated loss.

[0082] FIG. 9 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0083] Since the original video can be approximately represented using the NeRV model, an end-to-end compression technology using a neural network can be implemented by compressing and transmitting the NeRV model. An encoding pipeline for encoding the NeRV model may include all or part of a video overfitter (910), a model pruner (920), a model quantizer (930), and a weight encoder (940), as shown in the example of Fig. 9.

[0084] The video overfitter (910) expresses the input video frame as a NeRV model. At this time, the video encoding device trains the NeRV model using the aforementioned loss function, thereby allowing the NeRV model to express the video frame. The model pruner (920) simplifies the NeRV model structure, which has an MLP or a mixed form of MLP and convolutional layers, using pruning. For example, pruning sets weights smaller than a preset threshold to zero. The model quantization unit (930) quantizes the weights to which pruning has been applied (e.g., weights greater than the threshold). The weight encoding unit (940) applies entropy encoding to the quantized weights to generate a bitstream of the weights.

[0085] Meanwhile, as a reverse order of the encoding pipeline illustrated in FIG. 9, a decoding pipeline for decoding a NeRV model can be illustrated as in FIG. 10. The decoding pipeline can include all or part of a weight decoder (1010), a model dequantizer (1020), a model reconstructor (1030), and a video generator (1040), as illustrated in FIG. 10.

[0086] The weight decoding unit (1010) applies entropy encoding to the bitstream of the weights of the NeRV model to generate quantized weights. The model dequantization unit (1020) dequantizes the quantized weights to generate pruned weights. The model restoration unit (1030) restores the NeRV model based on the pruned weights. The video generation unit (1040) uses the restored NeRV model to generate restored video frames according to the time index.

[0087] As mentioned above, the NeRV model uses the time index t of the frame as input. Therefore, when a video having pixels of size t×h×w is input, the NeRV model samples the input video only t times, not t×h×w times, so a large gain in both encoding speed and decoding speed can be expected.

[0088] FIG. 11 is an exemplary diagram illustrating an NVP (Neural Video Representation) model according to one embodiment of the present disclosure.

[0089] As another example of INRVC technology, there exists a Neural Video Representation (NVP) model, as illustrated in Fig. 11. The NVP model generates a video spanning the t-axis, y-axis, and x-axis based on learnable keyframes and sparse positional features. The NVP model includes coordinate-to-latent mapping, which generates latent features (i.e., latent embeddings) from coordinates, and latent-to-RGB mapping, which generates RGB from latent features.

[0090] The coordinate latent mapping of the NVP model learns keyframe vectors and sparse positional features representing general representative features of the video across the t-axis, y-axis, and x-axis. The coordinate latent mapping learns keyframe vector z as a latent embedding of (x, y), (x, t), (y, t) from latent keyframes, which are a 2D grid of the video. xy , z xt , z yt . In addition, the coordinate latent mapping is a 3D grid of latent sparse location features, which are positional features z as latent embeddings of (x, y, t). xytHere, the 2D grid, keyframe vectors, 3D grid, and position features can all be learned. The NVP model concatenates the latent embeddings, i.e., keyframe vectors and position features, to generate the latent vector z.

[0091] The latent RGB mapping of the NVP model uses a synthesizer and a modulator to generate RGB outputs at (x, y, t) based on z. The synthesizer is a K-layer multilayer perceptron (MLP) that takes t as input and generates RGB outputs at (x, y, t). The modulator is a K-1 layer MLP that takes z as input and modulates the hidden features z at the intermediate stage. k and provide the generated hidden features to the synthesizer. Hidden features z k Using (k=1, 2, ... K-1), the synthesizer modulates the activation function values ​​in the intermediate stages. The latent RGB mapping of the NVP model generates RGB outputs from (x, y, t) as in Equation 4.

[0092]

[0093] In mathematical expression 4, ⊙ represents the elementwise product. A k and b k represents learnable parameters and biases. In the intermediate step, the synthesizer obtains input based on temporal information and adds A to the input. k Multiply by b k The values ​​generated by adding are applied to a sine function, which is a nonlinear activation function. The synthesizer generates intermediate outputs by modulating the output of the activation function based on hidden features. In the final stage, the synthesizer generates RGB outputs (r, g, b) based on the output of the previous stage, omitting modulation.

[0094] Meanwhile, the encoding pipeline and decoding pipeline as described above can also be used for compression and restoration of a multi-view INRVC model.

[0095] Hereinafter, the multi-view INRVC model and the INRVC model are used interchangeably.

[0096] Hereinafter, latent embedding, latent embedding vector, and embedding vector are used interchangeably.

[0097] III. Embodiments according to the present disclosure

[0098] The video encoding device illustrated in FIG. 1 trains a multi-view INRVC model in a multi-view INRVC encoding unit (110) using a video having multiple views, thereby enabling the multi-view INRVC model to generate videos of various views. At this time, information representing 4D coordinates (v, x, y, t) is used as the input of the multi-view INRVC model, and (r, g, b) values ​​corresponding to the input are used as GT (Ground Truth). Here, v represents a viewpoint of the video, t represents a time corresponding to an image in the video, and (x, y) represents a 2D coordinate in the image. The training unit updates the parameters (weights) of the multi-view INRVC model using the difference between the output of the multi-view INRVC model and the GT, thereby enabling the multi-view INRVC model to learn to generate images at various views and various times. The multi-view INRVC model, once trained, can include 3D spatial information inherent in the video based on the parameters in the model.

[0099] The neural network compressor (120) compresses the parameters of the trained multi-view INRVC model to generate a bitstream. For example, the multi-view INRVC model can be compressed using the encoding pipeline shown in FIG. 9. The video encoding device transmits the generated bitstream to the video decoding device.

[0100] The neural network structure of the multi-view INRVC model can be implemented using NVP, by adding an additional neural network model to NVP, or by modifying a portion of NVP. For example, as described above, NVP can be implemented using 2D / 3D grid-based coordinate latent mapping and MLP-based latent RGB mapping. In this case, the multi-view INRVC model according to the present disclosure can include 4D grid-based coordinate latent mapping, as in the examples of FIGS. 1 and 2, to process multiple views.

[0101] FIG. 12 is an exemplary diagram illustrating a multi-view INRVC model according to one embodiment of the present disclosure.

[0102] As illustrated in FIG. 12, the multi-view INRVC model according to the present disclosure utilizes a learnable 4D grid with additional multi-views. As illustrated in FIG. 11, the existing NVP model uses a 3D grid-based embedding representing positional information for the height, width, and time of a video (x, y, t). In the present disclosure, a 4D grid is utilized to represent positional information of (v, x, y, t) by adding a multi-view dimension v.

[0103] For example, as illustrated in Fig. 12, a video acquired at any point in time is equivalent to a video corresponding to (x, y, t). Therefore, the NVP model can be utilized for a 4D grid based on a video at any point in time. That is, the multi-view INRVC model can utilize coordinate latent mapping and latent RGB mapping.

[0104] The coordinate latent mapping of the multi-view INRVC model generates keyframe vectors as latent embeddings from a 2D grid for a multi-view video (v, x, y, t). The 2D grid may contain latent keyframes for, for example, (v, x), (v, y), (v, t), (x, y), (x, t), and (y, t). In the example of Fig. 12, some of the possible keyframe vectors are represented. The multi-view INRVC model generates 4D position features as latent embeddings of (v, x, y, t) from the 4D grid, which are 4D latent position features. In addition, the multi-view INRVC model may additionally include latent embeddings corresponding to camera parameters to distinguish multiple views. Therefore, the multi-view INRVC model generates latent vectors by combining keyframe vectors, 4D position features, and latent embeddings of camera parameters. Here, 2D grids, keyframe vectors, 4D grids, and 4D positional features can all be learned. Furthermore, latent embeddings of camera parameters can also be learned.

[0105] Additionally, the multi-view INRVC model can generate 3D positional features as latent embeddings from a 3D grid, which is a 3D latent positional feature. The 3D grid can include 3D latent positional features for, for example, (v, x, y), (v, x, t), (v, y, t), and (x, y, t). In the example of Fig. 12, the 3D grid is omitted.

[0106] When using the multi-view INRVC model, as mentioned above, the input including the viewpoint is expressed as a 4D coordinate (v, x, y, t). The multi-view INRVC model outputs the (r, g, b) values ​​in the 4D coordinate. That is, the multi-view INRVC model obtains one viewpoint v as input and generates a video corresponding to (x, y, t) as output.

[0107] In the multi-view INRVC model according to the present disclosure, the sizes of the 2D grid and the 3D grid can be determined according to the size of the 4D grid. Therefore, by increasing or decreasing the size of the learnable 4D grid, the video encoding device can adjust the trade-off between the size of the neural network that determines the size of the bitstream and the resilience of the multi-view video. The present disclosure verifies how the selection of the neural network size affects rate distortion, and adaptively selects the size of the neural network according to the verification result. That is, the video encoding device can train the structure of the multi-view INRVC model of various sizes, and then transfer the structure of the INRVC model that provides the optimal performance to the video decoding device.

[0108] As an example, a multi-view INRVC structure can be shared between a video encoding device and a video decoding device, and parameters can be transmitted separately. For example, a video encoding device and a video decoding device share multi-view INRVC structures that are configured differently according to the structure of a multi-view INRVC model using a 4D grid of an arbitrary size. The video encoding device selects which structure provides the best performance and determines an index of the selected multi-view INRVC structure. The video encoding device transmits the index together with the trained parameters to the video decoding device. The video decoding device determines the multi-view INRVC structure using the transmitted index, and applies the decoded parameters to restore the multi-view INRVC model.

[0109] As another example, the neural network compressor (120) can adjust the size and reproducibility of the bitstream using quantization parameters. The present disclosure verifies how the selection of the quantization parameter affects the rate distortion, and adaptively selects the quantization parameter based on the verification result. For example, as the value of the quantization parameter increases, the bit rate decreases. However, if the value of the quantization parameter increases beyond a certain threshold (i.e., the size of the quantization step increases), the fidelity of the reproducibility may decrease rapidly. Therefore, the video encoding device implements a multi-view INRVC model using various quantization parameters to find the quantization parameter that provides the optimal performance, and then transmits the found quantization parameter to the video decoding device.

[0110] A 4D grid for application to a multi-view INRVC model can be generated as follows. The grid can be defined in a way that constructs a multi-view INRVC model in the view direction. At this time, the video encoding device can transmit an IPS (INRVC Parameter Set) for each grid, thereby causing the video decoding device to update the relevant parameters and model.

[0111] FIGS. 13A to 13C are exemplary diagrams showing the configuration of a multi-view grid according to one embodiment of the present disclosure.

[0112] The video encoding device can organize videos from multiple views 0 to 8 into a grid as follows.

[0113] As an example, as in the example of FIG. 13a, the video encoding device groups videos by latitude and organizes them into a single grid, and generates a multi-view INRVC model corresponding to the organized grid.

[0114] As another example, as in the example of FIG. 13b, the video encoding device groups videos by latitude and organizes them into a single grid, and generates a multi-view INRVC model corresponding to the organized grid.

[0115] As another example, as in the example of FIG. 13c, the video encoding device groups videos listed in order of the smallest difference in latitude or longitude into a single grid, and generates a multi-view INRVC model corresponding to the configured grid.

[0116] Camera parameters include, as described above, intrinsic camera parameters for transforming from the image coordinate system to the camera coordinate system, and extrinsic parameters that specify the position and viewing direction of the camera in the world coordinate system. The intrinsic camera parameters include the focal length, the principal point on the image plane, and the skew coefficient, while the extrinsic camera parameters include rotation and translation coefficients.

[0117] The image encoding device can learn the aforementioned camera parameters as follows and then transmit them to the image decoding device.

[0118] When camera parameters for an arbitrary viewpoint are not given, the image encoding device uses a learnable embedding vector having the same size as the camera parameter information or a predefined size. In this case, the image encoding device can set the initial input to 0 or arbitrary values, and then learn to generate an embedding vector corresponding to the camera parameters. For example, the image encoding device can generate an embedding vector corresponding to the camera parameters using output values ​​based on MLP or CNN layers.

[0119] Given camera parameters for an arbitrary viewpoint, the image encoding device generates an embedding vector by applying positional encoding consisting of cosine and sine functions to the camera parameters, similar to Equation 1. The image encoding device can use the generated embedding vector as is. As another example, to derive an embedding vector corresponding to optimal camera parameters, the image encoding device can use a method for generating a learnable embedding that uses the given camera parameters as initial input. For example, the image encoding device can generate an embedding vector corresponding to optimal camera parameters using output values ​​based on MLP or CNN layers.

[0120] Below, the latent RGB mapping of the multi-view INRVC model is described.

[0121] As described above, the video encoding apparatus generates a latent vector z that includes latent embeddings of keyframe vectors, position features, and camera parameters. The latent RGB mapping of the multi-view INRVC model generates an RGB (or YCbCr) output at (v, x, y, t) based on z. The latent RGB mapping uses a synthesizer, which is a K-layer MLP, as described above, and a modulator that provides hidden features to the synthesizer. The modulator is a K-1 layer MLP. The synthesizer uses (v, t) as input to generate an RGB output at (v, x, y, t). The modulator uses z as input to generate hidden features z at an intermediate stage. k Generate (k=1, 2, ... K-1) and provide the generated hidden features to the synthesizer. Hidden feature z k Using , the synthesizer modulates the activation function values ​​at intermediate stages. The latent RGB mapping of the multi-view INRVC model generates RGB outputs at (v, x, y, t) as in Equation 5.

[0122]

[0123] In mathematical expression 5, ⊙ represents element-wise product. A k and b k represents learnable parameters and biases. In the intermediate step, the synthesizer obtains input based on time / point information, and adds A to the input. k Multiply by b k The values ​​generated by adding are applied to the sine function, which is a nonlinear activation function. The synthesizer generates the intermediate output by modulating the output of the activation function based on the hidden features. In the final stage, the synthesizer generates the RGB output (r, g, b) based on the output of the previous stage, omitting the modulation.

[0124] As another example, instead of the sine function in Equation 5, a wavelet function or a Gaussian function can be used as a nonlinear activation function. For example, a complex Gabor wavelet function can be used.

[0125] The video encoding device determines the input range of the multi-view INRVC model by determining the upper and lower bounds of the viewpoints used in training and encoding the multi-view INRVC model. The multi-view INRVC model, after completing training, can generate a video corresponding to the viewpoints within the input range. In other words, the input range represents the range of existence of 4D coordinates. The video encoding device transmits the input range of the multi-view INRVC model to the video decoding device. As another example, a preset input range may be used according to an agreement between the video encoding device and the video decoding device.

[0126] Meanwhile, while the INRVC model is used to reproduce multi-view video, the video encoding device additionally transmits existing depth information (depth map), and the video encoding device and video decoding device can use depth information together. Below, the learning of depth information for multi-view video is described.

[0127] An image encoding device may lossy / lossless compress depth information (depth map) and then transmit it to an image decoding device, and the image decoding device may decode the depth information and then use the decoded depth information. At this time, the image decoding device may compress and transmit depth information having the same resolution as a multi-view image. Alternatively, the image decoding device may compress and transmit low-resolution (or down-sampled) depth information, and the image decoding device may restore the low-resolution depth information and then apply super-resolution or up-sampling to the restored depth information. In the present disclosure, a method of learning depth information and then compressing, transmitting, and restoring it when depth information does not exist is proposed. Therefore, the image decoding device may generate / derive and use depth information, and may also output an image of an arbitrary viewpoint using the depth information.

[0128] FIG. 14 is an exemplary diagram illustrating an INRVC model that learns depth information according to one embodiment of the present disclosure.

[0129] As shown in the example of Fig. 14, the image encoding device can generate depth information by using the INRVC model including the 4D grid-based coordinate latent mapping, the utilization of camera parameter information, and the latent RGB mapping as a depth information model. At this time, the image encoding device uses (v, x, y, t) as described above as input. In addition, it uses the depth information at the (v, x, y, t) coordinates as GT for the input.

[0130] End-to-end learning may not be possible when depth information is not provided as a GT. Existing single-image-based depth prediction models can be utilized to generate depth information corresponding to GT. However, to maintain depth consistency by compensating for model errors and depth differences between each single image, learnable parameters α are required. i Wow β i is used. The output D of a single image-based model i Based on this, the image encoding device provides depth information D corresponding to GT. i * is calculated as in mathematical formula 6.

[0131]

[0132] The image encoding device stores depth information D i * The depth information model can be trained so that the difference between the output of the depth information model and the depth information model is reduced. As an example, the depth information D i * Based on the difference between the output of the depth information model and the output of the depth information model, a loss function can be defined as in Equation 7.

[0133]

[0134] In mathematical expression 7, N represents the number of coordinates used for training.

[0135] Meanwhile, a loss function can be defined as follows to train a multi-view INRVC model that generates multi-view videos. If a GT exists, the video encoding device can define the loss function using the difference between the output of the multi-view INRVC model and the GT. On the other hand, if a GT for the current viewpoint does not exist, the video encoding device can generate a GT for the current viewpoint using the GT of the reference viewpoint, and define the loss function based on the difference between the GT of the current viewpoint and the output of the multi-view INRVC model.

[0136] As an example, a loss function based on a point cloud projected with depth information can be defined. The image encoding device can define camera intrinsic parameters or a depth map D. * Projecting the point cloud P={P i * , i=0, 1, ..., N-1} is generated. N represents the number of viewpoints of the point cloud used for training. A loss function is defined to maintain consistency between point clouds for different viewpoints. For example, point clouds P at viewpoints i and j i Wow P i Based on this, the loss function is defined as in Equation 8.

[0137]

[0138] In Equation 8, ∥·∥2 represents the l2 loss.

[0139] As another example, a photometric loss function based on warping of RGB pixels can be defined. As in the example of Fig. 15, warping of RGB pixels refers to the process of shifting the coordinate system of a reference image to the world coordinate system using the corresponding depth information (or camera parameters) and transforming the world coordinate system to the coordinate system of the current image. An image encoding device can generate a frame targeting the current viewpoint using the aforementioned warping. The loss function can be defined based on the l2 loss between the reference image warped to the current viewpoint and the current image. Here, the current image represents an image corresponding to the current viewpoint, and the reference image represents an image corresponding to the reference viewpoint. In the example of Fig. 15, H represents the shift of RGB pixels in the reference image based on warping.

[0140] As another example, a loss function based on warping of a 4D grid can be defined. An image encoding device applies warping between viewpoints to a 4D grid-based embedding of a model. By transforming the coordinates of a reference viewpoint of the grid into 3D world coordinates using the corresponding depth (or camera parameters) and then utilizing warping to transform the 3D world coordinates into grid coordinates of the current viewpoint, the image encoding device can generate a frame targeting the current viewpoint. The loss function can be defined based on the difference between the grid-based embedding vector of the reference viewpoint warped to the current viewpoint and the current grid-based embedding vector.

[0141] In conclusion, the video encoding device transmits, as additional information, the camera parameters used for training and encoding the multi-view INRVC model, and the quantization parameters applied for compression of the multi-view INRVC model. In addition, the video encoding device may transmit, as additional information, information regarding the structure of the multi-view INRVC model and information related to depth information. Here, the information related to depth information may include syntax indicating whether to transmit depth information.

[0142] The image decoding device illustrated in FIG. 2 decodes parameters of a multi-view INRVC model from a bitstream using a neural network decompressor (210). For example, the multi-view INRVC model can be reconstructed using the decoding pipeline illustrated in FIG. 10. The image decoding device decodes, as additional information, camera parameters used for training and encoding the multi-view INRVC model, an input range of the multi-view INRVC model, and a quantization parameter applied to compression of the multi-view INRVC model. The image decoding device decodes, as additional information, information about the structure of the multi-view INRVC model and information related to depth information. In addition, when depth information is transmitted according to a syntax indicating whether to transmit depth information, the image decoding device additionally decodes depth information.

[0143] The multi-view INRVC decoding unit (220) generates a video from multiple views using the reconstructed multi-view INRVC model. After acquiring an arbitrary viewpoint j, the video decoding device can input the arbitrary viewpoint into the multi-view INRVC model to generate a video corresponding to the arbitrary viewpoint. At this time, the arbitrary viewpoint must be within the input range. For example, if the arbitrary viewpoint does not exist within the input range, the video decoding device may not provide a video corresponding to the viewpoint.

[0144] To convey the aforementioned additional information, an INRVC Parameter Set (IPS) is defined that contains the following syntax elements:

[0145] Information related to the structure of a multi-view INRVC model includes syntax defining the size of the 4D grid that constitutes the model, syntax defining the structure of the edges that constitute the model, syntax defining the format and precision of the parameters that constitute the model, syntax defining the upper and lower bounds on the size of the 4D grid that constitutes the model, syntax defining the upper and lower bounds on the number of connections that constitute the model, and syntax defining the upper and lower bounds on the precision of the parameters that constitute the model.

[0146] Information related to camera parameters includes camera internal parameters and camera external parameters as described above.

[0147] Information related to the quantization parameters of the neural network compressor (120) includes syntax defining upper and lower limits on the available sizes of the quantization parameters.

[0148] Information related to the input range includes syntax defining the upper and lower bounds of the 4D coordinates. As mentioned above, the 4D coordinates include a point in time, a time, and a 2D coordinate.

[0149] Information related to depth information includes syntax that indicates whether depth information should be transmitted.

[0150] For example, if the above-described syntax indicates transmission of depth information, the image encoding device additionally transmits the depth information to the image decoding device. On the other hand, if the above-described syntax does not indicate transmission of depth information, the image encoding device may additionally transmit information related to the structure of the INRVC model-based depth information model as illustrated in FIG. 14, and parameters of the depth information model, to the image decoding device. The image decoding device may generate a depth information model based on the information related to the structure of the depth information model and the parameters of the depth information model, and may generate depth information using the depth information model.

[0151] Hereinafter, a method for encoding or restoring multi-view video using the cities of FIGS. 16 and 17 is described.

[0152] FIG. 16 is a flowchart illustrating a method for a video encoding device to encode videos of multiple views according to an embodiment of the present disclosure.

[0153] The video encoding device acquires videos of multiple views (S1600).

[0154] The video encoding device constructs a multi-view INRVC model (S1602). Here, the multi-view INRVC model utilizes camera parameters corresponding to the 4D grid and videos from multiple views. The video encoding device can construct the structure of the multi-view INRVC model differently based on the size of the 4D grid.

[0155] If camera parameters for any point among multiple viewpoints are not provided, the image encoding device uses a learnable embedding vector having the same size as the camera parameter information or a predefined size. On the other hand, if camera parameters for any point are provided, the image encoding device can generate an embedding vector by applying positional encoding to the camera parameters and then use the generated embedding vector as is. As another example, the image encoding device can use a method for generating a learnable embedding that uses the given camera parameters as initial input.

[0156] The image encoding device performs training of a multi-view INRVC model based on multi-views, a 4D grid, and camera parameters (S1604).

[0157] As an example, when a multi-view INRVC model such as the one illustrated in FIG. 12 is used, an input including a viewpoint is expressed as a 4D coordinate (v, x, y, t). Here, the input includes a viewpoint v of a video, a time t corresponding to an image within the video, and (x, y) representing a 2D coordinate within the image. The multi-view INRVC model outputs (r, g, b) values ​​in 4D coordinates. That is, the multi-view INRVC model obtains one viewpoint v as an input and generates a video corresponding to (x, y, t) as an output.

[0158] A video encoding device inputs one of multiple viewpoints into a multi-view INRVC model and generates an output of the multi-view INRVC model from one viewpoint based on a 4D grid and camera parameters. The video encoding device calculates the difference between the output of the multi-view INRVC model and a GT (Ground Truth). Here, a video corresponding to one viewpoint, i.e., (r, g, b) values ​​corresponding to the input, are used as the GT. The video encoding device updates the parameters of the multi-view INRVC model based on the calculated difference.

[0159] For example, a video encoding device can generate an output of a multi-view INRVC model from one viewpoint as follows. The video encoding device generates a first latent embedding, which is a 4D positional feature corresponding to one viewpoint, based on a 4D grid. The video encoding device generates a second latent embedding corresponding to camera parameters of the current viewpoint. In addition, the video encoding device generates a third latent embedding from a 2D grid. The video encoding device combines the aforementioned latent embeddings to generate a latent vector. The video encoding device provides the latent vector to a modulator, which is a first MLP, to generate hidden features. The video encoding device inputs information about one viewpoint and time information to a synthesizer, which is a second MLP. The video encoding device can generate an output of the multi-view INRVC model from one viewpoint by modulating information about one viewpoint and time information based on the hidden features and synthesizing a video of one viewpoint.

[0160] The image encoding device determines the input range of the multi-view INRVC model based on the multi-views used for training (S1606).

[0161] For example, for the multi-viewpoints used in training the multi-viewpoint INRVC model, by determining the upper and lower limits of the aforementioned multi-viewpoints, the image encoding device determines the input range of the viewpoints input to the multi-viewpoint INRVC model.

[0162] The image encoding device determines quantization parameters and information about the structure of a multi-view INRVC model based on the results of training (S1608).

[0163] An image encoding device can select the optimal multi-view INRVC model structure in terms of rate distortion and training speed. Furthermore, the image encoding device can adaptively determine quantization parameters for rate distortion optimization.

[0164] The video encoding device encodes the parameters of the multi-view INRVC model using quantization parameters (S1610). For example, the video encoding device can compress the parameters of the multi-view INRVC model using an encoding pipeline as illustrated in FIG. 9.

[0165] The video encoding device encodes IPS (S1612).

[0166] Here, the IPS (INRVC Parameter Set) includes quantization parameters as additional information, information about the structure of the multi-view INRVC model, and information about the input range of the multi-view INRVC model. If camera parameters are provided, the IPS may additionally include camera parameters as additional information. In addition, the IPS may additionally include syntax indicating whether to transmit depth information (depth map) as additional information.

[0167] If the syntax indicating whether to transmit depth information indicates transmitting depth information, the image encoding device can additionally encode depth information.

[0168] FIG. 17 is a flowchart illustrating a method for a video decoding device to restore a video at a current point in time according to an embodiment of the present disclosure.

[0169] The video decoding device decodes parameters of a multi-view INRVC model and an IPS from a bitstream (S1700). Here, the IPS includes information about quantization parameters, the structure of the multi-view INRVC model, and the input range of the multi-view INRVC model as additional information. In addition, the IPS may additionally include syntax indicating whether to transmit camera parameters and depth information (depth map) as additional information.

[0170] If the syntax indicating whether to transmit depth information indicates transmitting depth information, the image decoding device can additionally decode depth information.

[0171] The image decoding device restores the multi-view INRVC model to match information about the structure of the multi-view INRVC model using parameters of the multi-view INRVC model and quantization parameters (S1702).

[0172] Here, the multi-view INRVC model utilizes a 4D (dimensional) grid and camera parameters. The image decoding device can, for example, determine the size of the 4D grid using information about the structure of the multi-view INRVC model. The image decoding device can decode IPS for each 4D grid.

[0173] If the IPS does not include camera parameters, the image decoding device utilizes a learned embedding vector corresponding to the camera parameters. Conversely, if the IPS includes camera parameters, the image encoding device can apply positional encoding to the camera parameters to generate an embedding vector, and then use the generated embedding vector as is. As another example, the image decoding device can use camera parameters as input to generate a learned embedding vector.

[0174] The video decoding device obtains the current point in time within the input range (S1704).

[0175] The video decoding device inputs the current viewpoint into a multi-view INRVC model and generates a video of the current viewpoint based on a 4D grid and camera parameters (S1706). As described above, the multi-view INRVC model obtains a single viewpoint v as input and generates a video corresponding to (x, y, t) as output.

[0176] As an example, an image decoding device generates a first latent embedding, which is a 4D positional feature corresponding to the current viewpoint, based on a 4D grid. The image decoding device generates a second latent embedding corresponding to the camera parameters of the current viewpoint. Furthermore, the image decoding device generates a third latent embedding from a 2D grid. The image decoding device combines the aforementioned latent embeddings to generate a latent vector.

[0177] The video decoding device generates hidden features by providing latent vectors to the first MLP, the modulator. The video decoding device inputs information about the current time point and time information to the second MLP, the synthesizer, and modulates the information about the current time point and time information based on the hidden features, thereby synthesizing the video of the current time point.

[0178] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0179] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0180] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0181] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0182] (Explanation of symbols)

[0183] 110: Multi-view INRVC encoding unit

[0184] 120: Neural network compressor

[0185] 210: Neural Network Decompressor

[0186] 220: Multi-view INRVC decoding unit

[0187]

[0188] CROSS-REFERENCE TO RELATED APPLICATION

[0189] This patent application claims priority to Korean patent application No. 10-2023-0085049, filed in Korea on June 30, 2023, and Korean patent application No. 10-2024-0072308, filed in Korea on June 3, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A method for restoring a current point in time video performed by a video decoding device, A step of decoding parameters of a multi-view INRVC (Implicit Neural Representation for Video Coding) model and an IPS (INRVC Parameter Set) from a bitstream, wherein the IPS includes quantization parameters and information about the structure of the multi-view INRVC model; A step of restoring the multi-view INRVC model so as to conform to information about the structure of the multi-view INRVC model by using the parameters of the multi-view INRVC model and the quantization parameter, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters; A step of obtaining the current view point; and A step of inputting the current time point into the multi-view INRVC model and generating a video of the current time point based on the 4D grid and the camera parameters. A method, characterized by including:

2. In paragraph 1, The steps for restoring the above multi-view INRVC model are: A method characterized in that the size of the 4D grid is determined using information about the structure of the multi-view INRVC model.

3. In paragraph 1, Information about the structure of the above multi-view INRVC model is: A method, characterized by including a syntax defining the size of the 4D grid, and a syntax defining the format and precision of parameters of the multi-view INRVC model.

4. In paragraph 1, The above IPS, A method, characterized in that it additionally includes the above camera parameters.

5. In paragraph 1, The steps for restoring the above multi-view INRVC model are: A method characterized in that, when the above IPS does not include the above camera parameters, a learned embedding vector corresponding to the above camera parameters is utilized.

6. In paragraph 1, The steps for decrypting the above IPS are: Decrypt the above IPS for each 4G grid, A method according to claim 1, wherein the 4G grid is generated based on a group of videos by latitude, a group of videos by longitude, or a group of videos listed in order of smallest difference in latitude or longitude.

7. In paragraph 1, The steps for generating the video of the current point in time are: A step of generating a first latent embedding, which is a positional feature corresponding to the current point in time, based on the above 4D grid; a step of generating a second latent embedding corresponding to the camera parameters of the current point in time; and A step of generating a latent vector by combining the first latent embedding and the second latent embedding. A method, characterized by including:

8. In paragraph 7, The steps for generating the video of the current point in time are: A step of providing the above latent vector to the first MLP to generate hidden features; and A step of synthesizing a video of the current time by inputting information about the current time and time information into a second MLP and modulating information about the current time and time information based on the hidden feature. A method, characterized in that it further comprises:

9. A method for encoding videos of multiple views, performed by a video encoding device, A step of acquiring videos of the above multiple viewpoints; A step of constructing a multi-view INRVC (Implicit Neural Representation for Video Coding) model, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters corresponding to videos of the multi-views; A step of training the multi-view INRVC model based on the multi-views, the 4D grid, and the camera parameters; A step of determining quantization parameters and information about the structure of the multi-view INRVC model based on the results of the training; A step of encoding parameters of the multi-view INRVC model using the above quantization parameters; and A step of encoding an IPS (INRVC Parameter Set) including the above quantization parameters and information about the structure of the multi-view INRVC model. A method, characterized by including:

10. In paragraph 9, A method, characterized in that when the above camera parameters are provided, the IPS additionally includes the camera parameters.

11. In paragraph 9, The steps for restoring the above multi-view INRVC model are: A method characterized in that, when the above camera parameters are not provided, a learnable embedding vector corresponding to the above camera parameters is utilized.

12. In paragraph 9, The step of encoding the above IPS is: Encode the above IPS for each 4G grid, A method according to claim 1, wherein the 4G grid is generated based on a group of videos by latitude, a group of videos by longitude, or a group of videos listed in order of smallest difference in latitude or longitude.

13. In paragraph 9, The steps for configuring the structure of the above multi-view INRVC model are: The structure of the multi-view INRVC model is configured differently based on the size of the above 4D grid, The step of determining information about the structure of the above multi-view INRVC model is: A step of selecting the structure of the multi-view INRVC model that is optimal in terms of rate distortion and speed of the training; and A step of including syntax elements indicating the structure of the multi-view INRVC model in information about the structure of the multi-view INRVC model. A method, characterized by including:

14. In paragraph 9, The step of training the above multi-view INRVC model is as follows: A step of inputting one of the above multi-viewpoints into the multi-viewpoint INRVC model and generating an output of the multi-viewpoint INRVC model at the one viewpoint based on the 4D grid and the camera parameters; A step of calculating the difference between the output of the multi-view INRVC model and GT (Ground Truth), wherein a video corresponding to one viewpoint is used as the GT; and A step of updating the parameters of the multi-view INRVC model based on the calculated difference. A method, characterized by including:

15. In paragraph 14, The step of generating the output of the above multi-view INRVC model is: A step of generating a first latent embedding, which is a positional feature corresponding to the one point in time, based on the above 4D grid; generating a second latent embedding corresponding to the camera parameters of the above one point in time; and A step of generating a latent vector by combining the first latent embedding and the second latent embedding. A method, characterized by including:

16. In paragraph 15, The step of generating the output of the above multi-view INRVC model is: A step of providing the above latent vector to the first MLP to generate hidden features; and A step of synthesizing a video of the one point in time by inputting information and time information about the one point in time into a second MLP and modulating the information and time information about the one point in time based on the hidden feature. A method, characterized in that it further comprises:

17. A method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: Step of acquiring videos of multiple points of view; A step of constructing a multi-view INRVC (Implicit Neural Representation for Video Coding) model, wherein the multi-view INRVC model utilizes a 4D (Dimensional) grid and camera parameters corresponding to videos of the multi-views; A step of training the multi-view INRVC model based on the multi-views, the 4D grid, and the camera parameters; A step of determining quantization parameters and information about the structure of the multi-view INRVC model based on the results of the training; A step of encoding parameters of the multi-view INRVC model using the above quantization parameters; and A step of encoding an IPS (INRVC Parameter Set) including the above quantization parameters and information about the structure of the multi-view INRVC model. A method, characterized by including:

Images