Method and apparatus for 3D video coding based on implicit neural network representation

The 3D video coding method employs an implicit neural network representation with a 3D INVR model and adaptive quantization to improve compression efficiency and quality, addressing the inefficiencies of existing methods in representing complex scenes and enhancing video quality from arbitrary viewpoints.

WO2024242387A1PCT designated stage expired Publication Date: 2024-11-28HYUNDAI MOTOR CO LTD +2
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/006487
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-05-09
Filing Date
2024-05-13
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Current 3D video coding methods are inefficient in representing and compressing video data, particularly when dealing with high-resolution and complex scenes, as they rely on explicit expression methods that are resolution-dependent and computationally costly.

Method used

The implementation of a 3D video coding method based on implicit neural network representation, specifically using a 3D Implicit Neural Video Representation (3D INVR) model that generates and transmits parameters, camera parameters, and adaptive quantization parameters to improve compression efficiency and quality, utilizing a multi-layer perceptron (MLP) and Convolutional Neural Network (CNN) structures.

Benefits of technology

This approach enhances 3D video coding efficiency and quality by enabling the generation of high-quality videos from arbitrary viewpoints, adaptively adjusting complexity based on the model's structure, and reducing computational requirements through efficient parameter transmission and compression.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006487_28112024_PF_FP_ABST
    Figure KR2024006487_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present embodiment discloses a 3D video coding method and apparatus based on an implicit neural network representation. In the present embodiment, an image decoding apparatus decodes, from a bitstream, parameters of a 3D INVR model, and an IPS. Here, the IPS includes a quantization parameter, information regarding a structure of the 3D INVR model, and information regarding the input range of the 3D INVR model. The image decoding apparatus reconstructs the 3D INVR model so as to match the information regarding the structure of the 3D INVR model by using the parameters of the 3D INVR model and the quantization parameter. The image decoding apparatus generates the current 3D video by acquiring a random view point within the input range and inputting the random view point into the 3D INVR model.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for 3D video coding based on implicit neural network representation

[0001] The present disclosure relates to a 3D video coding method and device based on implicit neural network representation.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Recently, research on implicit neural network representation models, which represent various data, including images, using neural network structures, has been active. Conventional video representation methods use explicit representations, which represent RGB pixel values ​​at each pixel location. To replace this explicit representation, implicit neural representations are being introduced, which represent functions that generate (r,g,b) values ​​from (x,y) coordinates (pixel locations) in an image using neural networks.

[0004] Among implicit neural representation methods, NeRF (Neural Radiance Fields) is a representative model used for synthesizing 3D video. Using a learning process to reconstruct all pixels within an image into world coordinates, NeRF outputs rendered results from an arbitrary viewpoint. At this time, NeRF uses as input indices in the world coordinate system and camera parameters required for projection into the world coordinate system. Therefore, to enhance 3D video encoding efficiency and improve 3D video quality, a method for efficiently constructing a 3D video representation model based on implicit neural network representations needs to be considered.

[0005] The present disclosure provides a 3D video coding method and device that generates, compresses, and transmits parameters of a 3D implicit neural video representation (3D INVR) model, generates and transmits camera parameters for improving the quality of an arbitrary viewpoint requested by a decoder side, and generates and transmits additional information for improving the compression method of the 3D INVR model.

[0006] In addition, the present disclosure aims to provide a 3D video coding method and device that generates additional information for improving the compression method of a 3D INVR model by adaptively changing the quantization parameters for compression of a 3D INVR model and the complexity according to the structure of the 3D INVR model.

[0007] According to an embodiment of the present disclosure, a method for restoring a current 3D (Dimensional) video, performed by a video decoding device, is provided, characterized by comprising the steps of: decoding parameters of a 3D (Dimensional) INVR (Implicit Neural Video Representation) model and an IPS (INVR Parameter Set) from a bitstream, wherein the IPS includes quantization parameters, information about a structure of the 3D INVR model, and information about an input range of the 3D INVR model; restoring the 3D INVR model using the parameters of the 3D INVR model and the quantization parameters so as to conform to the information about the structure of the 3D INVR model, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a convolutional neural network (CNN); acquiring an arbitrary view point within the input range; and inputting the arbitrary view point into the 3D INVR model to generate the current 3D video.

[0008] According to another embodiment of the present disclosure, a method for encoding 3D (Dimensional) videos, performed by a video encoding device, comprises the steps of: acquiring the 3D videos and camera parameters corresponding to the 3D videos; configuring a 3D INVR (Implicit Neural Video Representation) model, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a convolutional neural network (CNN); performing training of the 3D INVR model using viewpoints corresponding to the camera parameters and 3D videos corresponding to the camera parameters; determining an input range of the 3D INVR model based on viewpoints used in the training; determining a quantization parameter and information about a structure of the 3D INVR model based on a result of the training; encoding parameters of the 3D INVR model using the quantization parameter; And a method is provided, characterized by including a step of encoding an IPS (INVR Parameter Set) including the quantization parameter, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model.

[0009] According to another embodiment of the present disclosure, a method for providing video data to a video decoding device comprises: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein the encoding the video data comprises: obtaining 3D videos and camera parameters corresponding to the 3D videos; configuring a 3D INVR (Implicit Neural Video Representation) model, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a convolutional neural network (CNN); performing training of the 3D INVR model using viewpoints corresponding to the camera parameters and 3D videos corresponding to the camera parameters; determining an input range of the 3D INVR model based on viewpoints used in the training; determining a quantization parameter and information about a structure of the 3D INVR model based on a result of the training; encoding parameters of the 3D INVR model using the quantization parameter; And a method is provided, characterized by including a step of encoding an IPS (INVR Parameter Set) including the quantization parameter, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model.

[0010] As described above, according to the present embodiment, there is provided a 3D video coding method and device that generates, compresses, and transmits parameters of a 3D implicit neural video representation (3D INVR) model, generates and transmits camera parameters for improving quality at an arbitrary point in time requested by a decoder side, and additional information for improving a compression method of a 3D INVR model, thereby making it possible to improve 3D video encoding efficiency and 3D video quality in a video generation process at an arbitrary point in time.

[0011] In addition, according to the present embodiment, there is provided a 3D video coding method and device that generates additional information for improving the compression method of a 3D INVR model by adaptively changing the quantization parameters for compressing a 3D INVR model and the complexity according to the structure of the 3D INVR model, thereby making it possible to improve 3D video encoding efficiency and 3D video quality.

[0012] FIG. 1 is a block diagram illustrating a 3D INVR (Implicit Neural Video Representation)-based 3D image encoding device according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram illustrating a 3D INVR-based 3D image decoding device according to one embodiment of the present disclosure.

[0014] FIG. 3 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.

[0015] Figure 4 is an example diagram showing a SISR (Single Image Super Resolution) network.

[0016] Figure 5 is an example diagram showing a residual block used in SISR.

[0017] Figure 6 is an example diagram showing coordinate systems related to extraction of arbitrary points in NeRF (Neural Radiance Fields).

[0018] Figure 7 is an example diagram showing the NeRV (Neural Representations for Videos) model.

[0019] FIG. 8A and FIG. 8B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.

[0020] FIG. 9 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0021] FIG. 10 is an exemplary diagram illustrating a decryption pipeline of a NeRV model according to one embodiment of the present disclosure.

[0022] FIG. 11 is an exemplary diagram illustrating pruning using a bit mask according to one embodiment of the present disclosure.

[0023] FIG. 12 is an exemplary diagram illustrating warping of a reference image according to one embodiment of the present disclosure.

[0024] FIG. 13 is an exemplary diagram illustrating warping of a reference image / feature according to one embodiment of the present disclosure.

[0025] FIG. 14 is an exemplary diagram showing the definition of GoI according to one embodiment of the present disclosure.

[0026] FIG. 15 is a flowchart illustrating a method for an image encoding device to encode 3D videos according to an embodiment of the present disclosure.

[0027] FIG. 16 is a flowchart illustrating a method by which a video decoding device restores a current 3D video according to one embodiment of the present disclosure.

[0028] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0029] The present embodiment relates to encoding and decoding of 3D images (video). More specifically, the present invention provides a 3D video coding method and device that generates, compresses, and transmits parameters of a 3D Implicit Neural Video Representation (3D INVR) model, generates and transmits camera parameters for improving the quality of an arbitrary viewpoint requested by a decoder, and generates and transmits additional information for improving the compression method of the 3D INVR model.

[0030] FIG. 1 is a block diagram illustrating a 3D INVR-based 3D image encoding device according to one embodiment of the present disclosure.

[0031] In the example of FIG. 1, a 3D INVR-based 3D video encoding device (hereinafter, referred to as the 'video encoding device') generates (i.e., trains) a 3D INVR model to learn to generate a 3D video from various viewpoints, compresses the learned 3D INVR model to generate a bitstream, and transmits the generated bitstream. The video encoding device includes a 3D INVR encoding unit (110) including a 3D INVR model and a neural network compressor (120). Here, the components included in the video encoding device according to the present disclosure are not necessarily limited thereto. The video encoding device may additionally include a training unit (not shown) for training a 3D INVR model, or may be implemented in a form linked with an external training unit.

[0032] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0033] The video encoding device can store the bitstream of encoded video data on a non-transitory recording medium or transmit it to a 3D INVR-based 3D video decoding device using a communication network.

[0034] FIG. 2 is a block diagram illustrating a 3D INVR-based 3D image decoding device according to one embodiment of the present disclosure.

[0035] In the example of Fig. 2, a 3D INVR-based 3D video decoding device (hereinafter, referred to as the "video decoding device") decompresses a bitstream to restore a 3D INVR model, and generates a 3D video at various viewpoints using the restored 3D INVR model. The video decoding device includes a neural network decompressor (210) and a 3D INVR decoding unit (220) including a restored 3D INVR model.

[0036] Similar to the video encoding device illustrated in Fig. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor configured to execute the software functions corresponding to each component.

[0037] Below, before describing the components illustrated in FIGS. 1 and 2, the deep learning-related element technologies used in the present disclosure are described.

[0038] I. CNN(Convolutional Neural Network)

[0039] CNN, a neural network comprised of multiple convolutional layers and pooling layers, is a deep learning technique known to be ideal for image processing. Convolutional layers extract feature maps (also known as "features" or "characteristics") using multiple kernels or filters. The kernel coefficients that make up the filters are parameters determined during the learning process.

[0040] Among the convolutional layers of CNN, the front layer closer to the input extracts feature maps that respond to simple, low-level image features such as lines, points, or surfaces, while the back layer closer to the output extracts feature maps that respond to higher-level features such as textures and object parts.

[0041] FIG. 3 is an exemplary diagram showing the operation of a convolutional layer according to one embodiment of the present disclosure.

[0042] A convolutional layer generates a feature map from an input image using a convolution operation. The example in Fig. 3 illustrates a kernel (or filter) with a kernel size of 3×3. The kernel size is also referred to as the kernel size or filter size. The kernel has kernel parameters, also called weights. The kernel illustrated in Fig. 3 has a total of nine kernel parameters. The kernel parameters are initially set to arbitrary values, and their values ​​can be updated based on learning.

[0043] The convolution layer performs convolution operations using blocks of the same size as the kernel size in the input image. At this time, the blocks of the same size as the kernel size in the input image are referred to as windows.

[0044] When filtering an input image in raster scan order, the window movement size is called the stride. In the example of Fig. 3, the stride is 1. If the stride is set to 2, the convolution operation is performed by spacing the window by 2 samples, and as a result, the width and height of the feature map become half the width and height of the input image.

[0045] As mentioned above, a single convolutional layer can include multiple filters. The number of filters or kernels is called a channel. In other words, the number of channels is equal to the number of filters. Furthermore, the number of filters determines the dimensionality of the feature map.

[0046] Padding refers to a method of expanding input data by filling the area around it with a specific value before performing a convolution operation. Padding is primarily used to adjust the spatial size of the output data. The padding value can be determined by hyperparameters, but zero-padding is commonly used. Without padding, the spatial size of the output data decreases with each convolutional layer, potentially causing boundary information to disappear. Therefore, padding is used to prevent this problem. Specifically, padding can be used to match the spatial sizes of the output data from the convolutional layer to the input data.

[0047] The deconvolution layer performs the opposite operation to the convolution layer. It generates the desired data image from the input feature map as output.

[0048] The pooling layer performs pooling, a process of subsampling the feature map generated by the convolutional layer. The pooling layer uses a 2×2 window to select samples so that the output is half the width and height of the input. In other words, the pooling layer is used to reduce the size of the input image or input feature map by condensing a 2×2 region into a single sample.

[0049] The opposite concept of a pooling layer is defined as an unpooling layer. Unpooling layers, in contrast to pooling layers, function to expand the dimensionality and are primarily used after deconvolution layers.

[0050] The convolutional encoder-decoder architecture is a network structure composed of pairs of convolutional layers and deconvolutional layers. The convolutional encoder consists of a convolutional layer and a pooling layer, and outputs a feature map (or feature vector) from the input image. The final output vector of the convolutional encoder is also referred to as a latent vector. The convolutional decoder consists of a deconvolutional layer and an unpooling layer, and generates an output image from the feature map or latent vector.

[0051] The inputs and outputs of a convolutional encoder-decoder can be configured in various ways depending on the application and network purpose. For example, the inputs and outputs can be optical flow maps, saliency maps, image frames, etc.

[0052] Figure 4 is an example diagram showing an SISR network.

[0053] One example of CNN application is Single Image Super Resolution (SISR). The SISR network generates a high-resolution image from a low-resolution input image. As illustrated in Figure 4, the SISR network may include multiple convolutional layers. Each convolutional layer includes an activation function, such as the Rectified Linear Unit (ReLU). The parameters of the SISR network can be trained so that the resulting SR (Super Resolution) image approximates the Ground Truth (GT).

[0054] SR methods using CNN can improve SR performance by increasing the depth (e.g., increasing the number of convolutional layers). To overcome the overfitting problem that may occur in learning due to the increase in depth, a residual block that can perform skip connection and residual learning can be used in the SISR network. The residual block, as illustrated in Fig. 5, is a block that stores input features x l In addition to the path that applies the convolution operation, it includes a skip path. In addition, the residual block outputs x l+1 When generating, a path or skip path for applying convolution operations can also be selected based on learning efficiency. In the example of Fig. 5, the residual block includes a BN (Batch Normalization) layer.

[0055] For example, Enhanced Deep Residual Networks (EDSR) improves network performance by continuously connecting residual blocks to increase their depth. Another example is Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR), a CNN model based on the Visual Geometry Group (VGG) network. It uses residual learning, a method that adds residual frames to the final output. VDSR adds residual signals to the very end of the network, thereby adding them to the input signal.

[0056] II. Implicit neural representation model

[0057] Recently, research on implicit neural network representation models, which represent various data, including images, using neural network structures, has been active. Conventional video representation methods use an explicit representation method, which assigns RGB pixel values ​​to each pixel location. To replace this explicit representation method, implicit neural network representation models are being introduced, which represent functions that generate (r,g,b) values ​​from (x,y) coordinates (pixel locations) in an image using neural networks. Compared to explicit representation methods, implicit neural network representation models can be used regardless of image resolution.

[0058] As an example of 3D video coding that utilizes an implicit neural network representation model, there is NeRF (Neural Radiance Fields) technology.

[0059] By learning from images captured from multiple viewpoints of a multi-view video, NeRF synthesizes an image of an object viewed from any arbitrary position and orientation. After receiving a 5-dimensional coordinate consisting of a 3D position (x, y, z) that samples ray information in the direction the camera is looking and a viewing direction (θ, φ), NeRF outputs the RGB values ​​(r, g, b) and volume density σ corresponding to the input coordinates. Here, the 3D position (x, y, z) is the position in the world coordinate system, and θ and φ that constitute the viewing direction can represent the azimuth and altitude, respectively.

[0060] Compared to existing implicit neural network representations, NeRF offers the following improvements to better represent high-resolution, complex scenes. NeRF embeds positional information using positional encoding, and then uses the embedded index as input, as shown in Equation 1.

[0061]

[0062] As shown in the example in Equation 1, each spatial coordinate can be mapped to a 2L-dimensional vector γ(p). Instead of directly inputting the 3D input coordinates, each spatial coordinate can be mapped to a 2L-dimensional vector γ(p) using the sin and cos functions. By utilizing the embedded spatial index in training the NeRF model, the NeRF model can better predict data containing high-frequency variations.

[0063] NeRF is an implicit neural network representation model that uses a multi-layer perceptron (MLP) network containing multiple fully-connected layers (FC). NeRF represents an arbitrary scene by outputting RGB values ​​(r, g, b) and volume density σ corresponding to high-dimensional input coordinates. Then, the color C(r) of an arbitrary ray r passing through the scene can be rendered according to a volume rendering technique based on the RGB values ​​(r, g, b) and volume density σ. Since C(r) is differentiable, it is used for training NeRF together with GT (Ground Truth). At this time, an image that captures world coordinates (x, y, z) along the view direction (θ, φ) is used as GT for each 5-dimensional coordinate.

[0064] Meanwhile, to improve rendering efficiency, NeRF includes two networks, a coarse network and a fine network, instead of using a single MLP. NeRF uses stratified sampling to achieve N c After sampling the positions of the dogs, N c The output of the coarse network is calculated at each location. To generate more samples to apply to the fine network from the output of the coarse network, N c Composite color C based on the coarse network output of the dog c(r) is expressed as in mathematical formula 2.

[0065]

[0066] In mathematical expression 2, c i is N c The color of the coarse network calculated from the dog's location. w i is a value that depends on the distance between the volume and the sampled locations as a corresponding weight. Using inverse transform sampling, N with high weights from the aforementioned composite color f After sampling the location of the dog, NeRF c +N f Calculate the output of the fine network at the location of the dog, and color C based on the output of the calculated fine network. f (r) can be rendered. At this time, NeRF can be trained using the MSE (Mean Square Error) loss function between the color and GT according to the output of the coarse network and the fine network.

[0067] As mentioned above, NeRF outputs a rendered result at any point in time by using a learning process to reconstruct all pixels in the image into world coordinates. Therefore, a process is needed to transform all points projected to a point in the image coordinate system (x, y) into world coordinates. Therefore, the input of NeRF includes the camera parameters required for projection into the world coordinate system along with the index in the world coordinate system.

[0068] The process of extracting camera parameters is to convert pixels in an image into a world coordinate system [U, V, W] through a coordinate system [X, Y, Z] with a focal length Z = -1, as shown in the example of Fig. 6. At this time, prior information about the camera is essential, and the prior information, the camera parameters, includes internal parameters and external parameters. The internal camera parameters include the focal length, the principal point on the image plane, and the skew coefficient. According to the internal camera parameters, the object can be projected into the camera coordinate system [X, Y, Z]. In order to convert the information projected into the camera coordinate system into the world coordinate system [U, V, W], translation information about where the camera is located in the world coordinate system and rotation information about where it is looking are required. The external camera parameters include the aforementioned translation information and rotation information. When capturing an image, a matrix [R|t], which is a combination of a rotation matrix R and a translation vector t, is used to calculate the projection from the world coordinate system to the image coordinate system. Therefore, the camera extrinsic parameters can be extracted by calculating the inverse matrix of the combination matrix [R|t].

[0069] Meanwhile, the 5-dimensional coordinates, which are the inputs of NeRF, can be derived from the extracted camera parameters. As described above, the camera parameters can be derived based on the captured 2D images and the positions in the world coordinate system corresponding to the 2D images. When capturing images for GT use, the internal camera parameters can be fixed. When the internal camera parameters are used while being fixed, the 5-dimensional coordinates can be derived from the external camera parameters.

[0070] As another example, in addition to using the camera parameters used when shooting the video, a method such as colmap can also be used to extract feature points from re-view images and reconstruct information about the position and orientation of the camera within the world coordinate system.

[0071] As an example of 2D video coding using an implicit neural network representation model, there is Implicit Neural Representation for Video Coding (INRVC). The INRVC technology generates (r,g,b) values ​​for the (x,y) coordinates, which are pixel locations in an image. An example of the INRVC technology is the Neural Representation for Video (NeRV) model. As mentioned above, the explicit representation method expresses pixel values ​​on the (x,y,t) grid using the pixel locations (x,y) in the image and the time index t of the frame. The implicit neural network representation model outputs RGB pixel values ​​from the input representing the location (x,y,t). However, training the neural network at all locations (x,y,t) can significantly increase the computational complexity, so, as shown in the example of Figure 7, the NeRV model outputs the RGB image of the entire frame at the time index t using a neural network structure that only inputs the time index t.

[0072] FIG. 8A and FIG. 8B are exemplary diagrams illustrating the concept of the NeRV model according to one embodiment of the present disclosure.

[0073] To easily provide image output for a time index t input, it may be more effective to use a convolutional layer than to use an MLP network of an existing implicit representation model. The NeRV model can generate output using a structure in which NeRV blocks composed of multiple convolutional layers are stacked. As shown in the example of Fig. 8a, an MLP-based output and a NeRV block-based output can be compared. The NeRV block includes convolutional layers that perform convolution, a pixel shuffle layer, and an activation layer, as shown in the example of Fig. 8b. In the example of Fig. 8b, C represents the number of channels, W and H represent the width and height of the input of the NeRV block, and S represents an upscaling factor.

[0074] Additionally, the NeRV model embeds the time index t into a high-dimensional space as in Equation 3 and then uses the embedded index as input.

[0075]

[0076] As illustrated in Equation 3, the temporal index t can be mapped to a 2L-dimensional vector γ(t). By using the embedded temporal index t to train the NeRV model, the NeRV model can better predict video data containing high-frequency variations.

[0077] As the loss function of the NeRV model, a combination of L1 loss and SSIM (Structural Similarity Index) loss can be used. During training of the NeRV model, the aforementioned loss can be calculated at all pixel locations of the image estimated by the NeRV model based on the input and the ground truth (GT) image (i.e., the original image). The L1 loss is calculated using the absolute value of the difference between pixels in the estimated image and the GT image. The SSIM loss is calculated based on the mean, standard deviation, and correlation of pixels in the estimated image and the GT image. Thereafter, the weights (or parameters) that constitute the NeRV model during the training process can be updated based on the calculated loss.

[0078] FIG. 9 is an exemplary diagram illustrating an encoding pipeline of a NeRV model according to one embodiment of the present disclosure.

[0079] Since the original video can be approximately represented using the NeRV model, an end-to-end compression technology using a neural network can be implemented by compressing and transmitting the NeRV model. An encoding pipeline for encoding the NeRV model may include all or part of a video overfitter (910), a model pruner (920), a model quantizer (930), and a weight encoder (940), as shown in the example of Fig. 9.

[0080] The video overfitter (910) expresses the input video frame as a NeRV model. At this time, the video encoding device trains the NeRV model using the aforementioned loss function, thereby allowing the NeRV model to express the video frame. The model pruner (920) simplifies the NeRV model structure, which has an MLP or a mixed form of MLP and convolutional layers, using pruning. For example, pruning sets weights smaller than a preset threshold to zero. The model quantization unit (930) quantizes the weights to which pruning has been applied (e.g., weights greater than the threshold). The weight encoding unit (940) applies entropy encoding to the quantized weights to generate a bitstream of the weights.

[0081] Meanwhile, as a reverse order of the encoding pipeline illustrated in FIG. 9, a decoding pipeline for decoding a NeRV model can be illustrated as in FIG. 10. The decoding pipeline can include all or part of a weight decoder (1010), a model dequantizer (1020), a model reconstructor (1030), and a video generator (1040), as illustrated in FIG. 10.

[0082] The weight decoding unit (1010) applies entropy encoding to the bitstream of the weights of the NeRV model to generate quantized weights. The model dequantization unit (1020) dequantizes the quantized weights to generate pruned weights. The model restoration unit (1030) restores the NeRV model based on the pruned weights. The video generation unit (1040) uses the restored NeRV model to generate restored video frames according to the time index.

[0083] As mentioned above, the NeRV model uses the time index t of the frame as input. Therefore, when a video having pixels of size t×h×w is input, the NeRV model samples the input video only t times, not t×h×w times, so a large gain in both encoding speed and decoding speed can be expected.

[0084] Meanwhile, the encoding pipeline and decoding pipeline described above can also be used for compression and restoration of 3D INVR models.

[0085] Hereinafter, the 3D INVR model and INVR model are used interchangeably.

[0086] III. Embodiments according to the present disclosure

[0087] The video encoding device illustrated in FIG. 1 trains a 3D INVR model in a 3D INVR encoding unit (110) using 3D videos having various viewpoints, thereby enabling the 3D INVR model to generate 3D videos of various viewpoints. At this time, information representing various viewpoints is used as the input of the 3D INVR model, and images corresponding to each viewpoint are used as GT (Ground Truth). The training unit updates the parameters (weights) of the 3D INVR model using the difference between the output of the 3D INVR model and the GT, thereby enabling the INVR model to learn to generate 3D videos of various viewpoints. The 3D INVR model, once trained, can include 3D spatial information acquired by the 3D video based on the parameters in the model. Hereinafter, information representing a viewpoint used as an input is referred to as a viewpoint index.

[0088] The neural network compressor (120) compresses the parameters of the trained 3D INVR model to generate a bitstream. For example, the 3D INVR model can be compressed using the encoding pipeline shown in FIG. 9. The image encoding device transmits the generated bitstream to the image decoding device.

[0089] The neural network structure of the 3D INVR model can be implemented using NeRF, by adding an additional neural network model to NeRF, or by modifying parts of NeRF. For example, NeRF can be implemented using MLPs as described above. Alternatively, NeRF can be composed of a combination of MLPs and CNNs. In this case, NeRF can repeatedly include combinations of MLPs and CNNs.

[0090] When using a NeRF-based 3D INVR model, the image encoding device uses the 5-dimensional coordinates as described above as input indices. In addition, for each 5-dimensional coordinate, an image captured as a GT with world coordinates (x, y, z) along the view direction (θ, φ) is used. As described above, the 5-dimensional coordinates can be calculated from camera parameters corresponding to the GT. When the internal camera parameters are used while being fixed, the 5-dimensional coordinates can be calculated from the external camera parameters corresponding to the GT. The 3D video corresponding to the output of the 3D INVR model represents an image rendered in the world coordinate system by the 3D INVR model based on the 5-dimensional coordinates, or an image in the image coordinate system corresponding to the rendered image. For example, a 3D video can be composed of a plurality of rendered images having different viewpoints.

[0091] By increasing or decreasing the number of combinations of MLPs and CNNs when implementing a 3D INVR model, the present disclosure can adjust the tradeoff between the size of the neural network, which determines the size of the bitstream, and the fidelity of spatial representation of 3D video. The present disclosure verifies how the selection of the combination affects rate distortion, and adaptively selects the combination based on the verification result. For example, as the number of combinations of MLPs and CNNs increases, the bit rate increases and the fidelity of representation increases. However, if the number of combinations increases excessively beyond a certain threshold, the fidelity of representation may decrease rapidly. Therefore, a video encoding device implements a structure of a 3D INVR model including various combinations to find a structure that provides optimal performance, and then transmits information about the 3D INVR structure that provides optimal performance to a video decoding device. As the 3D INVR structure that provides optimal performance, the structure and parameters of the entire model can be transmitted, and the following methods can also be used.

[0092] As an example, the 3D INVR structure can be shared by a video encoding device and an image decoding device, and parameters can be transmitted separately. For example, the video encoding device and the image decoding device share a 3D INVR structure that is differently configured according to M, M+1, M+2, …, M+N combinations derived from the structure of a 3D INVR model having M+N modules including any M MLPs and any N CNNs. The video encoding device selects which structure provides the best performance and determines an index of the selected 3D INVR structure. The video encoding device transmits the index together with the trained parameters to the video decoding device. The video decoding device determines the 3D INVR structure using the transmitted index, and applies the decoded parameters to restore the 3D INVR model.

[0093] As another example, the neural network compressor (120) can adjust the size and reproducibility of the bitstream using quantization parameters. The present disclosure verifies how the selection of the corresponding quantization parameters affects rate distortion, and adaptively selects the corresponding quantization parameters based on the verification result. For example, as the value of the quantization parameter increases, the bitrate decreases. However, if the value of the quantization parameter increases beyond a certain threshold (i.e., the size of the quantization step increases), the fidelity of the reproducibility may decrease rapidly. Therefore, the image encoding device implements a 3D INVR model using various quantization parameters to find the quantization parameters that provide the optimal performance, and then transmits the found quantization parameters to the image decoding device.

[0094] The 3D INVR model requires camera parameters to output arbitrary intermediate viewpoint videos between viewpoints. As mentioned above, camera parameters include intrinsic camera parameters for transforming from image coordinates to camera coordinates, and extrinsic parameters that specify the camera's position and viewing direction in the world coordinate system. Intrinsic camera parameters include the focal length, camera principal point, and skew coefficient. Extrinsic camera parameters include rotation coefficients and translation coefficients.

[0095] An image encoding device can interpolate camera extrinsic parameters corresponding to surrounding viewpoints used in training and encoding a 3D INVR model, and can newly generate 5-dimensional coordinates representing an arbitrary intermediate viewpoint using the interpolated camera extrinsic parameters. At this time, the camera internal parameters are considered fixed. The image encoding device can additionally use the new 5-dimensional coordinates and the corresponding GT for training the 3D INVR model. At this time, the GT corresponding to the new 5-dimensional coordinates can be generated, for example, by interpolating GTs corresponding to the camera extrinsic parameters used in the interpolation. Specifically, the image encoding device moves GTs corresponding to surrounding viewpoints to a world coordinate system using the camera parameters, and then interpolates the moved GTs in the world coordinate system to generate a new GT. The image encoding device can generate a GT for the new 5-dimensional coordinates by moving the new GT to an image coordinate system using the interpolated camera parameters.

[0096] The video encoding device determines the input range of the 3D INVR model by determining the upper and lower limits of the components of the 5-dimensional coordinates used in training and encoding the 3D INVR model. In other words, the 3D INVR model, upon completion of training, can generate a 3D video corresponding to the 5-dimensional coordinates within the input range. The video encoding device transmits the input range of the 3D INVR model to the video decoding device. As another example, a preset input range may be used according to an agreement between the video encoding device and the video decoding device.

[0097] Meanwhile, although 3D INVR is used to reproduce 3D spatial information, the image encoding device additionally transmits existing depth information (depth map), and the image encoding device and the image decoding device can use the depth information together. The image encoding device can transmit the depth information to the image decoding device after lossy / lossless compression. Alternatively, the image decoding device can generate / derive the depth information without transmitting the depth information by the image encoding device.

[0098] In conclusion, the video encoding device transmits, as additional information, the camera parameters used for training and encoding the INVR model, and the quantization parameters applied for compression of the 3D INVR model. In addition, the video encoding device may transmit, as additional information, information regarding the structure of the 3D INVR model, the input range of the 3D INVR model, and information related to depth information. Here, the information related to depth information may include syntax indicating whether to transmit depth information.

[0099] The image decoding device illustrated in FIG. 2 decodes parameters of a 3D INVR model from a bitstream using a neural network decompressor (210). For example, the 3D INVR model can be reconstructed using the decoding pipeline illustrated in FIG. 10. The image decoding device decodes camera parameters used for training and encoding the INVR model and quantization parameters applied for compressing the INVR model as additional information. The image decoding device decodes information about the structure of the 3D INVR model, the input range of the 3D INVR model, and information related to depth information as additional information. In addition, if depth information is transmitted according to a syntax indicating whether to transmit depth information, the image decoding device additionally decodes the depth information.

[0100] The 3D INVR decoding unit (110) generates a 3D video from various viewpoint indices using the restored 3D INVR model. After acquiring an arbitrary intermediate viewpoint, the video decoding device inputs the arbitrary intermediate viewpoint into the 3D INVR model to generate a 3D video corresponding to the arbitrary intermediate viewpoint.

[0101] For example, when using a NeRF-based 3D INVR model, the video decoding device uses the 5-dimensional coordinates described above as input indices representing arbitrary intermediate viewpoints. For each 5-dimensional coordinate, the 3D INVR model reproduces an image looking at (x, y, z) along the view direction (θ, φ). At this time, the 5-dimensional coordinates representing arbitrary intermediate viewpoints must exist within the input range. For example, if the 5-dimensional coordinates do not exist within the input range, the video decoding device may not provide 3D video.

[0102] To convey the aforementioned additional information, an INVR Parameter Set (IPS) is defined that contains the following syntax elements:

[0103] Information related to the structure of the 3D INVR model includes syntax defining the size and number of MLPs and CNNs that constitute the model, syntax defining the structure of the edges that constitute the model, syntax defining the format and precision of the parameters that constitute the model, syntax defining upper and lower bounds on the size and number of MLPs and CNNs that constitute the model, syntax defining upper and lower bounds on the number of connections that constitute the model, and syntax defining upper and lower bounds on the precision of the parameters that constitute the model.

[0104] Information related to camera parameters includes internal parameters and camera external parameters as described above.

[0105] Information related to the quantization parameters of the neural network compressor (120) includes syntax defining upper and lower limits on the available sizes of the quantization parameters.

[0106] Information related to depth information includes syntax that indicates whether depth information should be transmitted.

[0107] Information related to the input range includes syntax defining the upper and lower bounds of the point in time.

[0108] Below, we describe a method for adjusting the tradeoff between the size of the neural network, which determines the bitstream size, and the fidelity of the spatial representation of 3D video by increasing or decreasing the number of combinations of MLPs and CNNs. Depending on the selection of the compression description method described below, or a combination of multiple compression methods, the video encoding device can adaptively compress the 3D INVR model.

[0109] The video encoding device can determine the overall performance of the INRV model, the performance of the compressed INRV model, the training speed, etc. by adjusting the number of MLP modules or CNN modules. As the number of modules increases and the size of the 3D INRV model increases, the fidelity of the representation increases in a convex shape. Therefore, the video encoding device can find the optimal point for the performance and size of the 3D INRV model. In addition, by adaptively adjusting the number of modules according to the task, the video encoding device can derive the optimal result for each task. When the quality of the output result must be high, the video encoding device configures the 3D INRV model by increasing the number of modules. However, as the number of modules increases, the complexity of the INRV model increases, which may require a long training time for the INRV model. If the number of modules increases excessively, the fidelity of the representation may decrease rapidly. In addition, compared to a lightweight model, the more complex the 3D INRV model, the lower the performance of the 3D INRV model when compressed with the same bits per pixel (bpp). Therefore, when the bit rate must be lower for reasons such as better compression and increased training speed, the video encoding device reduces the number of modules. Here, bpp can be defined as the value obtained by dividing the bitstream generated by encoding the trained INVR model by the resolution of the 3D video provided by the video encoding device.

[0110] As an example, the aforementioned NeRF may be composed of 10 consecutive MLP modules. As another example, NeRV may be composed of 1 MLP and 5 CNNs. As another example, HNeRV (Hybrid Neural Representation for Videos) may be composed of 5 combinations of CNNs and MLPs, and 6 additional CNNs. As described above, since the performance increases in a convex manner as the size of the 3D INRV model increases, the image encoding device may increase the number of modules until the performance converges. For example, the performance of the 3D INRV model may be improved by adding a CNN to the middle of the 3D INRV model. Experimentally, the performance of the 3D INRV model is improved until the number of CNNs added to the middle of the 3D INRV model increases to 10.

[0111] Below, we describe a method for adaptively reducing the number of MLPs and CNNs, or the number of parameters.

[0112] FIG. 11 is an exemplary diagram illustrating pruning using a bit mask according to one embodiment of the present disclosure.

[0113] During the training process of the 3D INRV model, the image encoding device can reduce the number of parameters using pruning. As an example of pruning, a bit mask, i.e., a binary mask consisting of 0s and 1s, is used. After learning the bit mask, the image encoding device multiplies the parameters of the INRV model by it, as shown in the example of Fig. 11. The bit mask determines the weights used for training and is updated according to the backpropagation of the loss function.

[0114] As another example of pruning, a method may be used to set the corresponding parameter to 0 if the absolute value of the parameter among the parameters of the 3D INRV model is lower than a preset threshold. As another example of pruning, a method may be used to set the weights of the parameters at positions outside of the preset positions in the parameter matrix to 0. For example, in a K×L parameter matrix, only the parameters located in the S×T region may be left, and the remaining parameters may be pruned to 0 (where K, L, S, and T are arbitrary positive integers, satisfying S≤K, T≤L).

[0115] During the training process of the 3D INRV model, the video encoding device uses a loss function that includes a factor that adjusts the size of the bitstream by adjusting the size of the parameters in addition to the video spatial reproducibility. As an example, the parameters of the 3D INRV model can be adjusted using the sparsity loss of the parameters. The sparsity loss is defined as in Equation 4.

[0116]

[0117] In Equation 4, |w|1 and |w|2 represent the L1 norm and the L2 norm.

[0118] By measuring the rate distortion of a trained 3D INRV model based on a loss function that includes sparse loss, the image encoding device determines the optimal number of MLPs and CNNs, or the optimal number of parameters. The sparse loss shown in Equation 4 is merely an example, and various sparse loss estimation methods may be used.

[0119] After training the 3D INRV model, the neural network compressor (120) compresses the parameters of the 3D INRV model using, for example, the encoding pipeline illustrated in FIG. 9. The neural network compressor (120) adjusts the tradeoff between the size of the bitstream and the performance of the final INVR model using parameters for sparsification, pruning, unification, local scaling, etc. For example, after the parameters of the model are configured as a matrix, the parameter matrix can be partitioned into any number of sub-blocks. Unification refers to an operation of setting one parameter value to some sub-blocks. Local scaling refers to an operation of scaling the parameter values ​​of some sub-blocks in the partitioned parameter matrix.

[0120] As an example, a video encoding device can transmit IPS using lossless compression.

[0121] Hereinafter, a technique for improving the quality of a video of an arbitrary viewpoint requested by a decoder is described, after generating it using an adaptive 3D INVR model. The video decoding device improves the image quality of the current viewpoint by referencing images of other viewpoints based on the current viewpoint by utilizing redundancy between frames of different viewpoints. In the present disclosure, references between RGB pixels and references at the feature level of a 3D INVR model are provided. As an example, for referencing, the video decoding device positions a reference camera coordinate system in a world coordinate system using depth information, and then transforms the world coordinate system into a camera coordinate system of a desired viewpoint. At this time, the video decoding device decodes the depth information, which is additional information transmitted by the video encoding device, and uses the decoded depth information. Alternatively, the video decoding device may derive the depth information and use the derived depth information. As another example, the video decoding device may transform the viewpoint of a reference image using camera parameters.

[0122] FIG. 12 is an exemplary diagram illustrating warping of a reference image according to one embodiment of the present disclosure.

[0123] As an example, an image decoding device improves the image quality of a current viewpoint by using warping of RGB pixels. As shown in the example of Fig. 12, warping of RGB pixels refers to the process of shifting the coordinate system of a reference image to a world coordinate system by using corresponding depth information (or camera parameters) and converting the world coordinate system to the coordinate system of the current image. The image decoding device generates a reference image for the current viewpoint by using the aforementioned warping. The image decoding device generates an improved current image by generating a weighted sum of the reference image warped to the current viewpoint and the current image. Here, the current image represents an image corresponding to the current viewpoint, and the reference image represents an image corresponding to the reference viewpoint. In the example of Fig. 12, H represents the shift of RGB pixels in the reference image based on warping.

[0124] FIG. 13 is an exemplary diagram illustrating warping of a reference image / feature according to one embodiment of the present disclosure.

[0125] As another example, an image decoding device can improve the image quality of the current viewpoint by using feature warping. At this time, the image decoding device performs warping between viewpoints at the feature level of the 3D INVR model. The feature warping method uses an MLP based on a neural network (NN), or uses a translation between the world coordinate system and the camera coordinate system based on depth information (or camera parameters), similar to RGB warping. For example, the last stage of the MLP of the NeRF model can be used as a reference feature. The NeRV block corresponding to the last stage of the NeRV model can be used as a reference feature. As an example, as in the example of FIG. 13, a reference feature / image warped using a CNN can be fused with the current feature / image. In the example of FIG. 13, warping refers to the process of converting the coordinates of the reference feature / image into the coordinates of the current feature / image. The image decoding device can concatenate the warped reference features / image and the current features / image, and then input the concatenated features into the CNN.

[0126] A GoI (Group of INVR) represents a method for configuring a 3D INVR based on a viewpoint and / or time. A video encoding device transmits an IPS for each GoI. A video decoding device can decode the IPS for each GoI to configure a 3D INVR model and related parameters.

[0127] FIG. 14 is an exemplary diagram showing the definition of GoI according to one embodiment of the present disclosure.

[0128] As illustrated in FIG. 14, as an example, a GoI is configured such that a single 3D INVR model represents multiple time points at a single time t. As another example, a GoI can be configured such that a single 3D INVR model represents multiple time points at a single time v. As another example, a GoI can be configured such that a single 3D INVR model represents multiple time points at multiple times (or multiple times at multiple time points).

[0129] Hereinafter, a method for encoding or restoring 3D video based on 3D INVR is described using the cities of FIGS. 15 and 16.

[0130] FIG. 15 is a flowchart illustrating a method for an image encoding device to encode 3D videos according to an embodiment of the present disclosure.

[0131] The video encoding device obtains 3D videos and camera parameters corresponding to the 3D videos (S1500).

[0132] The image encoding device constructs a 3D INVR model (S1502).

[0133] Here, the 3D INVR model includes a multilayer neural network and a CNN. The image encoding device can configure the structure of the 3D INVR model differently, for example, based on the size and number of MLPs and CNNs.

[0134] The video encoding device performs training of a 3D INVR model using viewpoints corresponding to camera parameters and 3D videos corresponding to camera parameters (S1504).

[0135] For example, when NeRF is used as a 3D INVR model, the elements that constitute viewpoints represent 5-dimensional coordinates. Here, the 5-dimensional coordinates include world coordinates (x, y, z) and viewpoint directions (θ, φ). The video encoding device can extract the components of the corresponding viewpoints using the camera parameters corresponding to each 3D video.

[0136] The video encoding device inputs viewpoints into a 3D INVR model to generate output. The video encoding device calculates the difference between the output of the 3D INVR model and the ground truth (GT). Here, the 3D videos acquired in the aforementioned step are used as the GT. The video encoding device updates the parameters of the 3D INVR model based on the aforementioned difference and a loss function including a sparse loss (e.g., as defined in Equation 4).

[0137] During the training process, the image encoding device can reduce the number of parameters of the 3D INVR model by using pruning.

[0138] The image encoding device determines the input range of the 3D INVR model based on the time points used for training (S1506).

[0139] For the viewpoints used in training the 3D INVR model, the image encoding device determines the input range of the 3D INVR model by determining the upper and lower limits of each element (e.g., 5-dimensional coordinates) constituting the aforementioned viewpoints.

[0140] The image encoding device determines quantization parameters and information about the structure of the 3D INVR model based on the results of training (S1508).

[0141] The video encoding device can select the optimal 3D INVR model structure in terms of rate distortion and training speed. Furthermore, the video encoding device can adaptively determine quantization parameters for rate distortion optimization.

[0142] The image encoding device encodes the parameters of the 3D INVR model using quantization parameters (S1510). For example, the image encoding device can compress the parameters of the 3D INVR model using an encoding pipeline as illustrated in FIG. 9.

[0143] The video encoding device encodes IPS (S1512).

[0144] Here, the IPS (INVR Parameter Set) includes quantization parameters as additional information, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model. Additionally, the IPS may additionally include syntax indicating whether to transmit camera parameters and depth information (depth map) as additional information.

[0145] If the syntax indicating whether to transmit depth information indicates transmitting depth information, the image encoding device can additionally encode depth information.

[0146] FIG. 16 is a flowchart illustrating a method by which a video decoding device restores a current 3D video according to one embodiment of the present disclosure.

[0147] The video decoding device decodes parameters of a 3D INVR model and IPS (INVR Parameter Set) from a bitstream (S1600).

[0148] The IPS (INVR Parameter Set) contains additional information such as quantization parameters, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model. Additionally, the IPS may additionally include syntax indicating whether to transmit camera parameters and depth information (depth map) as additional information.

[0149] The image decoding device restores the 3D INVR model to match information about the structure of the 3D INVR model using the parameters of the 3D INVR model and the quantization parameters (S1602).

[0150] Here, the 3D INVR model includes a multilayer neural network and a CNN. The image decoding device can, for example, determine the size and number of MLPs and CNNs using information about the structure of the 3D INVR model.

[0151] The video decoding device acquires a random point in time within the input range (S1604).

[0152] The video decoding device inputs an arbitrary point in time into the 3D INVR model to generate a current 3D video (S1606).

[0153] As an example, a video decoding device generates a reference image / feature for a current viewpoint using warping. Here, warping refers to a process of shifting the coordinate system of the reference image / feature to a world coordinate system using camera parameters and transforming the world coordinate system of the reference image / feature to the coordinate system of the current 3D video. Thereafter, the video decoding device generates an improved current 3D video by weighting the reference image / feature with the warping applied and the current 3D video / feature.

[0154] As another example, if the syntax indicating whether to transmit depth information (depth map) indicates the transmission of depth information, the video decoding device decodes the depth information. The video decoding device generates a reference image / feature for the current viewpoint using warping. Here, warping refers to the process of moving the coordinate system of the reference image / feature to the world coordinate system using the depth information and converting the world coordinate system of the reference image / feature to the coordinate system of the current 3D video. Thereafter, the video decoding device generates an improved current 3D video by weighting the reference image / feature to which the warping is applied and the current 3D video / feature.

[0155] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0156] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0157] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0158] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0159] (Explanation of symbols)

[0160] 110: 3D INVR encoding unit

[0161] 120: Neural network compressor

[0162] 210: Neural Network Decompressor

[0163] 220: 3D INVR decoding unit

[0164]

[0165] CROSS-REFERENCE TO RELATED APPLICATION

[0166] This patent application claims priority to Korean patent application No. 10-2023-0066090, filed in Korea on May 23, 2023, and Korean patent application No. 10-2024-0061136, filed in Korea on May 9, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A method for restoring a current 3D (Dimensional) video performed by a video decoding device, A step of decoding parameters of a 3D (Dimensional) INVR (Implicit Neural Video Representation) model and an IPS (INVR Parameter Set) from a bitstream, wherein the IPS includes quantization parameters, information about a structure of the 3D INVR model, and information about an input range of the 3D INVR model; A step of restoring the 3D INVR model so as to match information about the structure of the 3D INVR model using the parameters of the 3D INVR model and the quantization parameter, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a convolutional neural network (CNN); A step of acquiring an arbitrary view point within the above input range; and A step of generating the current 3D video by inputting the above arbitrary point in time into the 3D INVR model. A method, characterized by including:

2. In paragraph 1, The steps to restore the 3D INVR model are: A method characterized in that the size and number of the MLP and CNN are determined using information about the structure of the 3D INVR model.

3. In paragraph 1, Information about the structure of the above 3D INVR model, A method characterized by including a syntax defining the size and number of the MLP and CNN, and a syntax defining the format and precision of parameters of the 3D INVR model.

4. In paragraph 1, The above IPS, A method characterized by additionally including camera parameters.

5. In paragraph 4, A step of generating a reference image for a current point in time using warping, wherein the warping refers to a process of moving a coordinate system of the reference image to a world coordinate system using the camera parameters and transforming the world coordinate system of the reference image to a coordinate system of the current 3D video; A step of generating an improved current 3D video by weighting the reference image to which the above warping is applied and the current 3D video. A method, characterized in that it further comprises:

6. In paragraph 4, A step of generating a reference feature for a current viewpoint using warping, wherein the warping refers to a process of moving a coordinate system of the reference feature to a world coordinate system using the camera parameters and transforming the world coordinate system of the reference feature to a coordinate system of a feature of the current 3D video; A step of generating an improved current 3D video by weighting the reference features to which the above warping is applied and the features of the current 3D video. A method, characterized in that it further comprises:

7. In paragraph 1, The above IPS, A method characterized by additionally including syntax indicating whether to transmit depth information (depth map).

8. In paragraph 7, Further comprising a step of verifying the above syntax, A method characterized in that, if the syntax indicates transmission of the depth information, it further includes a step of decoding the depth information.

9. In paragraph 8, A step of generating a reference image for a current point in time using warping, wherein the warping refers to a process of moving a coordinate system of the reference image to a world coordinate system using the depth information and transforming the world coordinate system of the reference image to a coordinate system of the current 3D video; A step of generating an improved current 3D video by weighting the reference image to which the above warping is applied and the current 3D video. A method, characterized in that it further comprises:

10. In paragraph 1, The above decryption step is, Decrypt the above IPS by GoI (Group of INVR). A method, characterized in that the GoI is configured such that one 3D INVR model represents multiple points in time.

11. A method for encoding 3D (Dimensional) videos performed by a video encoding device, A step of obtaining the above 3D videos and camera parameters corresponding to the above 3D videos; A step of constructing a 3D INVR (Implicit Neural Video Representation) model, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a CNN (Convolutional Neural Network); A step of training the 3D INVR model using viewpoints corresponding to the above camera parameters and 3D videos corresponding to the above camera parameters; A step of determining the input range of the 3D INVR model based on the time points used in the above training; A step of determining quantization parameters and information about the structure of the 3D INVR model based on the results of the training; A step of encoding parameters of the 3D INVR model using the above quantization parameters; and A step of encoding an IPS (INVR Parameter Set) including the above quantization parameters, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model. A method, characterized by including:

12. In paragraph 11, The above IPS, A method characterized by further including syntax indicating whether to transmit the above camera parameters and depth information (depth map).

13. In paragraph 11, The steps to construct the structure of the 3D INVR model are: The structure of the 3D INVR model is configured differently based on the size and number of the MLP and CNN. The step of determining information about the structure of the above 3D INVR model is: A step of selecting the structure of the 3D INVR model that is optimal in terms of rate distortion and the speed of the training; and A step of including syntax elements indicating the structure of the above 3D INVR model in information about the structure of the above 3D INVR model. A method, characterized by including:

14. In paragraph 11, The step of determining the above quantization parameters is: A method characterized by adaptively determining the quantization parameters in terms of rate distortion optimization.

15. In paragraph 11, The step of determining the input range of the above 3D INVR model is: A method characterized in that the upper and lower limits of each element constituting the time points used for training the above 3D INVR model are determined.

16. In paragraph 11, The steps for training the above 3D INVR model are: A step of calculating the difference between the output of the above 3D INVR model and GT (Ground Truth), wherein the 3D videos are used as the GT; A method characterized by updating parameters of the 3D INVR model based on the above difference and a loss function including sparsity loss.

17. In paragraph 11, The steps for training the above 3D INVR model are: A method characterized by reducing the number of parameters of the 3D INVR model using pruning.

18. A method for providing video data to a video decoding device, A step of encoding the above video data into a bitstream; and A step of transmitting the above bitstream to the image decoding device Including, The step of encoding the above video data is: A step of obtaining 3D videos and camera parameters corresponding to the 3D videos; A step of constructing a 3D INVR (Implicit Neural Video Representation) model, wherein the 3D INVR model includes a multi-layer perceptron (MLP) and a CNN (Convolutional Neural Network); A step of training the 3D INVR model using viewpoints corresponding to the above camera parameters and 3D videos corresponding to the above camera parameters; A step of determining the input range of the 3D INVR model based on the time points used in the above training; A step of determining quantization parameters and information about the structure of the 3D INVR model based on the results of the training; A step of encoding parameters of the 3D INVR model using the above quantization parameters; and A step of encoding an IPS (INVR Parameter Set) including the above quantization parameters, information about the structure of the 3D INVR model, and information about the input range of the 3D INVR model. A method, characterized by including:

Citation Information

Patent Citations

  • Multi-view three-dimensional reconstruction method based on implicit neural representation

    CN115761178A

  • Neural rendering

    US20210248811A1

  • Implicit image and video compression using machine learning systems

    US20220385907A1