A multiview image and video compression method and system for 3D reconstruction

Abstract

The present invention relates to a multi-view compression method and system for 3D synthesis that evaluates the performance of a deep learning model by utilizing multi-view images compressed with MV-HEVC (Multiview High Efficiency Video Coding) as training data, comprising: (a) a step of encoding multi-view data using an MV-HEVC encoder designed to compress and transmit multi-view video data; (b) a step of utilizing the encoded multi-view images as training data for a 3D generating deep learning model; (c) a step of generating images from a new viewpoint using the compressed multi-view image data; and (d) a step of evaluating the impact of compression performance on the performance of the deep learning model. By evaluating the performance of a multi-view generating deep learning model against a 4D Gaussian Splatting model and analyzing the results, it can be confirmed that there is no significant difference in PSNR and SSIM values ​​compared to the case where original data is used, even when using a compressed dataset.

Description

Technology Field

[0001] The present invention relates to a multi-view compression method and system for 3D synthesis that evaluates the performance of a new viewpoint generation deep learning model based on 4D Gaussian Splatting using multi-view video data compressed with MV-HEVC (Multiview High Efficiency Video Coding).

[0002] In addition, the present invention relates to a multi-view compression method and system for 3D synthesis, which compresses multi-view data using MV-HEVC and then quantitatively analyzes the relationship between changes in the quality of a training dataset and deep learning model performance according to the compression strength. Background Technology

[0003] Recently, metaverse and extended reality (XR) technologies are gaining renewed attention in combination with artificial intelligence (AI) [Non-patent Literature 1]. The metaverse industry, which had been stagnant for some time, is presenting new possibilities for development based on the advancement of AI technology and the increasing demand for non-face-to-face environments.

[0004] These changes are expanding the demand for 3D content and free-viewpoint video technology, and active technological development is underway to enhance the immersion and quality of content [Non-patent Literature 2][Non-patent Literature 3]. Metaverse platforms are introducing new viewpoint generation technologies to enhance the immersion of content. Existing technologies have shown limitations in the efficiency of data processing and computational resources by providing only fixed images from specific viewpoints or relying on methods that generate all viewpoints in advance.

[0005] To solve these problems, virtual viewpoint synthesis technology has been introduced [Non-patent Literature 4]. This technology is attracting attention as a method that can simultaneously achieve data efficiency and realism by virtually generating an arbitrary viewpoint based on multi-view video.

[0006] Various systems, such as image domain systems, ray space systems, and model-based systems, are used for virtual viewpoint synthesis. Among these, representative technologies of model-based systems include NeRF (Neural Radiance Fields) and 3D Gaussian Splatting.

[0007] NeRF is a technology capable of generating images at any viewpoint by learning radiance values ​​in 3D space, and is attracting attention for its excellent expressiveness and realism. However, NeRF has limitations in that it requires large-scale computational resources during the learning and inference processes and must rely on large-scale original datasets [Non-patent Literature 6][Non-patent Literature 7].

[0008] 3D Gaussian Splatting is a technique proposed to alleviate these limitations, which simplifies complex computational processes by representing 3D data as a Gaussian distribution and enables efficient learning and real-time viewpoint synthesis [Non-patent Literature 8][Non-patent Literature 9]. In particular, 4D Gaussian Splatting is a model that extends 3D Gaussian Splatting to include temporal elements, contributing to the representation of dynamic scenes and the improvement of rendering performance [Non-patent Literature 10][Non-patent Literature 11][Non-patent Literature 12]. While these techniques contribute to extending the advantages of viewpoint generation networks, practical difficulties still remain in the storage and management of large-scale training data. Prior art literature

[0009] B. Hyewon and P. Jimin, "The resurrection of the metaverse with AI technology," Chosun Ilbo, Jan. 10, 2025. [Online]. Available: https: / / chosun.com. [Accessed: Jan. 20, 2025].A. Smolic, K. Mueller, P. Merkle, C. Fehn, P. Kauff, P. Eisert, and T. Wiegand, "3D Video and Free Viewpoint Video ?? Technologies, Applications and MPEG Standards," in Proceedings of the IEEE International Conference on Multimedia and Expo, 2006, pp. 2161??2164.M. Tanimoto, M. Tehrani, T. Fujii, and T. Yendo, "Free-viewpoint TV," IEEE Signal Processing Magazine, vol. 28, no. 1, pp. 67??76, Jan. 2011.L. Zhang, W. J. Tam, and D. Wang, "Multiview Video Sequence Analysis, Compression, and Virtual Viewpoint Synthesis," Journal of Visual Communication and Image Representation, vol. 21, no. 5??6, pp. 533??542, July 2010.C.Fehn, "Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Depth Image Based Rendering," Proceedings of the SPIE Conference on Three-Dimensional Image Processing and Applications, vol. 5302, pp. 93??104, 2004.Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM, 65(1):99??106, 2021.A. Pumarola, A. Agudo, A. Martinez, A. Sanfeliu, and F. Moreno-Noguer, "D-NeRF: Neural Radiance Fields for Dynamic Scenes," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10318??10327, 2021.Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Transactions on Graphics (ToG), 42(4):1??14, 2023.J. Zhang, L. Chen, H. Li, and W.Xu, "Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction," IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 11, pp. 3801??3812, Nov. 2023.Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang, "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering," School of CS, Huazhong University of Science and Technology; School of EIC, Huazhong University of Science and Technology; Huawei Inc., 2023.Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 130??141, 2023.Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM Siggraph Computer Graphics, 22(4): 65??74, 1988.T. Wiegand, G. Sullivan, and B. Girod, "Multiview Video Compression," IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 11, pp. 1550??1562, Nov. 2007.A. Vetro, T.Wiegand, and GJ Sullivan, “Efficient Compression of Multi-View Video Exploiting Inter-View Dependencies Based on H.264 / MPEG4-AVC,” Proceedings of the IEEE, vol. 97, no. 1, pp. 110??125, Jan. 2009.S. Donggyu and J. Hyunho, Understanding the HEVC Standard: High-Efficiency Video Coding Technology (Revised Edition), 2nd ed. Hongreung Science Publishing, Nov. 2015. [In Korean] Budagavi, Srinivasan I., Detlev Marpe, Gary J. Sullivan, J. Ridge Ohm, Thomas Wiegand, and Ye-Kui Wang. “Overview of the Multiview High Efficiency Video Coding (MV-HEVC) Standard.” IEEE Transactions on Circuits and Systems for Video Technology 26, no. 1 (January 2016): 35–45. Park, Si-nae, & Shim, Dong-kyu, "Multi-viewpoint video coding," Proceedings of the Summer Conference of the Institute of Electronics and Information Engineers, Pyeongchang, Gangwon-do, Korea, Jun. 2008, pp. 57-60. Park, Si-nae, & Shim, Dong-kyu, "Standard technology for 3D video coding," Journal of the Institute of Electronics and Information Engineers, vol. 37, no. 9, pp. 34-41, Sep 2010. ISO / IEC JTC1 / SC29 / WG11, "Call for proposals on 3D video coding technology," document N12036, Geneva, CH, Mar. 2011. H. Kang, S. Jang, and BTOh, "Voxel-based Point Clouds Rendering of G-PCC Compressed Point Clouds," in Proceedings of the Korean Institute of Information Scientists and Engineers (KIISE) Conference, pp. 1033??1038, 2023. The problem to be solved

[0010] The objective of the present invention is to solve the problems described above by providing a multi-view compression method and system for 3D synthesis that evaluates the performance of a deep learning model by utilizing multi-view images compressed with MV-HEVC (Multiview High Efficiency Video Coding) as training data [Non-patent Literature 13-20].

[0011] Existing technologies have lacked systematic analysis regarding the impact of highly compressed data on the training and inference performance of deep learning models, but the present invention quantitatively evaluates the relationship between changes in training data quality according to the compression strength (Quantization Parameter, QP) and deep learning model performance. Through this, it suggests the possibility of efficient storage and management of large-scale datasets and presents a practical solution that can reduce data resource requirements while maintaining the performance of deep learning models.

[0012] In other words, existing deep learning-based viewpoint generation models require large-scale data, and there are limitations in practicality regarding the storage and management of such data. The objective of the present invention is to solve this problem by providing a multi-viewpoint compression method and system for 3D synthesis that compresses multi-viewpoint data using MV-HEVC and then quantitatively analyzes the relationship between changes in the quality of the training dataset and the performance of the deep learning model according to the compression strength. means of solving the problem

[0013] To achieve the above objective, the present invention relates to a multi-view compression method and system for three-dimensional synthesis, comprising: (a) encoding multi-view data using an MV-HEVC encoder designed to compress and transmit multi-view video data; (b) utilizing the encoded multi-view image as training data for a three-dimensional generating deep learning model; (c) generating an image at a new viewpoint using the compressed multi-view image data; and (d) evaluating the effect of compression performance on the performance of the deep learning model.

[0014] In addition, the present invention relates to a multi-view compression method and system for three-dimensional synthesis, wherein in step (c), a new image is generated by adopting a 4D Gaussian splatting structure, and is characterized by generating the image in three stages: 3D point cloud generation, Gaussian generation and transformation, and rendering.

[0015] In addition, the present invention relates to a multi-view compression method and system for three-dimensional synthesis, wherein in step (c), feature points of each 2D image are extracted from a multi-view image, 2D key points are generated by matching identical feature points between images taken at different viewpoints, intrinsic and extrinsic parameters of the camera are estimated through camera correction, the generated 2D key points are re-projected into 3D space using a camera matrix, and pixel coordinates of the 2D image are converted into a 3D world coordinate system to form a point cloud. Effects of the invention

[0016] As described above, according to the multi-view compression method and system for 3D synthesis of the present invention, by evaluating the performance of a multi-view generation deep learning model for a 4D Gaussian Splatting model and analyzing the results, the effect of confirming that there is no significant difference in PSNR and SSIM values ​​compared to the case where the original data is used, even when using a compressed dataset, is obtained.

[0017] In other words, experimental results show that even when using a compressed dataset, the performance of the 4D Gaussian Splatting model does not show a significant difference in PSNR and SSIM values ​​compared to when using the original data. This proves that the performance of a deep learning model can be stably maintained even when using highly compressed data as training data.

[0018] The present invention confirms that it is possible to efficiently compress, store, and manage large-scale multi-view image datasets while maintaining the performance of deep learning models. Through this, it demonstrates the practicality of utilizing compressed datasets. Brief explanation of the drawing

[0019] FIG. 1 is a diagram illustrating the configuration of the entire system for implementing the present invention. FIG. 2 is a block diagram showing the configuration of an MV-HEVC encoder according to an embodiment of the present invention. FIG. 3 is a diagram showing the internal structure of 4D Gaussian Splatting according to an embodiment of the present invention. FIG. 4 is a graph between time and time points showing a PIP reference structure according to an embodiment of the present invention. Specific details for implementing the invention

[0020] Hereinafter, specific details for implementing the present invention will be explained with reference to the drawings.

[0021] In addition, in describing the present invention, identical parts are denoted by the same reference numerals, and their repeated description is omitted.

[0022] First, examples of the configuration of the overall system for implementing the present invention will be described with reference to FIG. 1.

[0023] As shown in FIG. 1(a), the multi-view compression method for three-dimensional synthesis according to the present invention can be implemented as a program system on a computer terminal (10) that compresses multi-view images.

[0024] That is, the compression system (30) can be implemented as a program system on a computer terminal (10), such as a PC, smartphone, tablet PC, or MP3 terminal. In particular, the compression system may be composed of a program system or a mobile application (or application, app) and installed and executed on the computer terminal (10). The compression system (30) provides a service for compressing multi-view images using the hardware or software resources of the computer terminal (10).

[0025] In addition, as another embodiment, as shown in FIG. 1(b), the compression system (30) may be configured and executed as a server-client system consisting of a compression client (30a) and a compression server (30b) on a computer terminal (10). That is, the compression system (30) may be configured as an online system on a network (80). In this case, it may be configured to include a database (40) for storing necessary data.

[0026] Meanwhile, the compression client (30a) and the compression server (30b) can be implemented according to the usual client and server configuration method. That is, the functions of the entire system can be distributed according to the performance of the client or the amount of communication with the server. In other words, they can be implemented in various forms of distribution depending on the server-client configuration method. Additionally, the compression server (30b) can be configured as a cloud server with multiple servers distributed therein.

[0027] Additionally, the computer terminal (10) on which the compression client (30a) is installed may be another online server. That is, the compression server (30b) may provide compression services to another online server or site.

[0028] Alternatively, as another embodiment, the system according to the present invention may be implemented as a single dedicated IC chip configured as a microprogram and driven by a microprocessor, or as a single electronic circuit such as an ASIC (Application-Specific Integrated Circuit). That is, it may be implemented in the form of software, an FPGA chip, or an electronic circuit composed of multiple circuit elements. Other possible forms may also be implemented.

[0030] The MV-HEVC encoder used in the present invention will be explained with reference to FIG. 2.

[0031] MV-HEVC (Multiview High Efficiency Video Coding) is an extension technology of the HEVC (H.265) standard designed to efficiently compress and transmit multiview video data [Non-patent Literature 7-11]. Figure 2 shows a conceptual diagram of an MV-HEVC encoder.

[0032] Based on inter-view prediction, this technology eliminates spatial redundancy by referencing data from other viewpoints, in addition to the intra- and inter-view predictions used in existing single-viewpoint methods. This minimizes redundancy in multi-viewpoint data and enhances compression efficiency.

[0033] MV-HEVC performs encoding and decoding by dividing data into a Base View and an Extended View. The Base View can be encoded and decoded independently and does not depend on other Views. On the other hand, the Extended View references the decoded data from the Base View and proceeds with encoding by additionally utilizing data from neighboring Views. In particular, the Extended View references data with the same Picture Order Count (POC) and uses Disparity Compensated Prediction.

[0034] Disparity compensation prediction reduces data redundancy and improves compression efficiency by selecting reference data based on disparity vectors, which represent the relative difference between time points. This process requires additional buffer space and management systems compared to conventional single-time-point-based encoding.

[0035] For efficient encoding and decoding, MV-HEVC manages complex reference structures using a high-level syntax called the Video Parameter Set (VPS). The VPS defines the number of encoded time points, the ID of each time point, and reference relationships between time points, thereby supporting the decoder in processing data in the correct order. Through this, MV-HEVC effectively manages the complex reference structures of multi-time point video data.

[0036] In the present invention, the inter-time prediction and encoding techniques of MV-HEVC are utilized to quantitatively analyze quality according to compression strength, and the impact of the compressed dataset on the performance of a new deep learning-based time-time generation model is evaluated.

[0037] Next, the 4D-Gaussian splatting used in the present invention will be explained with reference to FIG. 3.

[0038] In the present invention, a 4D Gaussian splatting [Non-patent Literature 4-6] model is adopted as a 3D generating deep learning model.

[0039] 4D Gaussian Splatting is a neural rendering technique that represents a 3D scene by applying a Gaussian distribution to each point based on 3D point cloud data, and dynamically renders it by reflecting spatial and temporal changes. This model learns the complex structure and temporal changes of the 3D scene and is used to generate images from new viewpoints.

[0040] 4D Gaussian Splatting consists of three stages: 3D point cloud and Gaussian generation, a Gaussian transformation field network, and rendering. The 3D point cloud used in 4D Gaussian Splatting is generated using the SfM technique. A 3D world coordinate system is defined based on specific point matching between 2D images, and a point cloud is generated through this. The generated point cloud is set as the center of the initial Gaussian distribution, and each Gaussian is assigned attributes such as magnitude, orientation, and color to represent the initial state of the scene. These initial Gaussians learn the dynamic scene through a transformation process over time.

[0041] The network for training the Gaussian deformation field network consists of a spatial-temporal structure encoder and a Gaussian deformation decoder for predicting each 3D deformation. Adjacent 3D Gaussians always share similar spatial and temporal information. To model the features of these 3D Gaussians, the spatial-temporal structure encoder consists of multi-resolution 2D planar modules and multilayer perceptrons.

[0042] The encoder receives 3D Gaussian and temporal information as input and utilizes six multi-resolution 2D planar modules to decompose and represent the spatial and temporal information of the 3D Gaussian. The planar modules used consist of (x,y), (x,z), (y,z), (x,t), (y,t), and (z,t), each representing a specific spatial and temporal relationship. The average position of each 3D Gaussian is represented by (x,y,z), and each planar module encodes the features of the corresponding region.

[0043] High-resolution planar modules are generated based on the base resolution and upsampling scale of the voxels. These planes compress the spatial information of the 3D Gaussian into 2D, while temporal information is indirectly integrated through the interpolation process of each plane. In the voxel feature calculation process, linear interpolation is used to calculate the voxel values ​​located at the grid vertices. This extracts detailed features of the 3D Gaussian. All calculated features are combined through a multilayer perceptron to output the final space-temporal features.

[0044] After all features of the 3D Gaussian are encoded, the deformation values ​​for position, rotation, and scale are calculated respectively through a multi-head Gaussian deformation decoder. The decoder consists of independent multi-layer perceptrons, which predict the deformations for position, rotation, and scale. Each deformation value is added to the original value to generate the deformed position, rotation, and scale values, and based on this, the final deformed 3D Gaussian is defined.

[0045] The deformed 3D Gaussian includes deformed position, scale, and rotation values, as well as the Gaussian's variance and color information. The multi-head Gaussian deformation decoder calculates each deformation independently and processes the deformation process of the entire Gaussian integrally.

[0046] A new viewpoint image is generated by performing differential splatting operations based on a modified 3D Gaussian. Differential splatting projects the distribution of the modified Gaussian onto a 2D image plane to calculate pixel values ​​by reflecting the characteristics of each Gaussian. This process considers not only the modified position, scale, and rotation values ​​but also dispersion and color information, thereby efficiently rendering a high-quality new viewpoint image.

[0047] Next, the configuration of a multi-view compression system for three-dimensional synthesis according to an embodiment of the present invention will be explained with reference to FIG. 4.

[0048] In the present invention, a multi-viewpoint image is encoded into MV-HEVC and then decoded to be used as a dataset. A 4D Gaussian splatting model is trained using the decoded dataset, and a new viewpoint image is generated through the trained model to analyze the impact of the compressed dataset on the performance of the deep learning model that generates the new viewpoint.

[0049] Figure 4 shows an overview of the overall structure to be tested in the present invention.

[0050] The first step of the network is the compression of multi-view images. The input multi-view images are grouped into three views, and each group is compressed through an MV-HEVC encoder. The result of the compression is generated as a bitstream, which is then restored through an MV-HEVC decoder.

[0051] The restored image is used as training data for a 4D Gaussian splatting model. The trained 4D Gaussian splatting model generates an image from a new camera viewpoint to be observed in the present invention through inference.

[0052] This is the subject of observation in the present invention. The newly generated viewpoint image is compared with the results of training using the uncompressed original dataset. Through this comparison, the impact of the compressed dataset on the performance of the 3D reconstruction and viewpoint generation deep learning model is analyzed.

[0053] Next, an MV-HEVC-based multi-view image compression method according to an embodiment of the present invention will be described with reference to FIG. 4.

[0054] MV-HEVC encodes multi-viewpoint data by utilizing a reference structure between various viewpoints. The reference structure may vary depending on the position of the reference viewpoint and the relationship between viewpoints, and in this invention, a PIP structure is adopted in which an intermediate viewpoint is set as the base viewpoint for three viewpoint images.

[0055] Figure 4 illustrates a PIP reference structure, in which an intermediate time point is independently encoded as a base time point, and the left and right time points are encoded by referencing the intermediate time point. The intermediate time point (Time Point 1) is encoded first, and subsequently, the left time point (Time Point 0) and the right time point (Time Point 2) are encoded and decoded through inter-view prediction by referencing the intermediate time point.

[0056] In this invention, experiments were conducted by grouping 21 viewpoint data into three viewpoints with similar camera position relationships. MV-HEVC encoding was performed on 6 groups excluding the observation target group out of a total of 7 groups, and in each group, the middle viewpoint was set as the primary viewpoint, and a PIP structure was applied between the left and right viewpoints based on this.

[0057] As shown in Fig. 4, the base view is encoded into a hierarchical reference structure with a Group of Pictures (GOP) size of 8. The I-picture generated through intra-frame prediction is used as the Key Picture, and during the encoding process of the extended view, encoding proceeds by referencing the Key Picture of the base view or the previously encoded Key Picture of the extended view. The Non-Key Picture is encoded by referencing an image with the same Picture Order Count (POC) following the existing hierarchical reference structure. The multi-view images encoded in this way are utilized as training data for a 3D generative deep learning model and are used to evaluate the impact of compression performance on the deep learning model's performance.

[0058] Next, a 4D-Gaussian Splatting-based learning and viewpoint generation method according to an embodiment of the present invention will be described.

[0059] In the present invention, a 4D Gaussian splatting structure was adopted to generate an image from a desired new viewpoint using compressed multi-viewpoint image data. 4D Gaussian splatting consists of three stages: 3D point cloud generation, Gaussian generation and transformation, and rendering. In the present invention, this structure was adopted while modifying some model structures to analyze the impact of the compressed dataset on the performance of a 3D reconstruction and viewpoint generation deep learning model.

[0060] The 3D point cloud used in conventional 4D Gaussian splatting is generated through the SfM-based SIFT algorithm. The SIFT algorithm extracts feature points from each 2D image in a multi-view image and generates 2D keypoints by matching identical feature points between images captured from different viewpoints. Subsequently, the intrinsic and extrinsic parameters of the camera are estimated through camera calibration. These generated 2D keypoints are re-projected into 3D space using a camera matrix, and the pixel coordinates of the 2D images are converted into a 3D world coordinate system. As a result, the generated 3D world keypoints form a point cloud.

[0061] In the present invention, a multi-viewpoint dataset including 21 viewpoints was used, and three of these viewpoints were selected as observation viewpoints. Point clouds and camera world coordinate systems were obtained from the 21 viewpoints generated through the SfM-based SIFT algorithm, and camera information corresponding to three of the observation viewpoints was extracted.

[0062] In the Gaussian deformation field network of 4D Gaussian splatting, Gaussian generation and training were performed using point cloud information for 18 time points, excluding the observation point, as input. During the inference process, fixed images were generated from three new time points using camera information from the observation point that had been excluded. These three inferred time points represent the performance of the 4D Gaussian splatting model, and the performance of the deep learning model was evaluated through PSNR by comparing the generated images with the original images. The modified structure of the present invention provides an environment that can effectively analyze the impact of a compressed dataset on model performance.

[0063] The present invention evaluated the performance of a new viewpoint generation deep learning model based on 4D Gaussian splatting using a multi-viewpoint image dataset compressed with MV-HEVC. Specifically, through experimental results, the relationship between changes in the quality of the compressed dataset and the performance of the deep learning model is quantitatively analyzed.

[0064] Experimental results showed that even when using a compressed dataset as training data, the performance of the deep learning model did not differ significantly compared to the results using the original dataset. This suggests that the deep learning model can effectively utilize key image information even as the compression strength increases.

[0065] In particular, the PSNR and SSIM values ​​of the new viewpoint images generated from the inference results of a model trained on data compressed according to QP values ​​maintained a level similar to the reference performance (Ground Truth). These results demonstrate that the viewpoint generation performance of a deep learning model can be maintained even when highly compressed data is used as a training dataset.

[0066] This presents a practical possibility of compressing, storing, and managing high-capacity training datasets while maintaining the performance of existing deep learning models.

[0068] Although the invention made by the inventor has been specifically described above according to the embodiments, the invention is not limited to the above embodiments and can be modified in various ways without departing from the gist thereof. Explanation of the symbols

[0069] 10 : Computer terminal 30 : Compression System 30a : Compression Client 30b : Compression server 40 : Database 80 : Network

Claims

Claim 1 A multi-view compression system for 3D synthesis, characterized by comprising: (a) a step of encoding multi-view data using an MV-HEVC encoder designed to compress and transmit multi-view video data; (b) a step of utilizing the encoded multi-view image as training data for a 3D generating deep learning model; (c) a step of generating an image at a new viewpoint using the compressed multi-view image data; and (d) a step of evaluating the effect of compression performance on the performance of a deep learning model. Claim 2 A multi-view compression method and system for 3D synthesis according to claim 1, characterized in that, in step (c), a new image is generated by adopting a 4D Gaussian splatting structure, and the generation is performed in three stages: 3D point cloud generation, Gaussian generation and transformation, and rendering. Claim 3 A multi-view compression method and system for 3D synthesis according to claim 2, wherein in step (c), feature points of each 2D image are extracted from a multi-view image, 2D key points are generated by matching identical feature points between images taken at different viewpoints, intrinsic and extrinsic parameters of the camera are estimated through camera correction, the generated 2D key points are re-projected into 3D space using a camera matrix, and pixel coordinates of the 2D image are converted to a 3D world coordinate system to form a point cloud.

Images