A sparse view three-dimensional scene reconstruction method, system, device and medium based on geometric prior and tensor structure prior

Abstract

The present application belongs to the technical field of computer vision and three-dimensional scene reconstruction, and discloses a sparse view three-dimensional scene reconstruction method, system, device and medium based on geometric prior and tensor structure prior, the reconstruction method comprising: obtaining camera parameters of a target view; under the camera parameters of the target view, rendering an optimized Gaussian scene representation into a target view image by using a three-dimensional Gaussian splatting rasterization rendering pipeline, thereby completing the reconstruction of the three-dimensional scene; the optimized Gaussian scene representation is obtained after optimizing the parameters and low-rank tensor factors of the three-dimensional Gaussian splatting scene representation according to a loss function. The present application realizes real-time rendering based on a three-dimensional Gaussian rasterization method, improves the rendering efficiency while ensuring the reconstruction quality, and has good engineering application value. By jointly optimizing the parameters and low-rank tensor factors of the initial Gaussian point cloud, the collaborative optimization between the explicit Gaussian representation and the implicit tensor representation is realized, which is beneficial to improving the three-dimensional scene reconstruction quality and the new view synthesis effect.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction and new view synthesis technology, specifically involving a sparse view 3D scene reconstruction method, system, device and medium based on geometric prior and tensor structure prior. Background Technology

[0002] 3D scene reconstruction aims to restore the geometric structure and appearance information of a scene from multi-view images and generate high-quality images from any viewpoint. It has wide applications in virtual reality, augmented reality, digital twins, autonomous driving, robot perception, and film and television production. In recent years, 3D Gaussian Splatting (3DGS) technology has gradually become an important research direction in the field of new view compositing due to its combination of high-quality rendering effects and real-time rendering capabilities.

[0003] Existing 3D Gaussian splashing methods typically utilize Structure from Motion (SfM) to recover camera parameters and sparse point clouds from multi-view images, and initialize a Gaussian model based on the sparse point cloud. Subsequently, 3D scene reconstruction is achieved by optimizing parameters such as the Gaussian model's position, scale, rotation, color, and opacity. However, these methods usually rely on a large number of input viewpoints to ensure scene coverage integrity and reconstruction accuracy. When the number of input viewpoints is limited, insufficient observation information can easily lead to problems such as incomplete recovery of scene geometry, missing unobserved areas, and decreased generalization ability of the reconstruction results.

[0004] To address the aforementioned issues, existing technologies primarily employ methods such as adding prior information or introducing regularization constraints to improve reconstruction quality under sparse viewpoint conditions. For example, some methods utilize additional generated images, depth information, or pre-trained models to provide auxiliary supervision, supplementing scene structural information; other methods suppress overfitting and improve model generalization ability through implicit representations of neural networks, feature constraints, or random deactivation mechanisms.

[0005] However, existing technologies still have the following shortcomings: On the one hand, the point clouds obtained by traditional SfM initialization are often too sparse under sparse viewpoint conditions, making it difficult to provide reliable geometric priors for unobserved areas, leading to problems such as structural fragmentation, holes, and artifacts in the reconstructed scene; on the other hand, most existing regularization methods constrain the feature space implicitly, lacking explicit modeling of scene structural information, making it difficult to effectively utilize the spatial correlation and structural continuity between Gaussians. Furthermore, as the number of Gaussians increases, the model is prone to overfitting, thereby reducing the geometric consistency and visual realism of the reconstruction results.

[0006] Therefore, how to fully utilize the prior geometric information of the scene under sparse perspective conditions, enhance the structural correlation and spatial continuity between Gaussian features, suppress model overfitting, and improve the quality of 3D scene reconstruction and rendering efficiency has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] This invention addresses the technical problems of existing sparse-view 3D scene reconstruction methods, such as incomplete recovery of scene geometry, poor reconstruction quality of unobserved areas, easy overfitting of models, and insufficient generalization ability of reconstruction results. It provides a sparse-view 3D scene reconstruction method, system, device, and medium based on geometric priors and tensor structure priors. This method can enhance the structural correlation and spatial continuity between Gaussian features, suppress model overfitting, and improve the reconstruction quality and rendering efficiency of 3D scenes.

[0008] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention proposes a sparse-view 3D scene reconstruction method based on geometric priors and tensor structure priors, comprising the following steps: Obtain camera parameters from the target's perspective; Under the camera parameters of the target viewpoint, the optimized Gaussian scene representation is rendered into an image of the target viewpoint using a 3D Gaussian splash rasterization rendering pipeline, thereby completing the reconstruction of the 3D scene; The optimized Gaussian scene representation is obtained by optimizing the parameters and low-rank tensor factor of the 3D Gaussian splash scene representation based on the loss function.

[0009] Furthermore, the optimized Gaussian scene representation is obtained through the following process: Obtain sparse view images of the scene to be reconstructed; Sparse viewpoint images are input into the VGGT model to perform dense geometric priors and obtain scene geometric prior information. Construct an initial Gaussian point cloud based on prior geometric information of the scene; The geometric and appearance properties of the initial Gaussian point cloud are mapped to the tensor representation space to construct the scene feature tensor; Perform vector-matrix decomposition on the scene feature tensor to obtain low-rank tensor factors that represent the geometric and appearance information of the scene. The low-rank tensor factor is constrained by total variational regularization. By jointly optimizing the parameters of the initial Gaussian point cloud and the low-rank tensor factor after total variational regularization constraint, the optimized Gaussian scene representation is obtained.

[0010] Furthermore, an initial Gaussian point cloud is constructed based on prior geometric information of the scene, including: The Gaussian center position is initialized based on the spatial position of each point in the midfield scenic cloud according to the scene geometric prior information. Initialize the Gaussian scaling parameters based on the local neighborhood information of the point cloud in the scene geometric prior information; Initialize the Gaussian color parameters and opacity parameters based on the scene appearance information in the scene geometry prior information; Based on the initialized Gaussian parameters, including Gaussian center position, Gaussian scale parameter, Gaussian color parameter, and opacity parameter, an initial Gaussian point cloud containing position, scale, rotation, color, and opacity attributes is constructed.

[0011] Furthermore, the geometric and appearance attributes of the initial Gaussian point cloud are mapped to a tensor representation space to construct a scene feature tensor, including: Map the initial Gaussian point cloud to a 3D regular mesh space; In a three-dimensional regular mesh space, geometric feature tensors and appearance feature tensors are constructed based on the geometric and appearance properties of the initial Gaussian point cloud. The geometric feature tensor and the appearance feature tensor are weighted and accumulated to generate the scene feature tensor.

[0012] Furthermore, a vector-matrix decomposition is performed on the scene feature tensor to obtain low-rank tensor factors representing the scene's geometric and appearance information, including: Construct multiple vector factors for the scene feature tensor along the spatial coordinate axes; Construct multiple matrix factors for the scene feature tensor on an orthogonal plane corresponding to the spatial coordinate axes; Multiple vector factors and multiple matrix factors constitute low-rank tensor factors that represent the geometric and appearance information of the scene.

[0013] Furthermore, total variational regularization is used to constrain the low-rank tensor factor, including: The scene feature tensor is reconstructed by the outer product of vector factors and matrix factors; Apply total variational regularization constraints to the scene feature tensor.

[0014] Furthermore, by jointly optimizing the parameters of the initial Gaussian point cloud and the low-rank tensor factor after total variational regularization constraints, the optimized Gaussian scene representation is obtained, including: The initial Gaussian point cloud is rendered using rasterization to obtain the rendered image; Update the Gaussian point cloud parameters based on the reconstruction loss between the rendered image and the sparse view image of the scene to be reconstructed; Update the low-rank tensor factor based on the reconstruction loss and the total variational regularization loss; Through iterative optimization, an optimized Gaussian scene representation is obtained; The reconstruction loss includes L1 loss and structural similarity loss.

[0015] Secondly, this invention proposes a sparse-view 3D scene reconstruction system based on geometric priors and tensor structure priors, comprising: The camera parameter acquisition module is used to acquire camera parameters from the target's viewpoint. The rendering module is used to render the optimized Gaussian scene representation into an image from the target viewpoint using a 3D Gaussian splash rasterization rendering pipeline, under the camera parameters of the target viewpoint, thereby completing the reconstruction of the 3D scene. The optimized Gaussian scene representation is obtained by optimizing the parameters and low-rank tensor factor of the 3D Gaussian splash scene representation based on the loss function.

[0016] Thirdly, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the above-mentioned sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors.

[0017] Fourthly, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned sparse perspective 3D scene reconstruction method based on geometric priors and tensor structure priors.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention achieves real-time rendering based on a 3D Gaussian rasterization method, improving rendering efficiency while ensuring reconstruction quality, and has significant engineering application value. By jointly optimizing the parameters of the initial Gaussian point cloud and the low-rank tensor factor, this invention achieves synergistic optimization between explicit Gaussian representation and implicit tensor representation, which is beneficial for improving the quality of 3D scene reconstruction and the effect of novel perspective synthesis.

[0019] Furthermore, this invention utilizes a pre-trained 3D base model to obtain prior geometric information of the scene and constructs an initial Gaussian point cloud based on this prior geometric information. This enables the provision of reasonable structural information for unobserved regions under sparse viewpoint conditions, improving the completeness of scene geometric structure reconstruction. Moreover, by fully utilizing both scene geometric and tensor structure prior information, it effectively suppresses overfitting under sparse viewpoint conditions, improves the accuracy of scene geometric structure reconstruction and the quality of unobserved region reconstruction, enhances the geometric consistency of the reconstruction results, and improves the efficiency of 3D scene reconstruction and rendering.

[0020] Furthermore, this invention maps the geometric and appearance attributes of Gaussian point clouds to a tensor representation space and uses vector-matrix decomposition to obtain low-rank tensor factors that characterize the geometric and appearance information of the scene. This enables explicit modeling of the structural relationships between Gaussian features, suppresses model overfitting, and improves the generalization ability of the reconstruction results.

[0021] Furthermore, this invention enhances the spatial continuity and local correlation of tensor features by applying total variational regularization constraints to tensor factors, thereby improving the reconstruction quality of unobserved regions and the consistency of scene structure.

[0022] This invention also proposes a sparse-view 3D scene reconstruction system, electronic device, and computer-readable storage medium based on geometric priors and tensor structure priors, which possess all the advantages of the above methods. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the sparse perspective 3D scene reconstruction method based on geometric prior and tensor structure prior in this invention. Figure 2 This is a flowchart of the sparse viewpoint 3D scene reconstruction method based on geometric prior and tensor structure prior in this invention. Figure 3 This is a schematic diagram of the sparse viewpoint 3D scene reconstruction system based on geometric priors and tensor structure priors in this invention. Figure 4 The radar chart shows the quantitative comparison results of the present invention and various reconstruction methods under different numbers of input views, where (a) is the result under 6 input views and (b) is the result under 12 input views. Figure 5 The reconstruction visualization results of this invention are compared using 12 input views; wherein, the first row is the reconstruction result of the chair model, the second row is the reconstruction result of the Lego model, the third row is the reconstruction result of the boat model, and the fourth row is the reconstruction result of the microphone model. Figure 6 This invention provides a comparison of reconstruction quality (PSNR) under different numbers of input views. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The sparse view based on geometry and structure prior enhancement of the present invention Figure 3 The 3D reconstruction method is applicable to multi-angle photographs captured by smartphones, action cameras, fixed-pose camera arrays, or other devices.

[0028] To address the common problems of overfitting, fragmented structure in occluded regions, and inability to balance geometric integrity with real-time rendering in existing 3D Gaussian sparse view reconstruction methods, this invention provides a sparse view 3D scene reconstruction method based on geometric and tensor structure priors. This method can suppress overfitting and improve the continuity of spatial features under extremely low input view conditions, while maintaining native 3DGS real-time rendering capabilities during the inference phase. The specific steps are as follows: 1) Obtain sparse view images (i.e., real images) of the scene to be reconstructed; 2) Input the sparse view image of the scene to be reconstructed into the pre-trained 3D base model (VisualGeometry Grounded Transformer, VGGT) to perform dense geometric prior and obtain scene geometric prior information, which includes the dense point cloud of the predicted scene and the corresponding camera pose information. 3) Convert the dense point cloud of the predicted scene in the scene geometric prior information into a 3D Gaussian splatting scene representation to obtain the initial Gaussian point cloud, specifically including: The Gaussian center position is initialized based on the spatial position of each point in the dense point cloud of the scene predicted from the scene's geometric prior information. The Gaussian scaling parameters are initialized based on the local neighborhood information of the dense point cloud of the predicted scene from the scene's geometric prior information. The Gaussian color parameters and opacity parameters are initialized based on the scene appearance information of the dense point cloud predicted from the scene geometric prior information. Based on the initialized Gaussian parameters (including the initialized Gaussian center position, Gaussian scale parameter, Gaussian color parameter, and opacity parameter) in the 3D Gaussian Splatting scene representation, an initial Gaussian point cloud containing position, scale, rotation, color, and opacity attributes is constructed.

[0029] 4) Map the geometric and appearance attributes of the initial Gaussian point cloud to a tensor representation space to construct a scene feature tensor. The scene feature tensor includes a geometric feature tensor and an appearance feature tensor. The geometric feature tensor is used to characterize the scale and rotation parameters of the Gaussian point cloud; the appearance feature tensor is used to characterize the color and opacity parameters of the Gaussian point cloud. Specific steps include: Map the initial Gaussian point cloud to a 3D regular mesh space; In a three-dimensional regular mesh space, geometric feature tensors and appearance feature tensors are constructed based on the geometric and appearance properties of the initial Gaussian point cloud. The geometric feature tensor and the appearance feature tensor are weighted and accumulated to generate the scene feature tensor.

[0030] 5) Perform vector-matrix decomposition on the scene feature tensor to obtain low-rank tensor factors representing the scene's geometric and appearance information. These low-rank tensor factors include multiple vector factors and multiple matrix factors. Specific steps include: Construct multiple vector factors for the scene feature tensor along the spatial coordinate axes; Construct multiple matrix factors for the scene feature tensor on an orthogonal plane corresponding to the spatial coordinate axes; Multiple vector factors and multiple matrix factors constitute low-rank tensor factors that represent the geometric and appearance information of the scene.

[0031] 6) Constrain the spatial continuity of the scene feature tensor through total variational regularization, including: Apply total variational regularization constraints to both vector factors and matrix factors; By constraining the feature variation amplitude between adjacent spatial locations through total variational regularization, the local correlation and spatial continuity (i.e., smoothness) of the scene feature tensor are improved.

[0032] 7) Jointly optimize the parameters of the initial Gaussian point cloud and the low-rank tensor factor after total variational regularization constraints to obtain the optimized Gaussian scene representation, including: The initial Gaussian point cloud is rendered using rasterization to obtain the rendered image; Update the Gaussian point cloud parameters based on the reconstruction loss between the rendered image and the real image; Update the low-rank tensor factor based on the reconstruction loss and the total variational regularization loss; Through iterative optimization, the final optimized Gaussian scene representation is obtained.

[0033] The reconstruction loss includes L1 loss and structural similarity loss.

[0034] 8) Given camera parameters for a target viewpoint, use the standard 3D-Gaussian Splatting rasterization rendering pipeline to render the optimized Gaussian scene representation as a target viewpoint image, thereby completing the reconstruction of the 3D scene.

[0035] Example 1 See Figure 1 and Figure 2 In this embodiment, sparse views based on geometry and structure prior enhancement are used. Figure 3 The 3D reconstruction method includes the following steps: 1) Obtain the set of sparse viewpoint images (i.e., real images) as input; 2) Geometric prior initialization step: The pre-trained 3D base model (Visual Geometry Grounded Transformer, VGGT) is used to perform forward inference on the sparse view image set of the input to obtain the dense point cloud and camera parameters (i.e. camera pose information) of the predicted scene.

[0036] 3) Convert the dense point cloud of the predicted scene into a 3D Gaussian splatting scene representation. For each 3D point in the 3D Gaussian splatting scene representation, initialize corresponding 3D Gaussian parameters, including Gaussian center position, scale parameter, rotation parameter, opacity parameter, and color parameter, thereby obtaining the initial Gaussian point cloud. :

[0037] in Indicates the first Gaussian primitives, This represents the Gaussian quantity.

[0038] By introducing the global geometric prior provided by VGGT, this invention can effectively alleviate the problems of sparse point clouds, missing geometric structures, and unstable initialization in traditional SfM under conditions of very few input viewpoints, and provide more accurate scene priors for subsequent Gaussian optimization.

[0039] 4) Tensor-structured modeling: Normalize the initial Gaussian point cloud to a regular space volume and discretize it into 128... 3 A regular voxel grid, based on trilinear interpolation and weighted summation of Gaussian properties in the grid neighborhood, is used to construct a scene feature tensor, i.e., a fourth-order feature tensor.

[0040] in, Represents the scene feature tensor. Represents the real number field. Indicates the first coordinate axis. Indicates the second coordinate axis. This represents the third coordinate axis; all three coordinate axes are orthogonal to each other. Indicates the feature dimension.

[0041] Arbitrary Gaussian center points are obtained by trilinear interpolation sampling of the tensor factors of the scene feature tensor. Corresponding geometric and appearance attributes:

[0042]

[0043] in, They respectively represent the corresponding number A Gaussian scale, rotation, color, and opacity property. This represents the trilinear interpolation operator. and Let represent the geometric (scale, rotation parameters) feature tensor and the appearance (color, opacity parameters) feature tensor, respectively. Using the above equation, a large number of independent 3D Gaussian parameters can be transformed into a shared tensor structure representation, thereby significantly enhancing scene structure consistency and reducing optimization difficulty.

[0044] 5) In order to achieve efficient scene representation, the present invention further uses the VM low-rank tensor decomposition method to perform vector-matrix decomposition on the scene feature tensor to obtain low-rank tensor factors that represent the geometric and appearance information of the scene. The low-rank tensor factors include multiple vector factors and multiple matrix factors.

[0045]

[0046]

[0047] In the formula This represents the outer product operation. Represents a set One of the elements, Represents element The corresponding one-dimensional geometric vector factor, Represents element The corresponding one-dimensional apparent vector factor, Representation and element Orthogonal planar two-dimensional geometric matrix factors Representation and element Orthogonal planar two-dimensional appearance matrix factors, footcode and Representing geometric features and apparent features This represents the number of factors, i.e., the rank in tensor decomposition.

[0048] 6) Joint optimization and regularization constraint steps: To ensure that the tensor representation has good spatial continuity and structural stability, total variation (TV) regularization constraints are introduced on the vector factors and matrix factors of the scene feature tensor to construct the TV regularization loss.

[0049] Among them, TV regular loss The construction method is as follows:

[0050] 7) Jointly optimize the tensor decomposition factor and all 3D Gaussian parameters to obtain the total loss function; Total loss function: ; in, For TV, the regularization weight hyperparameter, For color loss;

[0051] This represents the absolute error loss per pixel. For real images, To render the image, the initial Gaussian point cloud is rendered using rasterization to obtain the rendered image; For D-SSIM (Differentiable Structural Similarity) loss, for Loss weight hyperparameter, .

[0052] Furthermore, the Gaussian center position, scale parameters, rotation parameters, opacity parameters, color parameters, and the vector and matrix factors of the tensor are trained. After training and optimization iterations to a preset number of rounds, the intermediate auxiliary structure of the tensor is removed, and all geometric and appearance parameters of the Gaussian are fixed. Subsequent rendering uses only standard 3D Gaussian parameters and reuses the native 3DGS raster rendering pipeline. Furthermore, under different sparse input viewpoint configurations, the tensor mesh resolution and tensor rank are dynamically adjusted: the tensor rank is preferably set to 16, and the mesh resolution is preferably 512; the TV regularization weight hyperparameter... It is fixed at 0.05.

[0053] After training reaches the preset number of iterations, the optimized tensor representation and Gaussian scene representation are obtained. The tensor auxiliary structure (i.e. the optimized tensor representation) is stripped away, and all optimization results are solidified into the corresponding 3D Gaussian parameters.

[0054] 8) Real-time rendering output: Given camera parameters of a target viewpoint, the standard 3D-Gaussian Splatting rasterization rendering pipeline is used to render the optimized Gaussian scene representation as an image of the target viewpoint, thereby completing the reconstruction of the 3D scene.

[0055] Specifically, all Gaussian primitives are projected based on the target camera pose, and color accumulation and transparency fusion are performed using the rasterization rendering pipeline to generate the target viewpoint image. Since the tensor structure does not need to be preserved during the inference stage, only the standard Gaussian parameters are retained, so the native 3DGS real-time rendering framework can be fully reused to achieve real-time new viewpoint compositing while maintaining high reconstruction quality.

[0056] This invention addresses the geometric degradation, structural loss, and overfitting problems that easily occur in 3D scene reconstruction under sparse input viewpoint conditions. It proposes a sparse viewpoint prior that integrates geometric and structural priors. Figure 3 This 3D reconstruction framework achieves high-quality scene initialization by introducing global geometric priors provided by VGGT; establishes a global correlation between scene geometry and appearance information through tensor low-rank structured representation; enhances spatial continuity and structural consistency through TV regularization constraints; and improves Gaussian stability through a tensor-guided attribute inheritance mechanism. Experiments show that this invention can significantly improve reconstruction accuracy, texture fidelity, and new perspective synthesis quality under sparse viewpoint conditions, while maintaining the real-time rendering efficiency of native 3DGS, demonstrating good engineering application value.

[0057] To address the sparse view based on geometry and structure prior enhancement proposed in this invention... Figure 3 The 3D reconstruction method was validated by conducting the following experiments.

[0058] During the experiment, this invention uses a pre-trained 3D base model VGGT to generate the initial geometric prior of the scene, including dense point clouds and camera parameters, and initializes the 3D Gaussian Splatting representation.

[0059] In the tensor structured modeling stage, the scene is normalized to a bounded cubic space and discretized into a regular voxel mesh. In the experiment, the tensor mesh resolution was set to 512, the tensor decomposition rank to 16, and the TV regularization weights were set accordingly. Set to 0.05. Geometric feature tensors are used to encode scale and rotation parameters, while appearance feature tensors are used to encode color and opacity parameters. Both types of tensors are modeled using a vector-matrix (VM) low-rank decomposition method.

[0060] This invention implements all algorithms using the PyTorch framework and completes training and testing on an NVIDIA RTX 3090 graphics card. During training, the Adam optimizer is used for parameter updates, with the initial learning rate set to... The total loss function consists of color reconstruction loss and TV regularization loss:

[0061] The color loss is composed of a weighted average of L1 loss and D-SSIM loss.

[0062]

[0063] During the experiment, the tensor decomposition factor, as well as the position, scale, rotation, opacity, and color parameters of the Gaussian elements, were simultaneously optimized. For the Gaussian primitives added during the Gaussian adaptive densification process, their initial properties were automatically inherited through tensor field interpolation, thereby ensuring the stability and continuity of the training process.

[0064] Experiments have verified that this invention achieves excellent reconstruction results on multiple publicly available sparse view datasets. Compared to the original 3DGS method, this invention effectively alleviates overfitting and geometric degradation under sparse input view conditions, achieving significant improvements in evaluation metrics such as PSNR, SSIM, and LPIPS. Furthermore, the introduction of VGGT geometric prior significantly improves scene initialization quality, thereby reducing optimization oscillations in the early stages of training and accelerating model convergence.

[0065] Furthermore, this invention establishes global correlations between Gaussians through tensor low-rank structured representation, enabling the sharing and propagation of scene geometry and texture information within a spatial range. TV regularization constraints further enhance the continuity of the tensor field, effectively suppressing problems such as floating-point artifacts, texture drift, and local structural fragmentation that easily occur under sparse viewpoint conditions.

[0066] Furthermore, this invention introduces tensor-assisted structures during the training phase, while during the inference phase, the optimization results are solidified into standard Gaussian parameters, and all intermediate tensor structures are stripped away. Therefore, no additional tensor computation overhead is required during the rendering phase, allowing direct reuse of the native 3DGS rasterization rendering pipeline to achieve high-quality, real-time new perspective compositing.

[0067] A quantitative comparison was made between the present invention (TensorGS) and methods such as 3DGS, Mip-Splatting, 2DGS, SplatFields, and SparseNeRF.

[0068] As shown in Table 1, it can be observed that the present invention achieves optimal or near-optimal results in most test scenarios, especially under extremely sparse input viewpoint conditions, where it can obtain more accurate geometric structure restoration and higher quality texture detail reconstruction. Correspondingly... Figure 4 (a) and (b) and Figure 5 The visualization results demonstrate that the method of this invention achieves higher fidelity. It preserves complex scene features and produces more realistic output. Even in challenging areas, it maintains clean and accurate reconstruction.

[0069] Table 1 shows the comparison results of PSNR, SSIM, and LPIPS metrics for different methods on multiple public datasets.

[0070] from Figure 6 It can be observed that the proposed method consistently achieves better reconstruction performance than the original 3DGS and VGGT-initialized 3DGS under all view settings. The performance gain is particularly significant under extremely sparse view conditions, demonstrating the effectiveness of the proposed VGGT initialization and tensor feature modeling in stably optimizing and improving global structural consistency under insufficient observation.

[0071] The efficiency of the method of the present invention is compared with that of existing representative methods, including training time, GPU memory consumption and inference speed (FPS), as shown in Table 2. Although the method of the present invention introduces additional memory overhead due to its tensor-based representation, it maintains real-time rendering performance comparable to that of the original 3DGS and significantly improves reconstruction quality.

[0072] Table 2 Comparison of reconstruction efficiency between the present invention and different methods

[0073] Experimental results show that this invention, by fusing the geometric priors provided by VGGT with the tensor low-rank structure priors, can effectively improve sparse vision. Figure 3It improves the quality of scene reconstruction and enhances scene geometric integrity, texture realism, and consistency of new perspectives while maintaining real-time rendering efficiency. It has good engineering application value and promotion prospects.

[0074] Example 2 like Figure 3 As shown, the sparse view based on geometry and structure prior enhancement of the present invention Figure 3 3D reconstruction system, including: The camera parameter acquisition module is used to acquire camera parameters from the target's viewpoint. The rendering module is used to render the optimized Gaussian scene representation into an image from the target viewpoint using a 3D Gaussian splash rasterization rendering pipeline, under the camera parameters of the target viewpoint, thereby completing the reconstruction of the 3D scene. The optimized Gaussian scene representation is obtained by optimizing the parameters and low-rank tensor factor of the 3D Gaussian splash scene representation based on the loss function.

[0075] All relevant content of each step involved in the aforementioned embodiments of the sparse perspective 3D scene reconstruction method based on geometric priors and tensor structure priors can be referenced to the functional description of the corresponding functional module of the sparse perspective 3D scene reconstruction system based on geometric priors and tensor structure priors in the embodiments of the present invention, and will not be repeated here.

[0076] Example 3 In one embodiment of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors.

[0077] Example 4 In one embodiment of the present invention, a computer-readable storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory. Volatile memory, such as at least one disk storage medium. One or more instructions stored in the computer-readable storage medium can be loaded and executed by a processor to implement the sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors in the above embodiments.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A sparse-view 3D scene reconstruction method based on geometric priors and tensor structure priors, characterized in that, Includes the following steps: Obtain camera parameters from the target's perspective; Under the camera parameters of the target viewpoint, the optimized Gaussian scene representation is rendered into an image of the target viewpoint using a 3D Gaussian splash rasterization rendering pipeline, thereby completing the reconstruction of the 3D scene; The optimized Gaussian scene representation is obtained by optimizing the parameters and low-rank tensor factor of the 3D Gaussian splash scene representation based on the loss function.

2. The sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors according to claim 1, characterized in that, The optimized Gaussian scene representation is obtained through the following process: Obtain sparse view images of the scene to be reconstructed; Sparse viewpoint images are input into the VGGT model to perform dense geometric priors and obtain scene geometric prior information. Construct an initial Gaussian point cloud based on prior geometric information of the scene; The geometric and appearance properties of the initial Gaussian point cloud are mapped to the tensor representation space to construct the scene feature tensor; Perform vector-matrix decomposition on the scene feature tensor to obtain low-rank tensor factors that represent the geometric and appearance information of the scene. The low-rank tensor factor is constrained by total variational regularization. By jointly optimizing the parameters of the initial Gaussian point cloud and the low-rank tensor factor after total variational regularization constraint, the optimized Gaussian scene representation is obtained.

3. The sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors according to claim 2, characterized in that, Construct an initial Gaussian point cloud based on prior geometric information of the scene, including: The Gaussian center position is initialized based on the spatial position of each point in the midfield scenic cloud according to the scene geometric prior information. Initialize the Gaussian scaling parameters based on the local neighborhood information of the point cloud in the scene geometric prior information; Initialize the Gaussian color parameters and opacity parameters based on the scene appearance information in the scene geometry prior information; Based on the initialized Gaussian parameters, including Gaussian center position, Gaussian scale parameter, Gaussian color parameter, and opacity parameter, an initial Gaussian point cloud containing position, scale, rotation, color, and opacity attributes is constructed.

4. The sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors according to claim 2, characterized in that, The geometric and appearance properties of the initial Gaussian point cloud are mapped to a tensor representation space to construct a scene feature tensor, including: Map the initial Gaussian point cloud to a 3D regular mesh space; In a three-dimensional regular mesh space, geometric feature tensors and appearance feature tensors are constructed based on the geometric and appearance properties of the initial Gaussian point cloud. The geometric feature tensor and the appearance feature tensor are weighted and accumulated to generate the scene feature tensor.

5. The sparse viewpoint 3D scene reconstruction method based on geometric prior and tensor structure prior as described in claim 2, characterized in that, Perform vector-matrix decomposition on the scene feature tensor to obtain low-rank tensor factors representing the scene's geometric and appearance information, including: Construct multiple vector factors for the scene feature tensor along the spatial coordinate axes; Construct multiple matrix factors for the scene feature tensor on an orthogonal plane corresponding to the spatial coordinate axes; Multiple vector factors and multiple matrix factors constitute low-rank tensor factors that represent the geometric and appearance information of the scene.

6. The sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors according to claim 1, characterized in that, The low-rank tensor factor is constrained by total variational regularization, including: The scene feature tensor is reconstructed by the outer product of vector factors and matrix factors; Apply total variational regularization constraints to the scene feature tensor.

7. The sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors according to claim 2, characterized in that, By jointly optimizing the parameters of the initial Gaussian point cloud and the low-rank tensor factor after total variational regularization constraints, the optimized Gaussian scene representation is obtained, including: The initial Gaussian point cloud is rendered using rasterization to obtain the rendered image; Update the Gaussian point cloud parameters based on the reconstruction loss between the rendered image and the sparse view image of the scene to be reconstructed; Update the low-rank tensor factor based on the reconstruction loss and the total variational regularization loss; Through iterative optimization, an optimized Gaussian scene representation is obtained; The reconstruction loss includes L1 loss and structural similarity loss.

8. A sparse-view 3D scene reconstruction system based on geometric priors and tensor structure priors, comprising: The camera parameter acquisition module is used to acquire camera parameters from the target's viewpoint. The rendering module is used to render the optimized Gaussian scene representation into an image from the target viewpoint using a 3D Gaussian splash rasterization rendering pipeline, under the camera parameters of the target viewpoint, thereby completing the reconstruction of the 3D scene. The optimized Gaussian scene representation is obtained by optimizing the parameters and low-rank tensor factor of the 3D Gaussian splash scene representation based on the loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the sparse viewpoint 3D scene reconstruction method based on geometric priors and tensor structure priors as described in any one of claims 1 to 8.

Images