Lightweight three-dimensional reconstruction method and system based on three-dimensional Gaussian model

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to a lightweight three-dimensional reconstruction method and system based on a three-dimensional Gaussian model, and the method comprises the steps: constructing the three-dimensional Gaussian of a target scene based on a multi-view original image of the target scene through employing a 3DGS algorithm; identifying the minimum resolution of each Gaussian primitive in the three-dimensional Gaussian, setting a spherical region according to the minimum resolution, taking the intersection result of the spherical region and the Gaussian primitive as a region evaluation score, and trimming the redundancy region geometry in the three-dimensional Gaussian in combination with the opacity; and taking the maximum zoom scale of the Gaussian primitive as a spatial density proxy variable, and dynamically distributing spherical harmonic coefficient orders according to a local spatial density proxy variable so as to meet the rendering requirements of different three-dimensional Gaussian geometric regions. On the premise that the image synthesis quality is not lost, the model scale and the transmission overhead can be remarkably compressed, the resource constraint condition of the edge computing equipment can be effectively adapted, and the higher training speed and the better rendering effect are achieved.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a lightweight 3D reconstruction method and system based on a 3D Gaussian model. Background Technology

[0002] 3D reconstruction is a key technology for representing virtual reality in the real world, providing accurate spatial dimensions, shape, and positional information, and laying a solid foundation for its upper-level application—novel view synthesis (NVS). Novel view synthesis (NVS) aims to generate new 3D scenes or views using sparse images or videos with known camera parameters. This field has made significant progress over the past 25 years and has been widely applied in virtual / augmented reality, digital twins, and autonomous driving. 3DGS (3D Gaussian Splatting) is a novel 3D reconstruction and real-time rendering technology based on 3D Gaussian representation, primarily used in digital twins, target navigation, and novel view synthesis. In recent years, Neural Radiance Fields (NeRFs) and 3DGS have emerged as multi-view... Figure 3 The core paradigms of dimensional reconstruction have jointly driven the evolution of new perspective synthesis technology towards high fidelity and real-time performance. However, these two methods still face different challenges.

[0003] NeRF is an implicit 3D scene representation method based on Multi-Layer Perceptron (MLP), achieving high-quality novel perspective synthesis through learnable continuous radiative fields. However, due to the high cost of MLP optimization, real-time rendering is usually not achievable on handheld devices or low-end GPUs, limiting its use in practical scenarios. To address these issues, researchers have proposed various optimization methods based on voxels, hash grids, and Neural LightFields (NeLF), expanding the possibilities for NeRF. However, these methods still face the challenge of relatively slow speed due to frequent ray point sampling, or require a compromise between quality and speed. 3DGS, on the other hand, directly projects 3D Gaussian ellipsoids onto a 2D plane using GPU parallel rasterization, achieving real-time rendering speeds of >30 FPS (Frames Per Second). However, it requires a large number of Gaussian ellipsoids to maintain high fidelity during rendering. An optimized large-scale scene may consist of millions of Gaussian distributions, and each point and its attributes need to be stored independently, thus requiring significant storage space and incurring high costs.

[0004] To address these issues, domestic and international research primarily focuses on two objectives: first, reducing the number of Gaussian points required for scene representation, i.e., pruning unimportant Gaussians; and second, compressing the spatial usage of Gaussian attributes. Gaussian pruning mainly involves evaluating the scene contribution of each 3D Gaussian, identifying and removing primitives with little impact on the overall scene by setting thresholds or using specific algorithms, thus simplifying the model. Attribute compression typically refers to mapping Gaussian parameters to a lower-dimensional representation space, capturing key information, eliminating redundancy, and optimizing the overall model's footprint by reducing the space required to store attributes without sacrificing the importance of individual Gaussians. For example, when dealing with targets primarily composed of diffuse materials, spherical harmonic functions can represent view-dependent effects without requiring high precision or a large number of coefficients. Therefore, there is an urgent need for a 3D reconstruction method that synergistically optimizes storage efficiency and visual fidelity to meet the application needs of low-computing-power platforms such as mobile devices and edge devices. Summary of the Invention

[0005] To address the balance between compression ratio and data fidelity in 3D reconstruction models, this invention provides a lightweight 3D reconstruction method and system based on a 3D Gaussian model. By establishing a Gaussian trimming and SH coefficient adjustment framework, it is applicable to model optimization on low-computing-power platforms such as mobile devices.

[0006] According to the design scheme provided by this invention, on the one hand, a lightweight 3D reconstruction method based on a 3D Gaussian model is provided, comprising:

[0007] Based on multi-view original images of the target scene, a 3D Gaussian structure of the target scene is constructed using the 3DGS algorithm;

[0008] Identify the minimum resolution of each Gaussian primitive in a 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. Combine opacity to trim redundant region geometry in the 3D Gaussian.

[0009] The maximum scaling scale of Gaussian elements is used as a surrogate variable for spatial density. The order of spherical harmonic coefficients is dynamically allocated according to the surrogate variable for local spatial density to meet the rendering requirements of different 3D Gaussian geometric regions.

[0010] As a lightweight 3D reconstruction method based on a 3D Gaussian model, the present invention further includes constructing a 3D Gaussian model of the target scene using the 3DGS algorithm, comprising:

[0011] The original multi-view images are initialized to obtain point clouds and camera pose information. The initialization includes, but is not limited to: sparse reconstruction using structure for motion recovery (SfM) technology, obtaining point clouds using mono / binocular depth estimation, or obtaining point clouds by randomly initializing Gaussian positions.

[0012] A Gaussian model is initialized using point clouds, and a converged 3D Gaussian model is obtained through differentiable rendering and optimization. The differentiable rendering and optimization utilizes a cyclic iterative process to project the 3D Gaussian ellipsoid onto a 2D image plane through rasterization, and obtains the loss by comparing the raster-rasterized image with the real image. In adaptive densification, the Gaussian parameters are updated based on the loss through backpropagation and gradient.

[0013] As a lightweight 3D reconstruction method based on a 3D Gaussian model in this invention, further identifying the minimum resolution of each Gaussian primitive in the 3D Gaussian model includes:

[0014] For each Gaussian element, iterate through all views and filter out the set of valid viewpoints that contain the current Gaussian element;

[0015] A spatial analysis model is constructed based on the pixel projection area of ​​the nearest neighbor viewpoint. The minimum resolution of each primitive is obtained using the spatial analysis model. The spatial analysis model is used to obtain the minimum pixel coverage area of ​​each primitive based on the projection area of ​​the primitive in each view and the pixel diameter of the projection area.

[0016] As a lightweight 3D reconstruction method based on a 3D Gaussian model in this invention, a spherical region is further defined according to the minimum resolution, and the intersection result of the spherical region and the Gaussian elements is used as the region evaluation score, including:

[0017] Obtain the M nearest neighbor candidate primitives for each Gaussian primitive;

[0018] Using the current Gaussian primitive's geometric center as the origin, the radius of the spherical region is set using the pixel coverage area corresponding to the minimum resolution. The region is scaled along the main axis according to the radius of the spherical region, and the intersection test between the candidate primitive and the spherical region is performed. If the center of the candidate primitive is within the spherical region, it is considered to be intersecting. The number of intersecting candidate primitives is used as the redundancy evaluation score of the current Gaussian primitive.

[0019] As a lightweight 3D reconstruction method based on a 3D Gaussian model, this invention further includes, by combining opacity trimming of redundant region geometry in the 3D Gaussian model, the method further comprises:

[0020] The Gaussian members were ranked according to their evaluation scores;

[0021] An adaptive threshold for eliminating primitives is set based on the mean and standard deviation of all Gaussian primitive evaluation scores;

[0022] Primitives with evaluation scores greater than the adaptive threshold and primitive opacity less than the opacity filtering threshold will be deleted.

[0023] As a lightweight 3D reconstruction method based on a 3D Gaussian model in this invention, the method further includes dynamically allocating the order of spherical harmonic coefficients according to local spatial density proxy variables, comprising:

[0024] Based on the covariance matrix of the Gaussian elements and the largest eigenvalue of the covariance matrix, the scaling factor of each Gaussian element is calculated, and the maximum scaling factor of all Gaussian elements in the three axes is obtained. The maximum scaling factor of the Gaussian elements is used as a proxy variable for spatial density and normalized.

[0025] Extract the color attribute features of the geometry of each region in the 3D Gaussian. Based on the color attribute features, obtain the order of the spherical harmonic function coefficients to describe the color change characteristics of Gaussian elements in different directions in the scene. Based on the order of the spherical harmonic function coefficients and the spatial density proxy variable, dynamically adjust the pruning index of each region and the spherical harmonic coefficients of each order in the 3D Gaussian to adjust the rendering of sparse and dense regions in the 3D Gaussian.

[0026] As a lightweight 3D reconstruction method based on a 3D Gaussian model in this invention, further, the method dynamically adjusts the trimming indexes of each region of the 3D Gaussian model and each order of spherical harmonic coefficients according to the order of the spherical harmonic function coefficients and the spatial density proxy variable, including:

[0027] If the coefficients of the spherical harmonic function are of order 0, then the absolute values ​​of the Gaussian ellipsoidal primary color parameter tensor are taken element by element, and all Gaussian elements with absolute values ​​less than the threshold are marked by a mask, and all masked positions are zeroed.

[0028] If the coefficients of the spherical harmonic function are of other orders, the sensitivity is set according to the coefficients. The dynamic pruning threshold is set based on the sensitivity, threshold and normalized spatial density proxy variable. A dynamic mask is generated based on the dynamic pruning threshold and Gaussian ellipsoid parameter tensor. The positions of all masked positions are zeroed out in layers based on the dynamic mask.

[0029] Furthermore, this invention also provides a lightweight 3D reconstruction system based on a 3D Gaussian model, comprising: a Gaussian training module, a Gaussian trimming module, and a Gaussian adjustment module, wherein...

[0030] The Gaussian training module is used to construct a 3D Gaussian image of the target scene based on multi-view original images and using the 3DGS algorithm;

[0031] The Gaussian trimming module is used to identify the minimum resolution of each Gaussian primitive in a 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. It also trims redundant region geometry in the 3D Gaussian by combining opacity.

[0032] The Gaussian adjustment module is used to dynamically allocate the order of spherical harmonic coefficients based on the maximum scaling scale of Gaussian elements as a spatial density proxy variable, in order to meet the rendering requirements of different 3D Gaussian geometric regions.

[0033] The beneficial effects of this invention are:

[0034] This invention reduces the number of primitives more effectively by calculating the minimum spatial coverage of primitives on the projection plane and incorporating the geometric intersection in a specified evaluation area as a redundancy scoring mechanism. Furthermore, addressing the heterogeneity of scene detail distribution, it uses the maximum scaling scale of Gaussian primitives as a proxy variable for spatial density. By dynamically adjusting the order of spherical harmonic coefficients in different regions and dynamically allocating the order of spherical harmonic coefficients according to local spatial complexity, it optimizes training speed and achieves a precise balance between compression ratio and visual fidelity. This allows high-density areas to maintain viewpoint-related details while low-density areas use low-order compression, achieving a synergistic optimization of storage efficiency and visual fidelity. It is applicable to low-computing platforms such as mobile devices and edge devices. Experiments have verified that this solution can reduce redundant Gaussians by 50% during training, suppress geometric redundancy, and achieve faster training speed and better rendering effects. It better ensures timeliness when facing the computing power limitations of mobile devices and can be seamlessly integrated into existing 3DGS pipelines for efficient inference and deployment, offering significant deployment advantages. Attached image description:

[0035] Figure 1 This is a schematic diagram of a lightweight 3D reconstruction process based on a 3D Gaussian model in the embodiment.

[0036] Figure 2 This is a schematic diagram of the resolution-aware Gaussian trimming process in the embodiment;

[0037] Figure 3 This is a schematic diagram of the adaptive adjustment process of the spherical harmonic coefficients in conjunction with spatial density in the embodiment. Detailed implementation method:

[0038] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0039] Existing 3DGS compression methods have high computational resource requirements. Complex compression algorithms (especially those based on neural context modeling or entropy coding) demand extremely high GPU computing power and video memory, making them difficult to run on resource-constrained edge devices or mobile terminals. Many high compression ratio schemes have significant computational overhead in their decoding process, offsetting the original speed advantage of 3DGS. Furthermore, pursuing extremely high compression ratios (e.g., >30x) often results in unavoidable loss of visual quality, manifesting as blurred details, loss of high-frequency textures, color distortion, and artifacts in edge regions. To address the problem of excessively large 3DGS training models, this invention provides a lightweight 3D reconstruction method based on a 3D Gaussian model, specifically including the following:

[0040] S101. Construct a three-dimensional Gaussian image of the target scene based on multi-view original images and using the 3DGS algorithm;

[0041] S102. Identify the minimum resolution of each Gaussian primitive in the 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. Combine the opacity to trim the redundant region geometry in the 3D Gaussian.

[0042] S103. Using the maximum scaling scale of Gaussian elements as a spatial density proxy variable, the order of spherical harmonic coefficients is dynamically allocated according to the local spatial density proxy variable to meet the rendering requirements of different 3D Gaussian geometric regions.

[0043] like Figure 1 The Gaussian pruning and SH coefficient adjustment framework shown here is used to optimize models for mobile devices. The specific workflow can be summarized as follows: 1) Training based on multi-view images to obtain tens of thousands of Gaussian distributions to achieve a comprehensive representation of the scene; 2) Calculating the number of overlapping primitives within a spherical region defined by the minimum resolution around the Gaussian distribution to obtain an evaluation score, and periodically pruning redundant targets based on opacity; 3) Dynamically adjusting the rendering requirements of different regions based on spatial density, and processing the SH coefficients in stages to meet the detailed requirements of different scenes. Without sacrificing image synthesis quality, it can significantly compress model size and transmission overhead, achieving excellent performance on benchmark datasets and demonstrating significant deployment advantages on mobile hardware platforms. Its lightweight model architecture and low-power computing paradigm can effectively adapt to the resource constraints of edge computing devices. Compared to previous 3DGS compression technologies, it has faster training speed and better rendering effects, and is simple and pluggable, allowing seamless integration into existing 3DGS pipelines for efficient inference and deployment.

[0044] Specifically, constructing a 3D Gaussian representation of the target scene using the 3DGS algorithm can include:

[0045] The original multi-view images are initialized to obtain point clouds and camera pose information. The initialization includes, but is not limited to: sparse reconstruction using structure for motion recovery (SfM) technology, obtaining point clouds using mono / binocular depth estimation, or obtaining point clouds by randomly initializing Gaussian positions.

[0046] A Gaussian model is initialized using point clouds, and a converged 3D Gaussian model is obtained through differentiable rendering and optimization. The differentiable rendering and optimization utilizes a cyclic iterative process to project the 3D Gaussian ellipsoid onto a 2D image plane through rasterization, and obtains the loss by comparing the raster-rasterized image with the real image. In adaptive densification, the Gaussian parameters are updated based on the loss through backpropagation and gradient.

[0047] 3DGS is a point-based 3D scene representation method that explicitly represents the underlying geometry and appearance using anisotropic ellipsoids, demonstrating powerful capabilities in novel perspective compositing. 3DGS generates sparse point clouds from Structure from Motion (SfM) and initializes a set of Gaussian primitives carrying multiple attributes. It then optimizes the Gaussian parameters through differentiable rendering and backpropagation, dynamically adjusts the Gaussian distribution using adaptive density control, and finally utilizes GPU rasterization to achieve real-time rendering of novel perspectives.

[0048] Specifically, the minimum resolution for identifying each Gaussian primitive in a 3D Gaussian may include:

[0049] For each Gaussian element, iterate through all views and filter out the set of valid viewpoints that contain the current Gaussian element;

[0050] A spatial analysis model is constructed based on the pixel projection area of ​​the nearest neighbor viewpoint. The minimum resolution of each primitive is obtained using the spatial analysis model. The spatial analysis model is used to obtain the minimum pixel coverage area of ​​each primitive based on the projection area of ​​the primitive in each view and the pixel diameter of the projection area.

[0051] Specifically, a spherical region is set according to the minimum resolution, and the intersection result of the spherical region and the Gaussian unit is used as the region evaluation score. This can be designed to include:

[0052] Obtain the M nearest neighbor candidate primitives for each Gaussian primitive;

[0053] Using the current Gaussian primitive's geometric center as the origin, the radius of the spherical region is set using the pixel coverage area corresponding to the minimum resolution. The region is scaled along the main axis according to the radius of the spherical region, and the intersection test between the candidate primitive and the spherical region is performed. If the center of the candidate primitive is within the spherical region, it is considered to be intersecting. The number of intersecting candidate primitives is used as the redundancy evaluation score of the current Gaussian primitive.

[0054] Among them, combining opacity trimming of redundant region geometry in a 3D Gaussian can include:

[0055] The Gaussian members were ranked according to their evaluation scores;

[0056] An adaptive threshold for eliminating primitives is set based on the mean and standard deviation of all Gaussian primitive evaluation scores;

[0057] Primitives with evaluation scores greater than the adaptive threshold and primitive opacity less than the opacity filtering threshold will be deleted.

[0058] The original 3DGS directly uses opacity as the evaluation criterion for Gaussian contribution, but many redundant primitives are preserved. Therefore, this default strategy has room for improvement in controlling and optimizing the number of Gaussians. To solve this problem, it is necessary to reconsider the evaluation of Gaussian contribution.

[0059] To better perform primitive pruning, it is first necessary to identify redundant regions, i.e., spaces containing a large number of primitives where each unit contributes little to the final rendering result. The detail resolution of a target region by a single imaging device is limited by the preset sampling precision, and the representable spatial resolution gradually decreases with increasing observation distance. However, for any Gaussian region in the scene, a multi-resolution spatial sampling architecture can meet the detail requirements of different regions.

[0060] In this embodiment, for each primitive, by traversing all training views, a set of effective viewpoints containing that primitive is selected, and the pixel projection area (diameter denoted as l) of the nearest neighbor viewpoint is used as the basis for the calculation. min Constructing a spatial analysis model:

[0061] V valid ={v i |Proj(P,v i )∩View(v i )≠0} (1)

[0062]

[0063] In the formula, Proj(P, v) i ) is the primitive P in view v i The projection area in the View(v) i ) for view v i The viewport range, l min Diameter(Proj(P, v)) represents the pixel coverage diameter. i The pixel diameter is defined as follows: This represents the minimum pixel coverage area of ​​the primitive at this viewpoint, which needs to represent the highest resolution details. Simultaneously, to conservatively cover potentially redundant areas at the same detail level, the origin is taken as the geometric center of the primitive. min / 2 defines a spherical evaluation region with a radius of 2.

[0064]

[0065]

[0066] B(P)={x∈R 3 |||xc p ||2≤r p} (5)

[0067] In the formula, s axis Let λ be the scaling factor along the principal axis of the target primitive P, ∑p be the covariance matrix of the primitive P, and λ be the scaling factor along the principal axis of the target primitive P. max It is the largest eigenvalue of ∑p, representing the directional scale of the longest axis, c p This is the geometric center of the target primitive. Theoretically, this region only requires a small amount of Gaussian to fully characterize the scene. However, when the distribution density within the domain exceeds a certain threshold, it indicates that the representation accuracy of its geometric details has exceeded the discernible limit of the current viewpoint resolution, forming a redundant storage structure.

[0068] like Figure 2 As shown, the M nearest neighbor candidate primitives of the target primitive are found, and along their principal axis directions, the spherical region radius l is determined. min Scaling by 2 and performing an intersection test, if the center of the candidate primitive lies within the spherical evaluation region, it is considered an intersection:

[0069]

[0070] In the formula, P j As a candidate primitive, c pj The geometric center of the candidate primitives is used. The number of intersection points is used as the redundancy score, and the primitives are sorted according to the score. At the same time, to prevent excessive elimination of certain regions, primitives exceeding the adaptive threshold τ by 50% are filtered out, and primitives with low opacity are selected first.

[0071]

[0072] τ=(μ+κ*σ) (8)

[0073] P remove ={P|Score(P)>τ}∩{P|α p <α thres} (9)

[0074] In the formula, Score(P) is the local redundancy score, μ and σ are the mean and standard deviation of the redundancy scores of all primitives, respectively, k is the adjustment coefficient, and P remove Let a be the set of primitives to be deleted. p For the opacity of primitive P, a thresSet the opacity filtering threshold.

[0075] The proposed solution identifies the minimum resolution of each primitive through resolution-aware Gaussian pruning to define a spherical evaluation environment, and uses the intersection of this environment with the Gaussian primitives as the evaluation score, thus mitigating geometric redundancy at its source.

[0076] Specifically, the order of spherical harmonic coefficients can be dynamically assigned based on the local spatial density proxy variable, and may include:

[0077] Based on the covariance matrix of the Gaussian elements and the largest eigenvalue of the covariance matrix, the scaling factor of each Gaussian element is calculated, and the maximum scaling factor of all Gaussian elements in the three axes is obtained. The maximum scaling factor of the Gaussian elements is used as a proxy variable for spatial density and normalized.

[0078] Extract the color attribute features of the geometry of each region in the 3D Gaussian. Based on the color attribute features, obtain the order of the spherical harmonic function coefficients to describe the color change characteristics of Gaussian elements in different directions in the scene. Based on the order of the spherical harmonic function coefficients and the spatial density proxy variable, dynamically adjust the pruning index of each region and the spherical harmonic coefficients of each order in the 3D Gaussian to adjust the rendering of sparse and dense regions in the 3D Gaussian.

[0079] If the coefficients of the spherical harmonic function are of order 0, then the absolute values ​​of the Gaussian ellipsoidal primary color parameter tensor are taken element by element, and all Gaussian elements with absolute values ​​less than the threshold are marked by a mask, and all masked positions are zeroed.

[0080] If the coefficients of the spherical harmonic function are of other orders, the sensitivity is set according to the coefficients. The dynamic pruning threshold is set based on the sensitivity, threshold and normalized spatial density proxy variable. A dynamic mask is generated based on the dynamic pruning threshold and Gaussian ellipsoid parameter tensor. The positions of all masked positions are zeroed out in layers based on the dynamic mask.

[0081] In 3DGS, spherical harmonic coefficients (SHCs) are used to describe the color variation characteristics of Gaussian points in different directions within a scene. Their higher-order coefficients often correspond to high-frequency information of details. However, in real-world scenes, the higher-order components of many regions (such as smooth surfaces) may have small amplitudes. These low-amplitude coefficients contribute little to the final rendering result but cause redundant computation and storage overhead. Traditional methods use a global threshold to trim SHCs, ignoring the impact of scene spatial density distribution on detail requirements. To address this, in this embodiment, as... Figure 3 As shown, spherical harmonic coefficients are trimmed based on spatial density and order adaptation. The trimming index of spherical harmonic coefficients in each region and at each order is dynamically adjusted to achieve high-order detail suppression in sparse regions and detail preservation in dense regions, thereby achieving the effects of model compression and computational acceleration.

[0082] The 0th-order SH coefficients (DC components) describe the basic hue of the color. The mask dc is generated by directly taking the absolute value of each element of its parameter tensor features_dc. mask All elements smaller than the threshold are marked, and then zeroing is performed to sparsify the SH parameter and accelerate the process using matrix operations:

[0083]

[0084] DC mask =||(|f dc |≥τ) (11)

[0085] PruneDC(f dc ,τ)=f dc ⊙dc mask (12)

[0086] In the formula, f dc Let be the DC component of the Gaussian point, τ be the threshold, ⊙ denote element-wise multiplication, and ||() be the characteristic function that generates the binary mask. This process can effectively suppress the primary color components that contribute weakly to the rendering while maintaining the main color characteristics unchanged.

[0087] First to third-order SH coefficients more realistically depict high-frequency lighting details, such as specular highlights and shadow transitions. However, the number of coefficients increases with the square of the order, significantly increasing computational costs and potentially leading to overfitting or rendering jitter. Therefore, fine-grained control is necessary when combined with spatial density. The maximum scaling factor of Gaussian points is used as the spatial density index and normalized.

[0088]

[0089] In the formula, max-scale is the maximum scaling value of all Gaussian points along the three axes, ∈=10 -8 Used for numerical stability. A larger scaling factor indicates a sparser geometric distribution in the region, and a lower demand for high-frequency details. Dynamic thresholds are set for coefficients of orders 1 to 3.

[0090] T k =τ×k×(1+S norm (15)

[0091] In the formula, k∈{1,2,3} represents the sensitivity of higher-order coefficients to details, (1+s norm This enhances the representation capability of large-scale regions. After generating the threshold mask, a layered zeroing operation is performed to retain important coefficients while eliminating redundant parameters.

[0092] Mask k =||(|C k <T k |) (16)

[0093]

[0094] In the formula, C k Mask is the k-th order coefficient tensor. k The mask matrix generated for the k-th order coefficients.

[0095] Using scaling factors as an indicator of spatial density aims to achieve the effect of dynamically adjusting parameter thresholds in layers, shifting spherical harmonic function coefficients to lower orders, and suppressing attribute redundancy by identifying and pruning Gaussian points that contribute negligibly to the final rendered image, thereby producing a compact visual effect and reducing the amount of computation in the 3D reconstruction process. This allows 3DGS to meet the application goals of real-time and high frame rate while maintaining high-quality rendering.

[0096] Furthermore, based on the above method, this embodiment of the invention also provides a lightweight 3D reconstruction system based on a 3D Gaussian model, comprising: a Gaussian training module, a Gaussian trimming module, and a Gaussian adjustment module, wherein...

[0097] The Gaussian training module is used to construct a 3D Gaussian image of the target scene based on multi-view original images and using the 3DGS algorithm;

[0098] The Gaussian trimming module is used to identify the minimum resolution of each Gaussian primitive in a 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. It also trims redundant region geometry in the 3D Gaussian by combining opacity.

[0099] The Gaussian adjustment module is used to dynamically allocate the order of spherical harmonic coefficients based on the maximum scaling scale of Gaussian elements as a spatial density proxy variable, in order to meet the rendering requirements of different 3D Gaussian geometric regions.

[0100] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0102] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0103] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0104] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A lightweight 3D reconstruction method based on a 3D Gaussian model, characterized in that, Include: Based on multi-view original images of the target scene, a 3D Gaussian structure of the target scene is constructed using the 3DGS algorithm; Identify the minimum resolution of each Gaussian primitive in a 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. Combine opacity to trim redundant region geometry in the 3D Gaussian. The maximum scaling scale of Gaussian elements is used as a surrogate variable for spatial density. The order of spherical harmonic coefficients is dynamically allocated according to the surrogate variable for local spatial density to meet the rendering requirements of different 3D Gaussian geometric regions.

2. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 1, characterized in that, Constructing a 3D Gaussian representation of the target scene using the 3DGS algorithm, including: The original multi-view images are initialized to obtain point clouds and camera pose information. The initialization includes, but is not limited to: sparse reconstruction using structure for motion recovery (SfM) technology, obtaining point clouds using mono / binocular depth estimation, or obtaining point clouds by randomly initializing Gaussian positions. A Gaussian model is initialized using point clouds, and a converged 3D Gaussian model is obtained through differentiable rendering and optimization. The differentiable rendering and optimization utilizes a cyclic iterative process to project the 3D Gaussian ellipsoid onto a 2D image plane through rasterization, and obtains the loss by comparing the raster-rasterized image with the real image. In adaptive densification, the Gaussian parameters are updated based on the loss through backpropagation and gradient.

3. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 1, characterized in that, The minimum resolution for identifying each Gaussian primitive in a 3D Gaussian matrix includes: For each Gaussian element, iterate through all views and filter out the set of valid viewpoints that contain the current Gaussian element; A spatial analysis model is constructed based on the pixel projection area of ​​the nearest neighbor viewpoint. The minimum resolution of each primitive is obtained using the spatial analysis model. The spatial analysis model is used to obtain the minimum pixel coverage area of ​​each primitive based on the projection area of ​​the primitive in each view and the pixel diameter of the projection area.

4. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 1, characterized in that, A spherical region is defined based on the minimum resolution, and the intersection of the spherical region with Gaussian elements is used as the region evaluation score, including: Obtain the M nearest neighbor candidate primitives for each Gaussian primitive; Using the current Gaussian primitive's geometric center as the origin, the radius of the spherical region is set using the pixel coverage area corresponding to the minimum resolution. The region is scaled along the main axis according to the radius of the spherical region, and the intersection test between the candidate primitive and the spherical region is performed. If the center of the candidate primitive is within the spherical region, it is considered to be intersecting. The number of intersecting candidate primitives is used as the redundancy evaluation score of the current Gaussian primitive.

5. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 1 or 4, characterized in that, Combining opacity trimming with redundant region geometry in a 3D Gaussian, including: The Gaussian members were ranked according to their evaluation scores; An adaptive threshold for eliminating primitives is set based on the mean and standard deviation of all Gaussian primitive evaluation scores; Primitives with evaluation scores greater than the adaptive threshold and primitive opacity less than the opacity filtering threshold will be deleted.

6. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 1, characterized in that, The order of spherical harmonic coefficients is dynamically assigned based on the local spatial density proxy variable, including: Based on the covariance matrix of the Gaussian elements and the largest eigenvalue of the covariance matrix, the scaling factor of each Gaussian element is calculated, and the maximum scaling factor of all Gaussian elements in the three axes is obtained. The maximum scaling factor of the Gaussian elements is used as a proxy variable for spatial density and normalized. Extract the color attribute features of the geometry of each region in the 3D Gaussian. Based on the color attribute features, obtain the order of the spherical harmonic function coefficients to describe the color change characteristics of Gaussian elements in different directions in the scene. Based on the order of the spherical harmonic function coefficients and the spatial density proxy variable, dynamically adjust the pruning index of each region and the spherical harmonic coefficients of each order in the 3D Gaussian to adjust the rendering of sparse and dense regions in the 3D Gaussian.

7. The lightweight 3D reconstruction method based on a 3D Gaussian model according to claim 6, characterized in that, The pruning indices for each region and order of spherical harmonic coefficients in a 3D Gaussian are dynamically adjusted based on the surrogate variables of the spherical harmonic coefficient order and spatial density, including: If the coefficients of the spherical harmonic function are of order 0, then the absolute values ​​of the Gaussian ellipsoidal primary color parameter tensor are taken element by element, and all Gaussian elements with absolute values ​​less than the threshold are marked by a mask, and all masked positions are zeroed. If the coefficients of the spherical harmonic function are of other orders, the sensitivity is set according to the coefficients. The dynamic pruning threshold is set based on the sensitivity, threshold and normalized spatial density proxy variable. A dynamic mask is generated based on the dynamic pruning threshold and Gaussian ellipsoid parameter tensor. The positions of all masked positions are zeroed out in layers based on the dynamic mask.

8. A lightweight 3D reconstruction system based on a 3D Gaussian model, characterized in that, It includes: Gaussian training module, Gaussian trimming module, and Gaussian adjustment module, among which, The Gaussian training module is used to construct a 3D Gaussian image of the target scene based on multi-view original images and using the 3DGS algorithm; The Gaussian trimming module is used to identify the minimum resolution of each Gaussian primitive in a 3D Gaussian, set a spherical region based on the minimum resolution, and use the intersection result of the spherical region and the Gaussian primitive as the region evaluation score. It also trims redundant region geometry in the 3D Gaussian by combining opacity. The Gaussian adjustment module is used to dynamically allocate the order of spherical harmonic coefficients based on the maximum scaling scale of Gaussian elements as a spatial density proxy variable, in order to meet the rendering requirements of different 3D Gaussian geometric regions.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images