A transparent object three-dimensional Gaussian splashing reconstruction method and system based on efficient environment field modeling

Abstract

This invention belongs to the field of computer graphics and computer vision technology, and relates to a method and system for 3D Gaussian splash reconstruction of transparent objects based on efficient environmental field modeling. The method includes: acquiring multi-view images; constructing a geometric prior shell of the transparent object based on the multi-view images; generating a transparent Gaussian representation based on the geometric prior shell; constructing a lightweight voxel environmental field, providing the radiation information required for refraction and reflection through the lightweight voxel environmental field; and rendering the transparent object within a 3D Gaussian splash framework based on the transparent Gaussian representation and the radiation information to obtain a 3D representation of the transparent object. This invention maintains physically reasonable modeling of transparent refraction and reflection phenomena while reducing engineering complexity and computational overhead, and is suitable for geometric and appearance reconstruction, new perspective synthesis, and related physically realistic rendering scenarios of transparent or semi-transparent objects under multi-view image conditions.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and computer vision technology, specifically relating to a method and system for 3D reconstruction of transparent objects based on 3D Gaussian splashing and environmental field modeling. It is applicable to geometric and appearance reconstruction, new perspective synthesis, and related physically realistic rendering scenarios of transparent or semi-transparent objects under multi-view image conditions. Background Technology

[0002] With the development of multi-view geometry and deep learning, the technology for reconstructing 3D scenes based on multi-view images has become relatively mature. Traditional methods obtain sparse or dense point clouds through feature matching and triangulation, and then combine multi-view stereo (MVS) and surface reconstruction algorithms to generate mesh models. However, the above methods have certain limitations in terms of viewpoint continuity, the quality of new viewpoint synthesis, and the representation of complex materials. In recent years, implicit volume rendering methods, represented by Neural Radiance Field (NeRF), have achieved new viewpoint synthesis under the condition of only having multi-view RGB images and camera parameters by using a multi-layer perceptron (MLP) to model the radiance at each position and in the viewing direction in 3D space. NeRF obtains pixel colors by sampling and integrating along rays, and can better express complex lighting and geometric details, but its rendering and training speed is relatively slow. To improve efficiency, technical routes such as Instant-NGP, which are based on multi-resolution hash coding and sparse neural networks for acceleration, have emerged. These technologies improve the convergence speed and query efficiency of volume rendering through compact spatial coding. Although these methods have a clear advantage in speed, they are still mostly implicit field representations, the 3D structure is not explicit enough, and they are not yet perfect in modeling transparent materials.

[0003] 3D Gaussian Splatting (3DGS) is a class of explicit scene representation and differentiable rendering techniques proposed in recent years. This type of method approximates the scene using a set of 3D Gaussian functions with attributes such as position, covariance matrix, and color coefficients. Specifically, given multiple view images and camera parameters, by optimizing the Gaussian parameters, the error between the rendered image and the real image is minimized after sorting, projecting, and alpha-stacking these Gaussian functions in screen space using a rasterizer. Currently, 3DGS-based methods have been widely applied to tasks such as novel perspective compositing, opaque scene reconstruction, and dynamic scene modeling, achieving good results in scenes with opaque objects and diffuse materials.

[0004] In optical imaging, the appearance of transparent or translucent objects such as glass and liquids depends not only on their surface color but also strongly on refraction, reflection, and internal absorption and scattering. According to the principles of geometric optics, refraction can be described by Snell's law, reflection by Fresnel's equations, and light energy attenuation within the medium can be modeled by Beer-Lambert's law. In 3D reconstruction and new perspective synthesis tasks, obtaining a physically plausible transparent appearance requires comprehensive consideration of: the object's surface geometry and normal distribution; material parameters such as the refractive index (IOR), roughness, and absorption coefficient of the medium; and the environmental field or illumination distribution, which provides the color source for refracted and reflected light. Therefore, 3D reconstruction of transparent objects is generally considered more challenging than that of opaque objects.

[0005] Existing technologies still have the following shortcomings in terms of geometric modeling, lighting representation, computational efficiency, and training stability.

[0006] 1. Shortcomings of transparent object reconstruction methods based on implicit neural radiation fields.

[0007] (1) Limited ability to model lighting and environment makes it difficult to handle far-field and near-field effects simultaneously. Many transparent object reconstruction methods simplify the environment: one type only assumes that the lighting comes from an environment map at infinity, making it difficult to characterize the occlusion of refracted / reflected light, color crosstalk, and indirect light contribution from the table, walls, ground, and surrounding objects; another type attempts to explicitly model the environment and indirect light using additional networks, tensor decomposition, or buffers, but is often limited by network capacity and sampling cost—either it can only work under idealized environment assumptions, or it requires a large amount of sampling / inference to approximate the real effect, making it difficult to achieve both high fidelity and high efficiency. The observed color of a transparent object is not "the color of the object's surface itself," but is mainly determined by "what is seen" in the environment by the secondary light generated by reflection / refraction: the appearance of the same glass will change drastically under different backgrounds, different neighboring objects, and different occlusion relationships. If the environmental field (which can be understood as incident radiation / visibility at any spatial location and direction) is not modeled sufficiently, the color and occlusion of secondary rays will be systematically miscalculated, directly leading to misalignment of refractive textures, drift of reflected specular highlights, "form but not detail" transparent outlines, and incorrect pull on the gradient of geometry / material optimization. Ultimately, this manifests as difficulty in converging details and long-term stagnation of metrics.

[0008] (2) High computational cost, making it difficult to meet real-time or near-real-time requirements. Many existing transparent object reconstruction methods rely on NeRF and its variants, which simultaneously learn the density field, refractive index field or refractive direction field, and radiometric field in continuous space through MLP. Since dense sampling and multiple network forward inferences are required on each ray, the training and inference costs of these methods are very high, usually requiring tens of hours or even longer to complete a scene reconstruction, which is not suitable for interactive or real-time application scenarios.

[0009] (3) Insufficient interpretability of refraction / reflection physics, and details are prone to over-smoothing. In pursuit of learning stability, some methods directly use neural networks to predict the direction of refraction or the color of refracted light, implicitly embedding the effects of complex optical paths into the neural network parameters. Although this approach simplifies physical modeling, it brings two problems: First, the physical meaning of refraction and reflection is not clear enough, making it difficult to perform controllable editing or relighting rendering; second, neural networks tend to learn smooth refraction results, making it difficult to retain high-frequency refraction textures and fine specular details, resulting in quality depending on the neural network structure and loss design.

[0010] (4) Dependence on additional hardware or special shooting conditions. Some traditional transparent object reconstruction methods require special calibration patterns of light field cameras or manual annotation to restore the correspondence between light and background. The shooting process is complicated, which limits the application scope of the method on general multi-view datasets and ordinary devices.

[0011] 2. Shortcomings in lighting representation and indirect lighting modeling.

[0012] (1) Lighting is represented only by environment maps, ignoring secondary ray effects. Many methods for compositing or inverse rendering for new perspectives assume that all lighting comes from infinity and are uniformly represented by environment maps. These methods have difficulty in explicitly handling secondary rays caused by occlusion, reflection and refraction from nearby objects, resulting in a lack of realism in rendering results in complex scenes, especially in the refraction patterns and edge details of transparent objects.

[0013] (2) Indirect lighting has limited representational completeness and high computational cost. To more accurately model indirect lighting and mutual reflection, some methods explicitly integrate the incident radiation field in volume rendering, while others use multi-path MLP to jointly model the contributions of ambient light and near object light. Although these methods are physically more complete, they often require: A. a large amount of volume integral sampling, resulting in significant computational cost; B. neural networks only model outgoing radiation, making it difficult to directly query the incident light during reverse rendering; C. the use of low-frequency basis expansions such as spherical harmonic functions, making it difficult to accurately represent details of specular refraction and high-frequency reflection. Some works also use offline path tracing or offline light field probes to pre-calculate indirect lighting, which can obtain high-quality indirect illumination, but the rendering overhead is high and it is not suitable for dynamic images or interactive scenes.

[0014] 3. Shortcomings of the transparent object reconstruction method based on 3D Gaussian splashing.

[0015] (1) The overall framework process is cumbersome and computationally expensive, making it difficult to deploy efficiently on resource-constrained platforms. TransparentGS, within the 3DGS framework, achieves physical inverse rendering of transparent objects through modules such as transparent Gaussian units, delayed refraction, GaussProbe environment probes, and iterative queries. Although its training speed and rendering efficiency have been significantly improved compared to the pure NeRF method, the overall system still includes: A. a large number of geometric / material parameters of transparent Gaussian units; B. a set of environment field probes with huge memory overhead; C. an iterative algorithm for querying and updating probes. The system is still mainly geared towards desktop GPUs in terms of video memory and computing resources, and has not been specifically designed for resource-constrained devices such as mobile or embedded platforms.

[0016] (2) Ambient light is represented by probes, and the storage and computational overhead increases significantly with the scene size. TransparentGS uses GaussProbe to encode the local incident light field in 3D space. Although it can capture the illumination contribution of ambient light and nearby content at the same time, the number of probes, distribution density and number of parameters increase with the scene size, detail requirements and number of transparent objects. Even with the introduction of a depth-based iterative query strategy to reduce the number of probes queried per pixel, the ambient field part still consumes a large amount of video memory and computational complexity in large-scale scenes or multi-object scenarios.

[0017] In summary, existing 3D reconstruction techniques for transparent objects, especially those based on implicit neural radiation fields and TransparentGS-like 3DGS methods, still have significant shortcomings in terms of computational efficiency, environmental field representation overhead, and deployment on resource-constrained platforms. There is an urgent need for an environmental field representation with a simpler structure and lighter representation to achieve stable convergence under coarse geometric priors and easy cross-platform deployment for 3D reconstruction of transparent objects. Summary of the Invention

[0018] To address the problems of existing 3D Gaussian splashing methods based on implicit neural radiation fields and TransparentGS class in the reconstruction of transparent objects, such as strong geometric prior dependence, complex and resource-intensive representation of ambient lighting, and difficulty in efficient deployment on resource-constrained devices, this invention provides a 3D Gaussian splashing reconstruction method and system for transparent objects based on efficient ambient field modeling.

[0019] The technical solution adopted in this invention is as follows: A method for reconstructing 3D Gaussian splashes of transparent objects based on efficient environmental field modeling includes the following steps: Obtain multi-view images and construct a geometric prior shell for the transparent object based on the multi-view images; Generate a transparent Gaussian representation based on a geometric prior shell; Construct a lightweight voxel environment field to provide the radiation information required for refraction and reflection; Based on the transparent Gaussian representation and the radiation information, the transparent object is rendered within a three-dimensional Gaussian splashing framework to obtain a three-dimensional representation of the transparent object.

[0020] Furthermore, the step of constructing the geometric prior shell of the transparent object based on the multi-view images includes: Obtain the transparent region mask, the de-highlight image, and the original ground truth image from the multi-view image, and obtain the prior normal of the transparent region as the normal prior; Using the transparent region mask and the de-highlighted image, a supervisory map of the transparent region that retains only the de-highlighted image is constructed; In the geometry-first stage, differentiable rendering is used to obtain the current rendered image and rendering normals, and a loss function is constructed and optimized by combining the transparent region supervision map and the normal prior. Once the geometry priority phase converges, voxel fusion is performed using the learned geometric surfaces that possess cross-view depth continuity and normal consistency to construct a truncated signed distance field. The shell is obtained by extracting the zero level set from the truncated signed distance field as the reconstructed isosurface; A transparent initial point cloud is obtained by sampling on the shell, and the orientation of each point is initialized with the shell normal. At the same time, the normal direction scale is set to a fixed ratio of the tangent plane scale to form a thin shell, which serves as the geometric prior shell of the transparent object.

[0021] Furthermore, the transparent Gaussian representation includes: the Gaussian center position, a covariance matrix describing the shape and direction of the Gaussian, Gaussian opacity, Gaussian geometric normal, reflected color, absorption coefficient, transparent material mixing coefficient, refractive index, and the color of the transparent object itself.

[0022] Furthermore, the construction of the lightweight voxel environment field includes: Construct a voxel mesh and trim voxels according to their importance; The color features of the remaining voxels after cropping are vector-quantized, compressed, and post-processed to obtain a lightweight voxel environment field.

[0023] Furthermore, the step of cropping voxels based on voxel importance includes: calculating the importance of sampling points based on the cumulative transmittance and observation opacity during the ray rendering process, and allocating the voxel importance to neighboring voxels according to distance weight; adaptively determining the cropping threshold through a quantile method, and cropping voxels whose voxel importance is lower than the cropping threshold.

[0024] Furthermore, the vector quantization compression of the color features of the remaining voxels after cropping includes: training a global codebook using an importance-weighted clustering algorithm, so that each remaining voxel only stores an index pointing to the codebook.

[0025] Furthermore, the rendering of transparent objects within the 3D Gaussian splashing framework involves performing Mesh-Gaussian joint optimization within the 3D Gaussian splashing framework, including the following steps: Based on the equivalent hit point map and normal map obtained by rendering transparent Gaussian under multiple views as geometric clues, TSDF fusion is performed and the proxy mesh is periodically updated according to a preset interval. The updated proxy mesh is used for intersection and visibility determination of the secondary refraction path. The stable secondary ray path results are fed back to the transparent rendering and multi-view loss, thereby inversely constraining the geometric and material parameter updates of the transparent Gaussian, achieving stable geometric convergence and realistic secondary refraction.

[0026] Furthermore, during the execution of the Mesh-Gaussian joint optimization, the rendering of the transparent object specifically includes: Traverse the transparent Gaussian along the principal ray and obtain the equivalent hit point and equivalent normal of the principal ray by weighted averaging. Based on the line of sight, the equivalent normal, and the refractive index of the transparent Gaussian, calculate the reflection direction and refraction direction of the main interface. Starting from the equivalent hit point, a secondary ray is constructed along the refraction direction of the main interface. The intersection with the proxy mesh is used to obtain the exit point. The exit direction after passing through the medium is calculated by combining the proxy mesh normal. Radiation information is queried in the lightweight voxel environment field along the reflection direction and the emission direction, respectively, and the transparent rendering color is synthesized by combining the Fresnel coefficient.

[0027] Furthermore, during training, end-to-end optimization is performed based on multi-view supervision. The rendered image is compared with the real multi-view image and the loss is calculated. The loss includes reconstruction loss, normal consistency loss, mask loss and normal smoothing loss.

[0028] Furthermore, the normal consistency loss includes prior normal consistency loss and depth-derived normal consistency loss. When calculating the prior normal consistency loss, a normal mask is calculated based on the similarity between the prior normal and the rendered normal, and the normal mask is used to suppress erroneous normal priors.

[0029] Furthermore, when the optimization process converges or meets the preset termination condition, the trained transparent Gaussian 3D representation and the lightweight voxel ambient lighting model are output. During the new perspective rendering, the rendering results of the background environment and the transparent object are obtained respectively. The transparent object and the background environment are then fused to obtain the new perspective synthetic image.

[0030] A 3D Gaussian splash reconstruction system for transparent objects based on efficient environmental field modeling, comprising: The acquisition module is used to acquire multi-view images; The geometric prior shell generation module is used to construct the geometric prior shell of a transparent object based on multi-view images; A transparent Gaussian representation generation module is used to generate transparent Gaussian representations based on geometric prior shells; The voxel environment field construction module is used to construct lightweight voxel environment fields, which provide the radiation information required for refraction and reflection. The transparent object rendering module is used to render transparent objects within a three-dimensional Gaussian splashing framework based on the transparent Gaussian representation and the radiation information, thereby obtaining a three-dimensional representation of the transparent object.

[0031] This invention proposes a series of improvements within a three-dimensional Gaussian splashing framework, focusing on the geometric prior construction, ambient lighting representation, and overall framework scalability for the three-dimensional reconstruction of transparent objects. Through the combination of these techniques, this invention reduces engineering complexity and computational overhead while maintaining a physically sound model of transparent refraction and reflection phenomena. The beneficial technical effects resulting from these key points are described below.

[0032] Key Point 1: Transparent Geometry Training Optimization and Low-Memory-Constraining Secondary Refraction of Rays Based on TSDF Shell. Theoretically, this invention uses a proxy mesh as the geometric anchor point for the transparent medium boundary, enabling "geometric training optimization (TSDF fusion and mesh extraction)" and "transparent rendering optimization (Gaussian parameter learning)" to work collaboratively in a lightly coupled manner. While maintaining a lightweight overall framework, it achieves stable intersection of secondary ray paths, significantly improving the physical consistency and training stability of the refraction deformation of transparent objects. From an engineering perspective, this invention generates and updates the proxy mesh through periodic TSDF fusion, avoiding the training of additional implicit geometric networks and large-scale memory consumption. It balances implementation complexity, operational efficiency, and cross-platform deployment friendliness, effectively solving the problems of geometric instability of transparent objects, unrealistic secondary refraction, and excessive resource consumption in existing technologies.

[0033] Key Point 2: Lightweight Lighting Representation Based on Environment Field, Replacing Dense Probe Volumes. Theoretically, this invention transforms the discrete lighting representation, which originally relied on a 3D probe mesh, into a continuous environment field representation. This prevents the parameter scale of the environment representation from linearly increasing with scene spatial resolution and probe density, fundamentally alleviating the memory consumption and computational overhead problems caused by dense probes. In practical terms, while maintaining good representation of transparent refractive textures and mutual reflection details, this invention significantly reduces the peak memory usage and environment query time during the transparent reconstruction stage, making it easier to deploy transparent reconstruction functionality on resource-constrained devices (such as mobile GPUs and embedded platforms). By comparing experimental results with the open-source implementation of TransparentGS, the applicant observed that under similar scene settings, the environment representation implemented in this invention significantly alleviates the overall computational burden during the training and rendering stages while achieving similar rendering quality. This effectively solves the problems of high overhead and difficulty in expansion and deployment of environment field representations in existing technologies. Specific comparison items are shown in Table 2.

[0034] Key Point 3: Modular and Scalable Transparent 3DGS Reconstruction Framework Design. The modular design of this invention improves the maintainability and evolvability of the system, allowing developers to independently optimize and upgrade geometry, lighting, or material modules without disrupting the overall framework, thus accelerating algorithm iteration. Furthermore, the framework is highly compatible with existing opaque 3DGS pipelines, enabling business systems to incrementally integrate transparent modules on top of existing 3DGS systems, reducing engineering modification costs. It also provides a unified platform for the subsequent introduction of lightweight mobile technologies such as model distillation and parameter quantization. In summary, this invention, through structural simplification and modularization, achieves the natural extension and expansion of 3D Gaussian splashing in transparent object scenes, facilitating the transition of transparent object reconstruction technology from laboratory research to practical product deployment. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the steps of a method for reconstructing three-dimensional Gaussian splashes of transparent objects based on efficient environmental field modeling, according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the processing procedure of a three-dimensional Gaussian splash reconstruction method for transparent objects based on efficient environmental field modeling, according to another embodiment of the present invention.

[0037] Figure 3 It is an RGB image.

[0038] Figure 4 It is a transparent area mask.

[0039] Figure 5 It's a de-highlighting image.

[0040] Figure 6 It is the result of geometric reconstruction rendering.

[0041] Figure 7 It is a TSDF shell point cloud.

[0042] Figure 8 It is an image of a non-transparent object.

[0043] Figure 9 This is a block diagram of a transparent object 3D Gaussian splash reconstruction system based on efficient environmental field modeling, according to an embodiment of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0045] This invention aims to solve the following technical problems: (1) Existing 3D reconstruction methods for transparent objects typically rely on specific datasets or complex prior assumptions during the geometric initialization stage, resulting in insufficient generalization ability and poor robustness under different acquisition conditions and complex scenarios. This invention aims to solve the problem of how to construct a unified, lightweight, and highly transferable transparent geometric prior construction method to improve the stability and applicability of transparent object geometric initialization under multiple scenarios and multiple data distributions.

[0046] (2) Existing methods for modeling physical optical phenomena such as specular reflection and refraction of transparent objects typically rely on dense probes or high-dimensional ambient lighting representations, resulting in high memory consumption and computational complexity. These methods struggle to account for the combined effects of far-field ambient light and near-field objects on reflection and refraction, and are not suitable for deployment on platforms with limited computing resources. This invention aims to address how to provide a lightweight ambient field representation that can replace dense probes, while ensuring reasonable modeling of key optical phenomena, thereby significantly reducing the storage and computational overhead of ambient lighting representation.

[0047] This invention proposes a targeted technical solution for achieving 3D reconstruction of transparent objects within a 3D Gaussian splashing framework. This solution offers advantages such as a simple structure, lightweight environment representation, and stable training process. The core of this solution lies in introducing a unified geometric prior construction method and a TSDF (Truncated Signed Distance Function) proxy mesh generation mechanism combined with Mesh-Gaussian joint optimization, without introducing a heavy neural SDF training pipeline or consuming excessive GPU memory. Furthermore, it replaces dense probes with a lightweight environment field, thereby significantly reducing the overall system overhead while maintaining physical plausibility.

[0048] In terms of geometry, this invention first performs geometry-first joint optimization on multi-view images. By removing specular observations and using normal prior constraints, it suppresses pseudo-geometry introduced by reflection and refraction in transparent regions, and obtains stable depth / normal geometry output. Subsequently, the geometric output is truncated and fused with signed distance to construct a TSDF zero-equivalence shell of the transparent object as a geometric prior. Based on this, this invention adaptively samples the shell surface to generate an initial transparent point set, and establishes a local coordinate system with the shell normal. The normal direction is set to the thin shell scale to construct a Gaussian representation suitable for transparent rendering. During training, the proxy mesh is periodically updated at preset intervals or stages to continuously align it with the current Gaussian geometry and suppress local unstable structures such as "floating artifacts". Since the TSDF volume and mesh are mainly maintained by the CPU, and the video memory only handles depth rendering and a small amount of intersection data, the above method can ensure the intersection and stability of geometric boundaries while maintaining low video memory usage, thus meeting the goal of lightweight implementation.

[0049] (2) In terms of illumination representation, this invention constructs a lightweight voxel ambient illumination model, which is related to spatial position. and the direction of light The relevant incident radiation is modeled to replace the large-scale illumination probes that need to be deployed in the scene space in existing solutions. The voxel ambient lighting model provides a query interface in the form of a function. It can output radiance in real time at any location and in any direction. This allows for the simultaneous coverage of the contributions of both far-field ambient light and near-field scene content to reflected and refracted light. During the transparent rendering stage, the system works along the reflection / refraction direction... Structural rays The method involves sampling the rays in a finite number of steps and performing cumulative calculations based on the occlusion intensity. Early stopping is triggered when the transmission reaches the termination condition to ensure occlusion consistency and reduce invalid calculations. Furthermore, this invention employs an adaptive "contribution-based retention" selection method for voxel units, retaining only spatial units that contribute significantly to the incident color. It also uses a "dictionary table + index table" approach to store color-related storage, where multiple voxels share the same dictionary entry and only record the index. This significantly reduces the parameter scale and storage overhead of ambient lighting representation, making it suitable for resource-constrained devices and mobile platforms, while still effectively supporting the modeling of reflection and refraction lighting in complex transparent materials.

[0050] (3) In terms of overall framework design, this invention integrates modules such as TSDF geometric shell, transparent Gaussian representation, and environment field query into the 3D Gaussian splash pipeline in a loosely coupled manner. The interfaces between modules are clear and the replacement cost is low. On the one hand, this modular design allows the geometric prior to be replaced with surface or volume representations from other sources according to specific applications, and the environment field neural network can also choose different implementation forms according to hardware conditions. On the other hand, while maintaining the original real-time rendering advantages of 3DGS, this framework provides a physically reasonable, structurally simple, and scalable basic platform for the reconstruction of transparent objects, which facilitates the subsequent introduction of more complex optical models or lightweight strategies. Through the above design, this invention has formed a substantial difference from existing TransparentGS-like methods in terms of geometric prior construction, lightweight environment representation, and cross-platform deployment friendliness, which constitutes the main inventive point of this invention.

[0051] In one embodiment, the present invention provides a method for reconstructing three-dimensional Gaussian splashes of transparent objects based on efficient environmental field modeling, such as... Figure 1 As shown, it includes the following steps: Obtain multi-view images and construct a geometric prior shell for the transparent object based on the multi-view images; Generate a transparent Gaussian representation based on a geometric prior shell; Construct a lightweight voxel environment field to provide the radiation information required for refraction and reflection; Based on the transparent Gaussian representation and the radiation information, the transparent object is rendered within a three-dimensional Gaussian splashing framework to obtain a three-dimensional representation of the transparent object.

[0052] In one embodiment, constructing the geometric prior shell of the transparent object based on the multi-view images includes: Obtain the transparent region mask, the de-highlight image, and the original ground truth image from the multi-view image, and obtain the prior normal of the transparent region as the normal prior; Using the transparent region mask and the de-highlighted image, a supervisory map of the transparent region that retains only the de-highlighted image is constructed; In the geometry-first stage, differentiable rendering is used to obtain the current rendered image and rendering normals, and a loss function is constructed and optimized by combining the transparent region supervision map and the normal prior. Once the geometry priority phase converges, voxel fusion is performed using the learned geometric surfaces that possess cross-view depth continuity and normal consistency to construct a truncated signed distance field. The shell is obtained by extracting the zero level set from the truncated signed distance field as the reconstructed isosurface; A transparent initial point cloud is obtained by sampling on the shell, and the orientation of each point is initialized with the shell normal. At the same time, the normal direction scale is set to a fixed ratio of the tangent plane scale to form a thin shell, which serves as the geometric prior shell of the transparent object.

[0053] In one embodiment, the transparent Gaussian representation includes: the Gaussian center location, a covariance matrix describing the shape and direction of the Gaussian, Gaussian opacity, Gaussian geometric normal, reflected color, absorption coefficient, transparent material mixing coefficient, refractive index, and the color of the transparent object itself.

[0054] In one embodiment, constructing the lightweight voxel environment field includes: Construct a voxel mesh and trim voxels according to their importance; The color features of the remaining voxels after cropping are vector-quantized, compressed, and post-processed to obtain a lightweight voxel environment field.

[0055] In one embodiment, the step of clipping voxels based on voxel importance includes: calculating the importance of sampling points based on the cumulative transmittance and observation opacity during ray rendering, and assigning the voxel importance to neighboring voxels according to distance weight; adaptively determining a clipping threshold using a quantile method, and clipping voxels whose voxel importance is lower than the clipping threshold.

[0056] In one embodiment, the vector quantization compression of the color features of the remaining voxels after cropping includes: training a global codebook using an importance-weighted clustering algorithm, so that each remaining voxel only stores an index pointing to the codebook.

[0057] In one embodiment, rendering transparent objects within a 3D Gaussian splashing framework involves performing Mesh-Gaussian joint optimization within the framework, including the following steps: Based on the equivalent hit point map and normal map obtained by rendering transparent Gaussian under multiple views as geometric clues, TSDF fusion is performed and the proxy mesh is periodically updated according to a preset interval. The updated proxy mesh is used for intersection and visibility determination of the secondary refraction path. The stable secondary ray path results are fed back to the transparent rendering and multi-view loss, thereby inversely constraining the geometric and material parameter updates of the transparent Gaussian, achieving stable geometric convergence and realistic secondary refraction.

[0058] In one embodiment, the rendering of the transparent object during the Mesh-Gaussian joint optimization specifically includes: Traverse the transparent Gaussian along the principal ray and obtain the equivalent hit point and equivalent normal of the principal ray by weighted averaging. Based on the line of sight, the equivalent normal, and the refractive index of the transparent Gaussian, calculate the reflection direction and refraction direction of the main interface. Starting from the equivalent hit point, a secondary ray is constructed along the refraction direction of the main interface. The intersection with the proxy mesh is used to obtain the exit point. The exit direction after passing through the medium is calculated by combining the proxy mesh normal. Radiation information is queried in the lightweight voxel environment field along the reflection direction and the emission direction, respectively, and the transparent rendering color is synthesized by combining the Fresnel coefficient.

[0059] In one embodiment, the above method performs end-to-end optimization based on multi-view supervision during training. The rendered image is compared with the real multi-view image, and a loss is calculated. This loss includes reconstruction loss, normal consistency loss, masking loss, and normal smoothing loss. The normal consistency loss includes prior normal consistency loss and depth-derived normal consistency loss. When calculating the prior normal consistency loss, a normal mask is calculated based on the similarity between the prior normals and the rendered normals. The normal mask is used to suppress erroneous prior normals. When the optimization process converges or meets a preset termination condition, the trained transparent Gaussian 3D representation and a lightweight voxel ambient lighting model are output. During new perspective rendering, the rendering results of the background environment and the transparent objects are obtained separately. The transparent objects are then fused with the background environment to obtain a synthesized image from the new perspective.

[0060] In a preferred embodiment, the 3D reconstruction method for transparent objects based on 3D Gaussian splashing and environmental field modeling can be completed sequentially according to predetermined steps. Overall, the method takes multi-view images and camera parameters as input, first constructs a geometric prior shell of the transparent object, then generates a transparent Gaussian representation based on this shell, subsequently provides the radiation information required for transparent refraction and reflection through an environmental field neural network, and combines Mesh-Gaussian joint optimization to finally complete the rendering and end-to-end optimization of the transparent object within the 3D Gaussian splashing framework, obtaining a 3D representation of the transparent object that can be used for new perspective synthesis. Figure 2 This is the main flowchart of this embodiment. The process can be subdivided into the following 7 steps: Step S1: Collect or acquire multi-view scene data.

[0061] The system acquires a collection of multi-view RGB images containing transparent objects. ,in Indicates the first Images from multiple viewpoints, where K represents the number of viewpoints. Also included are the corresponding camera intrinsic and extrinsic parameters. ,in Indicates the first Camera intrinsic parameter matrix for each viewpoint, Let represent the rotation matrix and the translation vector, respectively.

[0062] In one alternative implementation, the system can also use a large model or preprocessing algorithm to obtain auxiliary prior information about the transparent object, including but not limited to pixel-level transparent region masks, specular removal images, and surface normal estimation results. This information will be used for subsequent geometric prior construction and constraints.

[0063] Step S2: Generate a geometric prior shell based on specular removal and normal priors.

[0064] Obtain the transparent region mask from a multi-view image De-highlighting images Compared with the original ground truth image Construct a supervisory map that retains only the transparent areas with de-specced views: In the geometry-first stage, differentiable rendering is used to obtain the current rendered image. With rendering normals And optimize with the following total loss: RGB loss For L1+SSIM loss: in, This indicates the rendering of the image.

[0065] Loss of transparent areas The formula used to constrain the spatial distribution of Gaussian elements is as follows: in, This is the opacity map obtained by Gaussian rasterization rendering from the current viewpoint. The transparent region mask obtained in step S1, This is the binary cross-entropy loss function. This loss term suppresses "floating artifacts" outside the mask range and promotes rapid convergence of Gaussian units within the transparent region by performing pixel-level supervision inside and outside the transparent region, thereby providing accurate boundary constraints for the subsequent construction of an accurate geometric prior shell.

[0066] Normal regularization loss It consists of prior normal consistency and depth-derived normal consistency, where the prior normal consistency uses robust masking to suppress erroneous priors. Transparency threshold: in, Indicates the prior normal. This represents the normal derived from the depth gradient. This represents the transpose of a vector. Indicates rendering normals, This represents the normal mask, with a value of either 0 or 1. Its function is to determine the normal mask based on prior information about the normal vector. and rendering normals The similarity between them determines whether to apply the normal prior loss.

[0067] Thin-film normalized loss This is used to induce Gaussians to exhibit a planar structure. Gaussian scale: in, Indicates the first The three scale components of a 3DGS.

[0068] In a preferred embodiment of the present invention, the parameter weights are as follows: Once the geometry-first phase converges, voxel fusion is performed using the geometric surfaces learned in this phase, which possess cross-view depth continuity and normal consistency, to construct a TSDF voxel field, i.e., a truncated signed distance field. : in, This represents the voxel coordinates or sampling point position in three-dimensional space. This refers to the truncated signed range field fusion operator, typically the TSDF Fusion algorithm. Indicates the first Rendered depth map from each observation perspective. This field The cutoff distance from each voxel unit to the nearest surface is stored discretely in space. Then, the zero-level set is extracted from this TSDF voxel field as the reconstructed isosurface to obtain the shell. : In the outer casing Upsampling yields a transparent initial point cloud and initialize each point with the shell normal. The orientation of the object (as the Z-axis of the local coordinate system) is used, and the normal direction scale is set to a fixed ratio of the tangent plane scale to form a thin shell, thereby obtaining the initial set of geometric primitives of the transparent object, i.e., the geometric a priori shell.

[0069] Step S3: Generate a transparent Gaussian representation based on the geometric shell.

[0070] The geometric shell obtained by the system in step S2 Based on this, initialize a set of three-dimensional Gaussians, each transparent Gaussian represented by the following parameters: in, It is the center of Gauss; It is the covariance matrix, which describes the shape and orientation of the Gaussian. It is Gaussian opacity; It is a Gaussian geometric normal; It is the reflected color (Tint); It is the absorption coefficient. It is the transparency factor; It is the refractive index; It is the color of the transparent object itself.

[0071] The initial position and orientation of the Gaussian set of the transparent object are determined by the position and normal of the geometric shell surface, thus ensuring that the spatial distribution of the Gaussian set is consistent with the shape of the transparent object.

[0072] Step S4: Construct a lightweight voxel environment field To provide incident radiance for reflected and refracted rays during the rendering of transparent objects, this step constructs a differentiable voxel environmental radiation field within the scene bounding box, and achieves lightweight storage and real-time querying for constrained devices through "voxel importance clipping + vector quantization compression + post-processing encoding".

[0073] Construct a voxel grid and store the volume density in each voxel. With color / appearance characteristics (denoted as) For any sampling point Obtained through trilinear interpolation The direction-dependent color is obtained by a lightweight mapping (which can be an identity or a small network). Its volume rendering cumulative form is: in For rays, The origin of the ray is indicated, usually referring to the coordinates of the camera center or the starting point of the reflected / refracted ray. In an environmental field query, it represents the location of the source emitting the probe ray. Indicates the sampling depth on the ray. Represents the unit direction vector of the ray. The interval between adjacent sampling. Represents rays The final accumulated radiometric value, i.e., the rendered color pixels, Represents cumulative transmittance, indicating the distance the ray travels to the [missing information - likely a specific point or stage]. The probability that a sample point is not occluded. Indicates the first The observation opacity of each sampling point This represents the total number of samples sampled discretely along the ray direction.

[0074] A representation based on multi-resolution hash encoding is established in three-dimensional space, and the environmental radiation field function is defined: in, This represents the spatial coordinates of the sampling points within the bounding box. The unit direction vector representing the direction of the observation viewpoint. This represents a parameterized mapping function, derived from a multi-resolution hash feature table. With lightweight multilayer perceptron (MLP) networks jointly build This represents the output incident radiance, which is the color vector of the sampling point in a specific direction.

[0075] To compress the voxel model, the importance score of the sampling point is first defined using intermediate values ​​from the voxel rendering process: The importance of each sampling point is assigned to its neighboring voxels based on distance weights. The importance of voxels was obtained. Where A represents the position of the landmass. and Set of sampling points in the neighborhood: Then, a cumulative percentage function for voxel importance is defined: in, This is an indicator function that takes a value of 1 when the condition within the parentheses is true, and 0 otherwise. Subsequently, the pruning threshold is adaptively determined using a quantile approach. : in, This represents the target retention ratio or quantile constant, and its value is typically between [0,1]. It is the inverse function of the cumulative proportion function. Prune low-importance voxels that satisfy the following formula: Low-importance voxels are clipped to significantly reduce the number of voxels without sacrificing rendering quality as much as possible.

[0076] Perform vector quantization on the color features of the remaining voxels: learning size is codebook ,in codebook The first in Each cluster center vector or representative feature vector stores only one index pointing to the codebook (approximately). Each bit points to a codebook vector, thus replacing voxel-by-voxel storage with a "codebook + index" approach. Its compression ratio can be estimated as: in voxel number This represents the number of channels (feature FP16).

[0077] The codebook is initialized and optimized using an importance-weighted clustering algorithm (such as Weighted K-Means): The codebook vector is updated using an importance-weighted moving average. Indicates the first codebook vectors The covered characteristic clustering regions (Voronoi cavities). Indicates the first The original high-dimensional appearance feature vector stored in the individual element entries. Indicates the first The importance weight of each entry (e.g., the cumulative gradient magnitude or radiance contribution at that point). Indicates the first The importance weight or radiation contribution of individual elements.

[0078] To reduce the quality loss caused by quantization, a portion of highly important voxels can be stored directly without quantization, and the retention threshold is also determined by the quantiles: in, This represents the retention ratio constant for high-precision voxels. That is, its importance is higher than... The voxels directly preserve the original features.

[0079] After determining the "voxel clipping + quantization mapping", the index mapping from voxels to the codebook is fixed, and joint fine-tuning is performed on the codebook vector and volume density. By using unquantized voxel features and small network parameters, the rendering result of the compressed model approaches the quality of the original model. Subsequently, low-bit quantization is performed on the volume density and unquantized voxel features, and entropy encoding is used to package the mask, codebook, and index, resulting in a MB-level environmental field model. : This environment field is used to provide fast, differentiable color lookups for reflected and refracted light during the rendering phase of transparent objects.

[0080] Step S5: Perform Mesh-Gaussian joint optimization within the 3D Gaussian splashing framework. For each viewpoint and pixel, the system constructs the pixel principal ray based on camera parameters: in Indicates the center of the camera. Indicates the viewing direction corresponding to the pixel. This represents the distance parameter along the ray.

[0081] (1) Calculation of the hit point and the normal line In a differentiable Gaussian rasterizer, the system traverses the transparent Gaussian objects associated with the current pixel along the principal ray and calculates the position of maximum response for each Gaussian object in the ray direction: in, Indicates the first A Gaussian in the ray The ray parameters that contribute the most to the density response in the direction. The equivalent hit point corresponding to the pixel is obtained as follows: in, This represents the cumulative transmittance. The equivalent normal is obtained through a weighted average: Other transparency properties (such as the transparency material blending factor) absorption coefficient refractive index Tint property Primary color attribute Similarly, it can be obtained through a weighted average.

[0082] (2) Calculation of reflection and refraction directions at the main interface Based on equivalent hit point Equivalent normal With line of sight Calculate the reflection direction: Calculate the direction of refraction: Simultaneously calculate the Fresnel coefficients: (3) Calculation of secondary physical refraction To achieve more realistic secondary physical refraction, this invention obtains the refraction direction at the main interface. Then, a proxy mesh extracted by TSDF fusion is introduced. As a transparent medium boundary, it is used for secondary ray intersection and outgoing direction calculation. During training, the system performs Mesh-Gaussian joint optimization: on the one hand, based on the equivalent hit point map obtained by rendering the transparent Gaussian under multiple views. With normal diagram The TSDF is used as a geometric cues for fusion and the proxy mesh is updated periodically at preset intervals (Gaussian→Mesh). On the other hand, the updated proxy mesh is used for intersection and visibility determination of secondary refraction paths, and the more stable secondary ray path results are fed back to transparent rendering and multi-view loss, thereby inversely constraining the geometric and material parameter updates of transparent Gaussian (Mesh→Gaussian) to achieve more stable geometric convergence and more realistic secondary refraction.

[0083] Definition of secondary rays: in This is a small offset used to avoid self-intersection. The distance parameter representing the direction of the secondary refracted ray represents the step size of the propagation within the medium from the point of impact on the main interface (after offset correction).

[0084] The exit point is obtained by intersecting with the proxy mesh. : in, This represents the proxy mesh surface extracted and periodically updated by TSDF fusion. The proxy mesh is then taken from... The normal at that point is (Obtained by interpolation of the triangle face normal).

[0085] The relative refractive index of the medium to air is denoted as . ,calculate: in, This represents the discriminant in the vector form of Snell's Law. It yields the exit direction of the transparent ray after it passes through the medium. and the point of launch Substitute environmental field You can obtain the radiation level by querying.

[0086] (4) Transparent color synthesis Rendering equation: in, It is the reflected radiance; It is an equivalent hit point map; It is the result of the reflected radiance query obtained from the environmental field model; This is the query result for incident radiance along the direction of outward refraction. Calculate using the following formula: in, It is a diagram of the launch point; It is the direction in which transparent light rays exit after passing through the medium. It is the absorption coefficient.

[0087] Rendering pixel color Calculate using the following formula: in, It is the Fresnel coefficient. It is the mixing coefficient of transparent materials. It is the color of the transparent object itself.

[0088] Step S6: Perform end-to-end optimization based on multi-view supervision In this step, the system compares the rendered image obtained in step S5 with the real multi-view image obtained in step S1 and calculates the loss: Reconstruction losses : Normal consistency loss , The normal derived from depth: Mask loss: in, This represents the binary cross-entropy loss function. This indicates the Gaussian rendering transparency located outside the scope of the transparent object segmentation mask.

[0089] Normal smoothing loss: in, This represents the current center pixel in the rendered image. Represents the center pixel The adjacent pixels within the 8-neighborhood or 4-neighborhood, Represents the pixel Render normal vectors, Representing adjacent pixels The rendering normal vector at that location, Represents the true RGB image in pixels The color value at that location.

[0090] In a preferred embodiment of the present invention, the parameter weights are as follows: By performing backpropagation and optimization iteration on the total loss mentioned above, the system jointly updates the Gaussian parameters of the transparent object and the environmental field parameters, so that the rendering result gradually approaches the real observed image, thereby achieving consistent reconstruction of the geometry and appearance of the transparent object.

[0091] Step S7: Output the 3D representation of the transparent object and the rendering results from the new perspective. Once the optimization process converges or meets the preset termination condition, the system outputs the trained transparent Gaussian 3D representation and a lightweight voxel ambient lighting model. Based on this output, the present invention can not only synthesize new perspectives of transparent objects from any new camera viewpoint, but also perform anti-rendering applications: while maintaining geometric invariance, by replacing the ambient lighting model or adjusting material parameters, it can achieve rapid re-rendering after relighting and material editing, thereby supporting interactive editing and regeneration.

[0092] When rendering from a new perspective, the system obtains the rendering results of the background environment separately. Rendering results with transparent objects The opacity (or foreground weight) of the transparent object at that pixel is obtained by the volume rendering accumulation of the transparent Gaussian. .in Indicates projection onto pixels Transparent Gaussian set, The contribution weight of a single transparent Gaussian to a pixel. This represents the forward cumulative transmittance. The final composite image from the new perspective blends the transparent object with the background as follows: in In the transparent rendering pipeline, the lighting is determined by both material parameters and the ambient lighting model, with the ambient lighting model providing illumination along the reflection / refraction direction. The incident color query ensures that transparent objects can still present the same reflection and refraction effects as the scene content under new perspectives, and supports subsequent relighting and material editing and re-rendering output.

[0093] This invention, within a three-dimensional Gaussian splashing framework, proposes a series of improvements regarding the geometric prior construction, ambient lighting representation, and overall framework scalability for the three-dimensional reconstruction of transparent objects. Through the combination of these techniques, this invention reduces engineering complexity and computational overhead while maintaining a physically sound model of transparent refraction and reflection phenomena, thereby solving the aforementioned technical problems and achieving the invention's objective. The beneficial technical effects resulting from this invention will be explained below, focusing on the key points of this application.

[0094] Key Point 1: Transparent geometry training optimization based on TSDF shell and low memory light refraction.

[0095] This invention utilizes multi-view RGB images ( Figure 3 ), transparent area mask ( Figure 4 ), desaturated image ( Figure 5 Information such as its normal priors is used to first perform geometric reconstruction on the transparent object to obtain its geometric shape model. Figure 5 The shell representation of the transparent object is obtained by TSDF fusion. Figure 6 Based on this, a Gaussian initialization scheme suitable for transparent rendering is constructed. Subsequently, during the transparent rendering stage, geometric cues such as depth / hit point and normals corresponding to the transparent surface are obtained and used as TSDF fusion inputs to construct a TSDF volume on the CPU side. Figure 7 It extracts triangular meshes to obtain closed, smooth and stable transparent medium boundary proxy meshes, continuously aligns with the current Gaussian geometry and suppresses fragmented structures such as "floating artifacts", while providing stable and intersectable boundary constraints for subsequent secondary ray refraction paths.

[0096] This invention enables physically realistic secondary refraction of light. In the main ray stage, it still uses equivalent hit points, equivalent normals, and transparent Gaussian material parameters (such as refractive index) to calculate the refraction / reflection direction. In the secondary ray stage, it introduces a ray tracing scheme based on the proxy mesh to perform intersection and visibility determination on the secondary refraction / reflection paths within the transparent medium, thereby achieving a more physically realistic model of secondary refraction. Since the TSDF volume and proxy mesh are primarily maintained on the CPU side, and video memory is only used for rendering depth / hit points and a small amount of intersection data transfer, this invention significantly reduces video memory usage while achieving more realistic secondary refraction, avoiding the high video memory and computational overhead caused by introducing a heavy neural SDF training pipeline.

[0097] The beneficial technical effects are as follows: Theoretically, this invention uses a proxy mesh as the geometric anchor point of the transparent medium boundary, and coordinates "geometric training optimization (TSDF fusion and mesh extraction)" and "transparent rendering optimization (Gaussian parameter learning)" in a lightly coupled manner. While maintaining the overall lightweight framework, it achieves stable intersection of secondary ray paths, significantly improving the physical consistency and training stability of the refraction deformation of transparent objects. From an engineering perspective, this invention generates and updates the proxy mesh through periodic TSDF fusion, avoiding the training of additional implicit geometric networks and large-scale memory consumption. It balances implementation complexity, operational efficiency, and cross-platform deployment friendliness, thus effectively solving the problems of geometric instability of transparent objects, unrealistic secondary refraction, and excessive resource consumption in existing technologies. A comparison with other transparent object reconstruction geometry optimization methods is shown in Table 1.

[0098] Table 1. Comparison of Geometric Optimization for Transparent Reconstruction Key Point 2: Lightweight illumination representation based on the ambient field, replacing dense probe bodies.

[0099] This invention addresses the need for obtaining the reflection / refractive color of transparent objects by proposing a compressed voxel environment field as a lighting representation, replacing the large-scale lighting probes deployed in the scene space in existing probe-based solutions. Specifically, the system first uses a segmentation mask of the transparent region ( Figure 4 This suppresses / culls transparent objects from the input view, only affecting non-transparent backgrounds. Figure 8 Modeling and rendering are performed to obtain the environmental field.

[0100] In terms of environmental field construction, this invention uses a voxel grid-like radiation field as the basic representation and introduces a joint compression mechanism of "voxel importance clipping + vector quantization compression + post-processing encoding": By assessing the importance of voxels, voxels that contribute less are pruned to significantly reduce storage. Vector quantization is performed on the feature vectors of the preserved voxels, and codebook indexing is used instead of high-precision storage. The residual weights are further quantized, resulting in a final model storage of MB, while significantly compressing the environment representation size while maintaining usable rendering quality.

[0101] The beneficial technical effects are as follows: Theoretically, this invention transforms the discrete lighting representation, which originally relied on a 3D probe mesh, into a continuous environment field representation. This prevents the parameter scale of the environment representation from increasing linearly with scene spatial resolution and probe density, fundamentally alleviating the memory consumption and computational overhead problems caused by dense probes. In practical terms, while maintaining good representation of transparent refractive textures and mutual reflection details, this invention significantly reduces the peak memory usage and environment query time during the transparent reconstruction stage, making it easier to deploy transparent reconstruction functionality on resource-constrained devices (such as mobile GPUs and embedded platforms). By comparing the experimental results with the open-source implementation of TransparentGS, the applicant observed that under similar scene settings, the environment representation implemented in this invention significantly alleviates the overall computational burden during the training and rendering stages while achieving similar rendering quality. This effectively solves the problems of high overhead and difficulty in expansion and deployment of environment field representation in existing technologies. Specific comparison items are shown in Table 2.

[0102] Table 2. Comparison of Environmental Field Methods Key Point 3: Modular and scalable transparent 3DGS reconstruction framework design.

[0103] This invention designs TSDF geometry optimization, transparent Gaussian representation, and environment field query as relatively independent modules in its overall architecture. These modules exchange and combine data through clear interfaces, facilitating the replacement or expansion of specific implementation methods according to actual needs. For example, the geometry optimization part can be replaced with other SDF, deep fusion, or opaque 3DGS reconstruction results; the environment field part can be replaced with radiation field neural networks of different architectures or other lighting representations; and the transparent rendering part can also be extended to more complex refraction / scattering models under the same interface.

[0104] The beneficial technical effects are as follows: This modular design improves the maintainability and evolvability of the system, allowing developers to independently optimize and upgrade geometry, lighting, or material modules without disrupting the overall framework, thereby accelerating algorithm iteration. Furthermore, the framework is highly compatible with existing opaque 3DGS pipelines, enabling business systems to incrementally integrate transparent modules on top of existing 3DGS systems, reducing engineering modification costs. It also provides a unified platform for the subsequent introduction of lightweight mobile technologies such as model distillation and parameter quantization. In summary, this invention, through structural simplification and modularization, achieves the natural extension and expansion of 3D Gaussian splashing in transparent object scenes, facilitating the transition of transparent object reconstruction technology from laboratory research to practical product deployment, thus enhancing the overall engineering usability and industrial value of 3D reconstruction of transparent objects.

[0105] Through the above key points and their beneficial technical effects, it can be seen that the technical solution proposed in this invention works together in terms of geometric training, ambient lighting representation and overall framework scalability. It not only solves the problems of strong geometric dependence, large ambient representation overhead and high deployment difficulty in the prior art, but also provides a more unified solution in theory and a more user-friendly and efficient solution in practice for the 3D reconstruction of transparent objects.

[0106] Another embodiment of the present invention provides a three-dimensional Gaussian splash reconstruction system for transparent objects based on efficient environmental field modeling, such as... Figure 9 As shown, it includes: The acquisition module is used to acquire multi-view images; The geometric prior shell generation module is used to construct the geometric prior shell of a transparent object based on multi-view images; A transparent Gaussian representation generation module is used to generate transparent Gaussian representations based on geometric prior shells; The voxel environment field construction module is used to construct lightweight voxel environment fields, which provide the radiation information required for refraction and reflection. The transparent object rendering module is used to render transparent objects within a three-dimensional Gaussian splashing framework based on the transparent Gaussian representation and the radiation information, thereby obtaining a three-dimensional representation of the transparent object.

[0107] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here. Each of the above modules can be implemented entirely or partially through software, hardware, or a combination thereof.

[0108] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0109] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0110] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0111] In the implementation of this invention, various technically equivalent variations and alternative combinations can exist regarding geometric prior construction, transparent Gaussian representation, environmental field modeling, and training and deployment methods. These alternatives, without altering the basic technical concept of this invention, can achieve the same or similar technical effects and should be considered equivalent solutions within the scope of protection of this invention.

[0112] 1. For the geometric priors and shell construction phase, other forms of 3D surface or volume representation can be used to replace the TSDF shell / surrogate mesh. For example, a dense point cloud obtained by multi-view stereo matching, and a mesh obtained by Poisson reconstruction or other surface reconstruction algorithms can be used as the shell / surrogate mesh. Alternatively, isosurfaces extracted by a neural SDF network or voxel occupancy fields obtained by a depth fusion algorithm can be used. Furthermore, in the Mesh-Gaussian joint optimization framework of this invention, the mesh obtained in any of the above methods can be used as a "surrogate mesh" to participate in the training loop: on the one hand, geometric cues such as depth / hit points and normals obtained by transparent Gaussian rendering drive the generation or periodic update of the mesh (Gaussian→Mesh); on the other hand, the mesh is used for intersection and visibility determination of secondary refraction / reflection paths, and feeds back more stable secondary ray path results to transparent rendering and multi-view loss to constrain the update of transparent Gaussian parameters (Mesh→Gaussian). As long as the geometric prior is still used to constrain the distribution and orientation of transparent Gaussians in space, to provide approximate shell information for the shape of transparent objects, and to provide geometric intersection constraints for refraction paths (especially secondary physical refraction), without changing the basic mechanism of representing geometry and materials through Gaussians in transparent rendering, it can be considered functionally equivalent to the TSDF shell scheme. Furthermore, the shell structure can be transformed from a double-layer shell to a single-layer shell or a multi-layer shell. The shell thickness can be obtained by a fixed constant, by scene statistics estimation, or by network prediction. As long as it still approximates the spatial range of the transparent medium and provides geometric constraints for the refraction path, it belongs to the alternative implementations of this invention.

[0113] 2. Regarding the specific representation of the Transparent Gaussian, different kernel functions or parameterization methods can be used while maintaining the basic idea of ​​"explicit particles / basis functions + physical parameters". For example, the anisotropic 3D Gaussian can be replaced with other differentiable anisotropic kernel functions, or different choices can be made in the decomposition form of the covariance matrix and the parameter range constraints; some material parameters (such as refractive index, absorption coefficient, roughness, etc.) can also be replaced by Gaussian-independent parameters, which are uniformly generated by low-dimensional latent variables or multi-scale fields. As long as reasonable reflection and refraction behaviors can still be derived from the Gaussian position and normal during the rendering stage, the functionality remains equivalent. In addition, the same form as TransparentGS can be used for the mixing of transparent and opaque behaviors, or it can be replaced by other mixing functions that satisfy energy conservation. As long as the reflection and refraction channels are still continuously interpolated through the transparency weight, the same invention purpose can be achieved.

[0114] 3. Regarding the representation and querying of the environmental field, the environmental field used in this invention can be replaced by other models with position-direction correlated radiance query capabilities. For example, an environmental field based on voxel meshes and spherical harmonic function expansion, an environmental light representation based on point clouds or surface maps, or an NGP-type model with multi-resolution hash encoding can be used. Alternatively, an environmental map can be used in the far field, and a combination of local light field networks can be used in the near field. As long as the corresponding radiance can still be queried based on the equivalent hit point and reflection / refraction direction during transparent rendering, and used to calculate the color of reflected and refracted light, it can be considered an equivalent replacement for the environmental field. Furthermore, the specific query strategy can be a single direct query, or a limited number of hierarchical queries or an importance sampling strategy can be introduced. As long as a reasonable balance is maintained between computational load and accuracy, and a transparent refraction and mutual reflection effect similar to the original scheme can be achieved, all are alternative solutions of this invention.

[0115] 4. Regarding the training process and loss function design, this invention provides a preferred combination of multiple losses and a phased training strategy in the specification. However, the specific loss form and weight configuration can be replaced according to the scenario and implementation requirements. For example, the reconstruction loss can be selected or combined from L1, L2, perceptual loss, contrast loss, etc.; the normal consistency constraint can be replaced by the included angle loss or other geometric regularizations; and the transparency and energy constraints can be represented by different forms of sparse regularizations or penalty terms. The splitting of the training phase and the parameter freezing order can also be merged into a single-stage joint optimization, or further subdivided into more stages. As long as the overall training process still revolves around the goal of "optimizing the transparent Gaussian and environmental field parameters under geometric prior constraints to make the rendering result consistent with the multi-view image" and can ensure stable convergence of the system, it is considered an equivalent alternative implementation to the scheme of this invention.

[0116] 5. Regarding system architecture and deployment, the modules of this invention can be implemented on the same computing device or in a multi-device or distributed environment. For example, geometric prior construction can be completed offline by the CPU or server-side GPU, while transparent rendering and environment querying can be performed on client or mobile devices; the environment field can be distilled, quantized, or structurally pruned after training before being deployed to different computing platforms. As long as the overall process still includes core steps such as geometric prior construction, transparent Gaussian rendering, and environment radiation querying, and ultimately achieves physically reasonable 3D reconstruction and new perspective synthesis of transparent objects in the scene, it can be considered a different implementation approach of the technical solution of this invention.

[0117] In summary, without departing from the basic concept of this invention, all substitutions and modifications made to geometric prior forms, transparent Gaussian parameterization, environmental field representation, training process design, and system architecture deployment, as long as they remain equivalent in hierarchy and function and can achieve the technical problems and expected technical effects that this invention aims to solve, should be included within the scope of protection of this invention.

Claims

1. A method for reconstructing three-dimensional Gaussian splashes of transparent objects based on efficient environmental field modeling, characterized in that, Includes the following steps: Obtain multi-view images and construct a geometric prior shell for the transparent object based on the multi-view images; Generate a transparent Gaussian representation based on a geometric prior shell; Construct a lightweight voxel environment field to provide the radiation information required for refraction and reflection; Based on the transparent Gaussian representation and the radiation information, the transparent object is rendered within a three-dimensional Gaussian splashing framework to obtain a three-dimensional representation of the transparent object.

2. The method according to claim 1, characterized in that, The construction of the geometric prior shell of the transparent object based on the multi-view images includes: Obtain the transparent region mask, the de-highlight image, and the original ground truth image from the multi-view image, and obtain the prior normal of the transparent region as the normal prior; Using the transparent region mask and the de-highlighted image, a supervisory map of the transparent region that retains only the de-highlighted image is constructed; In the geometry-first stage, differentiable rendering is used to obtain the current rendered image and rendering normals, and a loss function is constructed and optimized by combining the transparent region supervision map and the normal prior. Once the geometry priority phase converges, voxel fusion is performed using the learned geometric surfaces that possess cross-view depth continuity and normal consistency to construct a truncated signed distance field. The shell is obtained by extracting the zero level set from the truncated signed distance field as the reconstructed isosurface; A transparent initial point cloud is obtained by sampling on the shell, and the orientation of each point is initialized with the shell normal. At the same time, the normal direction scale is set to a fixed ratio of the tangent plane scale to form a thin shell, which serves as the geometric prior shell of the transparent object.

3. The method according to claim 1, characterized in that, The transparent Gaussian representation includes: the center position of the Gaussian, the covariance matrix describing the shape and direction of the Gaussian, the Gaussian opacity, the Gaussian geometric normal, the reflected color, the absorption coefficient, the mixing coefficient of the transparent material, the refractive index, and the color of the transparent object itself.

4. The method according to claim 1, characterized in that, The construction of the lightweight voxel environment field includes: Construct a voxel mesh and clip voxels according to their importance; the clipping of voxels according to their importance includes: calculating the importance of sampling points based on the cumulative transmittance and observation opacity during the ray rendering process, and assigning them to neighboring voxels according to distance weight to obtain the voxel importance; adaptively determining the clipping threshold through a quantile method, and clipping voxels whose voxel importance is lower than the clipping threshold; The color features of the remaining voxels after clipping are vector-quantized, compressed, and post-processed to obtain a lightweight voxel environment field. The vector-quantization compression of the color features of the remaining voxels after clipping includes: training a global codebook using an importance-weighted clustering algorithm so that each remaining voxel only stores an index pointing to the codebook.

5. The method according to claim 1, characterized in that, The rendering of transparent objects within the 3D Gaussian splashing framework involves performing Mesh-Gaussian joint optimization within the framework, including the following steps: Based on the equivalent hit point map and normal map obtained by rendering transparent Gaussian under multiple views as geometric clues, TSDF fusion is performed and the proxy mesh is periodically updated according to a preset interval. The updated proxy mesh is used for intersection and visibility determination of the secondary refraction path. The stable secondary ray path results are fed back to the transparent rendering and multi-view loss, thereby inversely constraining the geometric and material parameter updates of the transparent Gaussian, achieving stable geometric convergence and realistic secondary refraction.

6. The method according to claim 5, characterized in that, During the execution of the Mesh-Gaussian joint optimization, the rendering of the transparent object includes: Traverse the transparent Gaussian along the principal ray and obtain the equivalent hit point and equivalent normal of the principal ray by weighted averaging. Based on the line of sight, the equivalent normal, and the refractive index of the transparent Gaussian, calculate the reflection direction and refraction direction of the main interface. Starting from the equivalent hit point, a secondary ray is constructed along the refraction direction of the main interface. The intersection with the proxy mesh is used to obtain the exit point. The exit direction after passing through the medium is calculated by combining the proxy mesh normal. Radiation information is queried in the lightweight voxel environment field along the reflection direction and the emission direction, respectively, and the transparent rendering color is synthesized by combining the Fresnel coefficient.

7. The method according to claim 1, characterized in that, During training, end-to-end optimization is performed based on multi-view supervision. The rendered image is compared with the real multi-view image, and the loss is calculated. The loss includes reconstruction loss, normal consistency loss, mask loss, and normal smoothing loss. The normal consistency loss includes prior normal consistency loss and depth-derived normal consistency loss. When calculating the prior normal consistency loss, a normal mask is calculated based on the similarity between the prior normal and the rendered normal, and the normal mask is used to suppress erroneous normal priors. When the optimization process converges or meets the preset termination condition, the trained transparent Gaussian 3D representation and the lightweight voxel ambient lighting model are output. During the new perspective rendering, the rendering results of the background environment and the transparent object are obtained separately. The transparent object and the background environment are fused to obtain the new perspective synthetic image.

8. A three-dimensional Gaussian splash reconstruction system for transparent objects based on efficient environmental field modeling, characterized in that, include: The acquisition module is used to acquire multi-view images; The geometric prior shell generation module is used to construct the geometric prior shell of a transparent object based on multi-view images; A transparent Gaussian representation generation module is used to generate transparent Gaussian representations based on geometric prior shells; The voxel environment field construction module is used to construct lightweight voxel environment fields, which provide the radiation information required for refraction and reflection. The transparent object rendering module is used to render transparent objects within a three-dimensional Gaussian splashing framework based on the transparent Gaussian representation and the radiation information, thereby obtaining a three-dimensional representation of the transparent object.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images