Physical attribute inversion and three-dimensional reconstruction method based on Gaussian splashing and micro rendering

Abstract

The invention discloses a physical attribute inversion and three-dimensional reconstruction method based on Gaussian splashing and micro-rendering, and belongs to the technical field of computer vision, and the method comprises the steps: generating a sparse point cloud based on a multi-view image to initialize a Gaussian ball; determining a Gaussian pivot point based on a Gaussian ball, constructing a self-adaptive tetrahedral mesh in combination with gradient optimization, extracting an explicit surface from the self-adaptive tetrahedral mesh, and sampling to obtain geometric information; predicting the physical attribute of each sampling point through an implicit texture network, and providing ambient light through an HDR ambient light map; geometric information, physical attributes and ambient light are used as input, a differentiable PBR renderer is adopted to generate a physical rendering image, multi-term loss function reverse optimization is constructed, and collaborative optimization and inversion of three-dimensional geometry and surface physical attributes are realized. According to the method, the problems of texture dislocation, thin-wall structure disappearance and high rendering calculation cost under dynamic topology are effectively solved, and a digital model with fine geometry and a realistic material can be efficiently recovered from a multi-view image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for physical property inversion and 3D reconstruction based on Gaussian splashing and differentiable rendering. Background Technology

[0002] In recent years, with the rapid development of the digital economy, the demand for high-quality 3D digital assets has grown exponentially. 3D scenes can be represented in various forms, such as point clouds, meshes, multiple views, and neural radiation fields. 3D Gaussian point clouds are a novel method for representing 3D scenes, enabling real-time rendering of photorealistic images and suitable for representing and compressing 3D volumetric videos.

[0003] 3D reconstruction, a key research direction in computer vision, plays a crucial role in the construction of the metaverse and the development of scene simulators required for reinforcement learning. Furthermore, it demonstrates broad application potential in fields such as digital twins, virtual reality, autonomous driving, and robotics. These fields have an urgent need for high-precision, high-efficiency 3D scene reconstruction to achieve accurate simulation and interaction of the real world. An ideal 3D reconstruction technology should possess three core characteristics: high-precision geometry, decoupled physically based materials (PBR), and efficient rendering and training speeds. However, current mainstream technologies struggle to simultaneously meet these three requirements, primarily due to the following technical bottlenecks: (1) Limitations of geometric representation: Implicit field methods, such as Neural Radiation Fields (NeRF), achieve photorealistic synthesis of new viewpoints through volumetric rendering, but their geometric surfaces are implicitly contained within the density field of the neural network. Extracting explicit meshes typically requires running the MarchingCubes algorithm as post-processing, which often results in the loss of high-frequency geometric details (such as edges and thin slices) and poor mesh topology quality, making it difficult to use for physical simulations. Furthermore, implicit field training converges slowly, typically requiring several hours.

[0004] Explicit point cloud methods, such as 3D Gaussian Splash (3DGS), utilize 3D Gaussian ellipsoids as discrete primitives, achieving real-time rendering speeds and rapid training. However, 3DGS is essentially a discrete "glowing cloud" lacking continuous surface definition. Meshes extracted directly from 3DGS often contain significant noise, holes, or non-manifold structures, failing to meet the demands of industrial-grade modeling.

[0005] Fixed-mesh methods, such as existing inverse rendering technologies like NVIDIA's nvdiffrec, typically employ Deep Moving Tetrahedron (DMTet) technology, which defines a signed distance field (SDF) on a tetrahedral mesh with a fixed resolution. This fixed topology lacks adaptability: for large-scale scenes, insufficient resolution leads to blurred details; for sparse objects, it results in a huge waste of computational resources.

[0006] (2) The challenge of decoupling physical properties: Light baking problem: Traditional 3DGS and NeRF typically "bake" lighting information into color parameters (spherical harmonics). This means that the reconstructed object surface color includes the shadows and highlights from the shooting. If these models are placed in a new virtual lighting environment, the original lighting will clash with the new lighting, producing a highly unrealistic visual effect.

[0007] The efficiency bottleneck of reverse rendering: Existing material inversion methods attempt to separate albedo, roughness, and ambient light from images. However, to simulate realistic lighting reflections (the integral of the rendering equation), Monte Carlo integration is often required, which results in extremely high memory consumption and slow speed during training. If spherical harmonic functions are used to approximate lighting, only low-frequency lighting can be recovered, and sharp specular reflections (such as reflections on metal surfaces) cannot be reconstructed.

[0008] Therefore, traditional 3D reconstruction methods suffer from limitations in geometric reconstruction accuracy, difficulty in adaptively updating mesh topology, and difficulty in efficiently decoupling physical materials from ambient lighting. Summary of the Invention

[0009] To address the limitations of existing technologies, such as limited geometric reconstruction accuracy, difficulty in adaptively updating mesh topology, and inefficient decoupling of physical materials and ambient lighting, this invention aims to provide a physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering. This method utilizes Gaussian splashing for geometric initialization and representation, relies on the dynamic topology of Gaussian splashing to drive the generation of tetrahedral meshes, and extracts differentiable surfaces to predict the material properties of sampled points on the surface. Furthermore, a differentiable physical rendering pipeline is introduced, using segmentation and summation approximation to reduce the complexity of full-frequency ambient lighting calculations. A multivariate loss function gradient backpropagation optimization is established to optimize HDR ambient light maps, explicit triangular mesh surfaces, and Gaussian splashing image sequences, achieving collaborative optimization and inversion of physical properties and 3D geometry. This solves the problems of texture misalignment, thin-walled structure loss, and rendering accuracy issues under dynamic topology, enabling efficient recovery of digital models with detailed geometry and realistic materials from multi-view images.

[0010] To achieve the above-mentioned objectives, an embodiment provides a method for physical property inversion and 3D reconstruction based on Gaussian splashing and differentiable rendering, comprising the following steps: Sparse point clouds are generated based on multi-view images. A set of Gaussian spheres is initialized as a Gaussian splash image sequence based on the sparse point clouds. The contribution of each Gaussian splash image sequence is calculated using Gaussian splash rendering to filter Gaussian pivot points. An adaptive tetrahedral mesh is constructed based on the Gaussian pivot points. The surface of the explicit triangular mesh is dynamically extracted from the adaptive tetrahedral mesh to obtain geometric information. Sampling is performed on the surface of an explicit triangular mesh, and the physical properties of each sampling point are predicted through an implicit material network. At the same time, ambient lighting is provided through an HDR ambient light map. Using the geometric information, physical properties, and ambient lighting of each sampling point as input, a differentiable PBR renderer is used to generate physically rendered images. Based on the differences between the generated physically rendered image and the real image, multiple loss functions are constructed for gradient backpropagation to optimize implicit material network parameters, HDR ambient light maps, explicit triangular mesh surfaces, and Gaussian splash image sequences, thereby achieving the synergistic optimization and inversion of physical properties and 3D geometry to generate 3D reconstructed images.

[0011] Preferably, the method for generating sparse point clouds based on multi-view images includes: using the structure-of-motion reconstruction algorithm to perform feature matching on multi-view images, calculating the camera intrinsic matrix and camera extrinsic matrix of each image, and obtaining sparse point clouds.

[0012] Preferably, the parameters that can be optimized in the Gaussian splash image sequence include: position vector, rotation quaternion, scaling vector, opacity scalar, and spherical harmonic coefficients.

[0013] Based on these optimizable parameters, preliminary optimization using the Gaussian Splash (3DGS) rendering pipeline can enable the Gaussian spheres, which form a Gaussian splash image sequence, to roughly cover the geometric contours of the scene, facilitating the complete acquisition of subsequent geometric information.

[0014] Preferably, the step of calculating the contribution of each Gaussian splash image sequence using Gaussian splash rendering to filter Gaussian pivot points includes: The average contribution of Gaussian spheres as a Gaussian splash image sequence is calculated using Gaussian splash rendering. The average contribution is the mean of the transmittance-weighted opacity of each Gaussian sphere as a Gaussian splash image sequence across all viewpoints. The average contribution is sorted, and the top N are selected as the initial Gaussian spheres for the Gaussian splash image sequence to establish a Gaussian set. For each Gaussian sphere in the Gaussian set as the initial Gaussian sphere for the Gaussian splash image sequence, the principal axis direction is calculated based on the covariance matrix. Multiple corner points extending along the principal axis direction to the bounding box boundary and a Gaussian sphere center point are determined, and the Gaussian pivot point is selected.

[0015] In this system, corner points are the vertices of the oriented bounding box, typically located at the edge of the Gaussian sphere (or even extending outwards), while the center point is located inside. Thus, the center point (inner) and corner points (outer) naturally form a set of points spanning the object's surface, which facilitates the subsequent construction of a tetrahedral mesh that encloses the object using the Delaunay algorithm, ensuring that the mesh density precisely matches the object's surface.

[0016] Preferably, the construction of an adaptive tetrahedral mesh based on the Gaussian pivot point includes: By combining the Gaussian splash image sequence parameters optimized by gradient backpropagation, the selected Gaussian pivot points are adaptively updated. Based on the updated Gaussian pivot points, an adaptive tetrahedral mesh is constructed using Delaunay triangulation.

[0017] By performing Delaunay triangulation on all selected pivot points, a triangulation is constructed for a given set of Gaussian pivot points, ensuring that the circumcircle of each triangle does not contain any points from other point sets. This creates an adaptive tetrahedral mesh that tightly wraps the object's surface, transforming the discrete set of Gaussian spheres into a continuous, compact voxelized space (tetrahedral mesh), providing a carrier for subsequent surface extraction. Furthermore, by combining the parameters from the Gaussian splash image sequence optimized by gradient backpropagation, an adaptive tetrahedral mesh is constructed through dynamic topology, allowing the mesh density to automatically adapt to the complexity of the object's surface.

[0018] Preferably, dynamically extracting the explicit triangular mesh surface from the adaptive tetrahedral mesh includes: At each Gaussian pivot point of the constructed adaptive tetrahedral mesh, a signed distance field value is defined; Using the differentiable tetrahedron traveling algorithm, zero isovalue points are calculated by linear interpolation on the tetrahedron edges where the sign distance field value crosses zero. An explicit triangular mesh surface is generated based on the tetrahedron edges and the zero isovalue points. The symbolic distance field value of the explicit triangular mesh surface is iteratively optimized based on gradient backpropagation, and the explicit triangular mesh surface is dynamically extracted.

[0019] Preferably, the implicit material network is constructed by initializing an implicit material network based on a multilayer perceptron. Based on volumetric texture, the three-dimensional coordinates of sampling points on the surface of the explicit triangular mesh are input into the implicit material network based on the multilayer perceptron. Through a series of fully connected layers and nonlinear activation functions, physical properties are output for the shading calculation of the sampling points. The physical properties include: base color, roughness, metallicity, and normal perturbation.

[0020] Traditional UV mapping requires pre-unfolding a 3D mesh onto a 2D plane and establishing a fixed correspondence between each vertex and texture pixel. If the mesh topology changes drastically during optimization (e.g., opening or closing holes, splitting or merging), the original UV mapping relationship becomes invalid, leading to severe texture misalignment, stretching, or tearing. This application, however, combines volumetric textures with a multilayer perceptron, where material properties are determined solely by 3D coordinates. Regardless of mesh deformation or topology changes, the coordinate query result for the same spatial point remains stable. Mesh vertices can move freely, and faces can be added or removed, but the coordinates of newly sampled points on the surface can always obtain reasonable and consistent material properties through the same multilayer perceptron. Furthermore, besides avoiding texture misalignment caused by mesh topology changes, using volumetric textures allows for stable and continuous propagation of material and geometric gradients in the backpropagation context, preventing discrete and non-differentiable jumps due to parameter changes, achieving complete decoupling of material and geometry.

[0021] Preferably, when using a differentiable PBR renderer to generate physically rendered images, the Split-Sum approximation is introduced to solve the rendering equations, which is used to account for the specular reflection term. It is approximately the product of the pre-filtered environment map and the environment BRDF map, and the approximate calculation formula is as follows: ,in, For ambient light, For surface roughness, For the environment BRDF map lighting, For normal line, The direction of the incident light.

[0022] More preferably, the generation of physically rendered images using a differentiable PBR renderer includes: Using geometric information, physical properties, and ambient lighting as inputs, a pre-filtered environment map and an environment BRDF map are generated, respectively. The pre-filtered environment map is obtained by performing differentiable convolution or filtering on the ambient lighting of the HDR environment map to generate a set of multi-level asymptotic textures to simulate specular reflection blur of different roughness. The environment BRDF map is calculated by integrating Fresnel terms and geometric terms, and is used to calculate and store an integral lookup table. In the real-time rendering stage, the specular reflection direction is calculated using the geometric information, physical properties, and ambient lighting of each sampling point as input. A multi-level asymptotic texture of the pre-filtered environment map is selected based on the roughness, and the pre-filtered environment map is sampled along the specular reflection direction to obtain the pre-filtered color. A two-dimensional vector is obtained based on the integral lookup table and the roughness sampling integral lookup table. The pre-filtered color, the two-dimensional vector, and the Fresnel term are multiplied to obtain the final specular reflection color. The final specular and diffuse colors are combined to generate a physically rendered image.

[0023] Preferably, the constructed multiple loss functions include volume rendering loss, geometric bidirectional consistency loss, and anti-corrosion regularization loss; Among them, volume rendering loss It provides gradient optimization at the appearance level to calculate the photometric error between the generated physically rendered image and the real image, and iteratively optimizes the implicit material network parameters and HDR ambient ray map parameters. This represents the mean absolute error of pixel color. Indicates structural difference loss. and These are weight parameters; Geometric bidirectional consistency loss It provides geometric gradient optimization to constrain the depth map obtained by explicit surface rasterization. Depth map obtained by Gaussian rendering The geometric shell remains consistent, ensuring that the explicit surface does not deviate from the visual shell of the Gaussian cloud. For loss weighting coefficients, It is an L1 norm; Corrosion resistance regularization loss It provides gradient optimization at the structural prior level to constrain the signed distance field value of all Gaussian spheres that serve as the center points of the Gaussian splash image sequence to be negative, thus preventing the thin-walled structure from disappearing during the optimization process. It is a function with maximum value. For the first The symbolic distance field value at the center point of a Gaussian sphere.

[0024] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. Based on the contribution of Gaussian splash image sequence, a tetrahedral mesh is dynamically constructed, which maintains the regularity of the explicit surface and inherits the flexibility and differentiability of the implicit representation; the geometric parameters of the Gaussian splash image sequence are optimized by gradient backpropagation, and dynamic topology construction is adaptively performed according to the current geometric parameters.

[0025] 2. From the adaptive tetrahedral mesh, renderable explicit triangular mesh surfaces are extracted. By defining implicit surfaces in voxel space, the implicit surfaces are transformed into explicit meshes, allowing geometric gradients to propagate in reverse. This further maps the geometry (surface point coordinates) to the appearance (material properties). The use of implicit material networks and volumetric textures based on MLP ensures compatibility and continuity between material representation and dynamic geometry, fundamentally avoiding texture stretching and misalignment problems caused by dynamic mesh changes (such as topological splitting and hole closure). This successfully decouples the physical properties of the scene and outputs standard display surfaces and PBR texture maps, which can be directly imported into the engine for secondary creation and relighting.

[0026] 3. By constructing multiple loss functions, gradients are provided at the appearance, geometry, and structural prior levels to drive material and lighting optimization, force explicit meshes to align with Gaussian shells and protect geometric details, demonstrating a complete closed loop driven by Gaussian splashing to promote the collaborative optimization of geometry and appearance, and achieving high-fidelity physical property inversion and 3D geometric reconstruction.

[0027] 4. Combining the efficient initialization advantages of Gaussian splashing with the approximate solution of complex rendering equations by Split-Sum lighting, it achieves high-quality rendering with only two texture samplings, effectively recovering high-frequency environmental reflections, ensuring physical realism while reducing computational overhead, and significantly shortening the training time for reverse rendering of complex scenes. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering provided in the embodiment.

[0030] Figure 2 This is a schematic diagram of the physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering provided in the embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0032] The inventive concept of this invention addresses the problems in 3D reconstruction methods, such as limited geometric reconstruction accuracy, difficulty in adaptively updating mesh topology, and inefficient decoupling of physical materials and ambient lighting. Based on this, this invention provides a 3D reconstruction and physical property inversion method based on Gaussian splashing and differentiable rendering. It utilizes Gaussian splashing for geometric initialization and representation, relies on the dynamic topology of Gaussian splashing to drive the generation of tetrahedral meshes, and extracts differentiable surfaces to predict the material properties of sampling points on the surface. Furthermore, it introduces a differentiable physical rendering pipeline, uses segmentation and summation approximation to reduce the complexity of full-frequency ambient lighting calculations, and establishes a multivariate loss function gradient backpropagation optimization for HDR ambient light maps, explicit triangular mesh surfaces, and Gaussian splashing image sequences. This achieves collaborative optimization and inversion of physical properties and 3D geometry, solving the problems of texture misalignment, thin-walled structure disappearance, and rendering accuracy issues under dynamic topology. It can efficiently recover digital models with detailed geometry and realistic materials from multi-view images and shorten the time for reverse rendering of complex scenes.

[0033] This embodiment uses the 3D reconstruction of a multi-view outdoor scene image consisting of 100 images with a resolution of 800×800 pixels as an example to illustrate the implementation process of the present invention in detail. Figure 1 As shown in the embodiment, a physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering is provided, including the following steps: S1. Generate sparse point clouds based on multi-view images. Initialize a set of Gaussian spheres as a Gaussian splash image sequence based on the sparse point clouds. Calculate the contribution of each Gaussian splash image sequence using Gaussian splash rendering to filter Gaussian pivot points. Construct an adaptive tetrahedral mesh based on the Gaussian pivot points. Dynamically extract the surface of the explicit triangular mesh from the adaptive tetrahedral mesh to obtain geometric information.

[0034] In the embodiments, such as Figure 2 As shown, the first step is to read 100 input 2D RGB image data. The structure-of-motion (COMO) algorithm is then used to perform feature matching on the image sequence, calculate the camera intrinsic and extrinsic parameter matrices (including rotation and translation) for each image, and obtain a sparse point cloud.

[0035] Each point in the sparse point cloud is initialized as a 3D Gaussian sphere, forming a Gaussian splatter image sequence. Each Gaussian sphere in the Gaussian splatter image sequence contains the following set of optimizable parameters: position vector, rotation quaternion, scaling vector, opacity scalar, and spherical harmonic coefficients.

[0036] Using the Gaussian Splatter (3DGS) rendering pipeline to perform preliminary optimization of the position, rotation, size, and opacity of the Gaussian spheres allows the Gaussian spheres, as part of a Gaussian splatter image sequence, to roughly cover the geometric outline of the scene, facilitating the complete acquisition of subsequent geometric information.

[0037] During training iterations, instead of using a pre-set fixed grid, the topology is dynamically constructed based on the current Gaussian splash image sequence. Specifically: Pivot sampling: In this embodiment, based on Gaussian spheres, the average contribution of each Gaussian sphere across all training viewpoints is calculated to eliminate internal or invalid geometry; the average contribution is the mean of the transmittance-weighted opacity of each Gaussian sphere across all viewpoints. The Gaussian spheres with the highest average contribution are sorted and selected as the active Gaussian set. For each Gaussian sphere in the active Gaussian set, nine Gaussian pivot points are generated. These nine pivot points consist of one Gaussian sphere center point and eight corner points extending along the principal axis of the Gaussian sphere to the bounding box boundary. The eight corner points are the vertices of the oriented bounding box, typically located at the edge of the Gaussian sphere (or even extending outwards), while the center point is located inside.

[0038] Topology Construction: Combining the Gaussian splash image sequence parameters optimized by gradient backpropagation, the selected Gaussian pivot points are adaptively updated. Based on the updated Gaussian pivot points, Delaunay triangulation is used. Delaunay triangulation refers to constructing a triangulation for a given set of points such that the circumcircle of each triangle does not contain any points from other point sets, thus constructing an adaptive tetrahedral mesh that tightly wraps around the object's surface. In this way, the center point (inner) and corner points (outer) can adaptively form a set of points spanning the object's surface, and then the Delaunay algorithm is used to construct an adaptive tetrahedral mesh that wraps around the object.

[0039] Differentiable explicit surface extraction: In the embodiment, a learnable scalar parameter, the symbolic distance field (SDF) value, is defined at each Gaussian pivot point of the constructed adaptive tetrahedral mesh.

[0040] The Differentiable Marching Tetrahedra (DMTet) algorithm is used to detect tetrahedral edges where the SDF symbol is flipped. Zero-isotropy points are calculated using linear interpolation, and an explicit triangular mesh surface is generated by connecting the tetrahedral edges and zero-isotropy points. Then, a geometric consistency constraint is introduced to iteratively optimize the symbolic distance field of the explicit triangular mesh surface through gradient backpropagation. This forces the depth map obtained from rasterization of the explicit mesh to be consistent with the depth map obtained from Gaussian volume rendering, ensuring that the extracted mesh closely adheres to the geometry represented by the Gaussian cloud, thus dynamically extracting the explicit triangular mesh surface.

[0041] Based on the above process, geometric information is extracted.

[0042] S2. Sample on the surface of the explicit triangular mesh, predict the physical properties of each sampling point through the implicit material network, and provide ambient lighting through HDR ambient light map.

[0043] Physical property inversion: In this embodiment, the mesh topology is fixed at this stage, and an implicit material network based on a multilayer perceptron (MLP) and a learnable HDR ambient light cubemap are initialized, with a focus on optimizing material and lighting parameters. The implicit material network is used to predict the physical properties of the sampling points, and the HDR ambient light cubemap has a resolution of 6×512×512 and is randomly initialized to provide ambient lighting.

[0044] Among them, the implicit material network based on multilayer perceptron (MLP) adopts volumetric texture. The three-dimensional coordinates of sampling points on the explicit surface are input into the implicit material network based on multilayer perceptron. Through a series of fully connected layers and nonlinear activation functions, the output physical properties that follow the Disney PBR model are used for the coloring calculation of the sampling points. The physical properties include: base color, roughness, metallicity, and normal perturbation.

[0045] Traditional UV mapping requires pre-unfolding a 3D mesh onto a 2D plane and establishing a fixed correspondence between each vertex and texture pixel. If the mesh topology changes drastically during optimization (e.g., opening or closing holes, splitting or merging), the original UV mapping relationship becomes invalid, leading to severe texture misalignment, stretching, or tearing. This application, however, combines volumetric textures with a multilayer perceptron, where material properties are determined solely by 3D coordinates. Regardless of mesh deformation or topology changes, the coordinate query result for the same spatial point remains stable. Mesh vertices can move freely, and faces can be added or removed, but the coordinates of newly sampled points on the surface can always obtain reasonable and consistent material properties through the same multilayer perceptron, achieving complete decoupling of material and geometry. Furthermore, besides avoiding texture misalignment caused by mesh topology changes, using volumetric textures allows for stable and continuous propagation of material and geometric gradients in the backpropagation context, preventing discrete and non-differentiable jumps due to parameter changes, thus achieving complete decoupling of material and geometry.

[0046] S3. Using the geometric information, physical properties, and ambient lighting of each sampling point as input, a differentiable PBR renderer is used to generate a physically rendered image.

[0047] In this embodiment, to address the issue of large computational complexity in the lighting integral of the rendering equation, the present invention introduces a differentiable split-sum approximation rendering equation to approximate the specular reflection term. It is approximately the product of the pre-filtered environment map and the environment BRDF map, and the approximate calculation formula is as follows: ,in, For ambient light, For surface roughness, For the environment BRDF map lighting, For normal line, For the incident light direction, the left half (illumination part) calculates the result of the ambient light L being blurred by the surface roughness D; the right half (material part) calculates the reflection characteristics of the material itself, which are independent of the specific ambient light, and calculates the image photometric loss based on this.

[0048] This model allows the system to recover high-frequency environmental reflections (such as clear reflections of surrounding objects) with extremely low computational cost (requiring only two texture samples).

[0049] Furthermore, using the geometric information, physical properties, and ambient lighting of each sampling point as input, a pre-filtered environment map and an environment BRDF map are generated respectively. The pre-filtered environment map is obtained by performing differentiable convolution or filtering on the ambient lighting of the HDR environment map to generate a set of multi-level asymptotic textures to simulate specular reflection blur of different roughness. The environment BRDF map is calculated by integrating Fresnel terms and geometric terms, and is used to calculate and store the integral lookup table. In the real-time rendering stage, the specular reflection direction is calculated using the geometric information, physical properties, and ambient lighting of each sampling point as input. A multi-level asymptotic texture of the pre-filtered environment map is selected based on the roughness, and the pre-filtered environment map is sampled along the specular reflection direction to obtain the pre-filtered color. A two-dimensional vector is obtained based on the integral lookup table and the roughness sampling integral lookup table. The pre-filtered color, the two-dimensional vector, and the Fresnel term are multiplied to obtain the final specular reflection color. The final specular and diffuse colors are combined to generate a physically rendered image.

[0050] S4. Based on the difference between the generated physically rendered image and the real image, multiple loss functions are constructed for gradient backpropagation to optimize implicit material network parameters, HDR ambient light mapping, explicit triangular mesh surface and Gaussian splash image sequence, so as to achieve the coordinated optimization and inversion of physical properties and 3D geometry, and generate 3D reconstructed image.

[0051] In the embodiments, multiple loss functions are constructed to optimize the implicit material network parameters, HDR ambient light map parameters, SDF values ​​of explicit triangular mesh surfaces, and Gaussian splatter image sequences through stochastic gradient descent (SGD) backpropagation.

[0052] Multiple loss functions include volumetric rendering loss, geometric bidirectional consistency loss, and anti-corrosion regularization loss; Among them, volume rendering loss It provides gradient optimization at the appearance level to calculate the photometric error between the generated physically rendered image and the real image, and iteratively optimizes the implicit material network parameters and HDR ambient ray map parameters. This represents the mean absolute error of pixel color. Indicates structural difference loss. and These are weight parameters; Geometric bidirectional consistency loss It provides geometric gradient optimization to constrain the depth map obtained by explicit surface rasterization. Depth map obtained by Gaussian rendering The geometric shell remains consistent, ensuring that the explicit surface does not deviate from the visual shell of the Gaussian cloud. For loss weighting coefficients, It is an L1 norm; Corrosion resistance regularization loss It provides gradient optimization at the structural prior level to constrain the signed distance field value of all Gaussian spheres that serve as the center points of the Gaussian splash image sequence to be negative, thus preventing the thin-walled structure from disappearing during the optimization process. It is a function with maximum value. For the first The symbolic distance field value at the center point of a Gaussian sphere.

[0053] Finally, topology updates are stopped. The XAtlas algorithm is used to generate non-overlapping UV coordinates for the final mesh. A 2048×2048 resolution texture map is constructed. For each pixel, the implicit material network is queried based on its corresponding 3D surface coordinates, and the continuous neural material field is discretized and saved as a texture image. The final output is a complete digital asset containing a high-precision triangular mesh, PBR material map, and HDR ambient occlusion map, which can be directly used in industrial graphics engines.

[0054] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for physical property inversion and 3D reconstruction based on Gaussian splashing and differentiable rendering, characterized in that, Includes the following steps: Sparse point clouds are generated based on multi-view images. A set of Gaussian spheres is initialized as a Gaussian splash image sequence based on the sparse point clouds. The contribution of each Gaussian splash image sequence is calculated using Gaussian splash rendering to filter Gaussian pivot points. An adaptive tetrahedral mesh is constructed based on the Gaussian pivot points. The surface of the explicit triangular mesh is dynamically extracted from the adaptive tetrahedral mesh to obtain geometric information. Sampling is performed on the surface of an explicit triangular mesh, and the physical properties of each sampling point are predicted through an implicit material network. At the same time, ambient lighting is provided through an HDR ambient light map. Using the geometric information, physical properties, and ambient lighting of each sampling point as input, a differentiable PBR renderer is used to generate physically rendered images. Based on the differences between the generated physically rendered image and the real image, multiple loss functions are constructed for gradient backpropagation to optimize implicit material network parameters, HDR ambient light maps, explicit triangular mesh surfaces, and Gaussian splash image sequences, thereby achieving the synergistic optimization and inversion of physical properties and 3D geometry to generate 3D reconstructed images.

2. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 1, characterized in that, The method for generating sparse point clouds based on multi-view images includes: using the structure-of-motion algorithm to perform feature matching on multi-view images, calculating the camera intrinsic and extrinsic parameter matrices for each image, and obtaining sparse point clouds.

3. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 1, characterized in that, The parameters that can be optimized in a Gaussian splatter image sequence include: position vector, rotation quaternion, scaling vector, opacity scalar, and spherical harmonic coefficients.

4. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 1, characterized in that, The method of calculating the contribution of each Gaussian splash image sequence using Gaussian splash rendering to filter Gaussian pivot points includes: The average contribution of Gaussian spheres as a Gaussian splash image sequence is calculated using Gaussian splash rendering. The average contribution is the mean of the transmittance-weighted opacity of each Gaussian sphere as a Gaussian splash image sequence across all viewpoints. The average contribution is sorted, and the top N are selected as the initial Gaussian spheres for the Gaussian splash image sequence to establish a Gaussian set. For each Gaussian sphere in the Gaussian set as the initial Gaussian sphere for the Gaussian splash image sequence, the principal axis direction is calculated based on the covariance matrix. Multiple corner points extending along the principal axis direction to the bounding box boundary and a Gaussian sphere center point are determined, and the Gaussian pivot point is selected.

5. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 4, characterized in that, The method of constructing an adaptive tetrahedral mesh based on Gaussian pivot points includes: By combining the Gaussian splash image sequence parameters optimized by gradient backpropagation, the selected Gaussian pivot points are adaptively updated. Based on the updated Gaussian pivot points, an adaptive tetrahedral mesh is constructed using Delaunay triangulation.

6. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 5, characterized in that, Dynamically extract explicit triangular mesh surfaces from adaptive tetrahedral meshes, including: At each Gaussian pivot point of the constructed adaptive tetrahedral mesh, a signed distance field value is defined; Using the differentiable tetrahedron traveling algorithm, zero isovalue points are calculated by linear interpolation on the tetrahedron edges where the sign distance field value crosses zero. An explicit triangular mesh surface is generated based on the tetrahedron edges and the zero isovalue points. The symbolic distance field value of the explicit triangular mesh surface is iteratively optimized based on gradient backpropagation, and the explicit triangular mesh surface is dynamically extracted.

7. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 6, characterized in that, The implicit material network is constructed by initializing an implicit material network based on a multilayer perceptron. Based on volumetric texture, the three-dimensional coordinates of sampling points on the surface of the explicit triangular mesh are input into the implicit material network based on the multilayer perceptron. Through a series of fully connected layers and nonlinear activation functions, physical properties are output for the shading calculation of the sampling points. The physical properties include: base color, roughness, metallicity, and normal perturbation.

8. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 1, characterized in that, When generating physically rendered images using a differentiable PBR renderer, the Split-Sum approximation is introduced to solve the rendering equations, specifically to account for specular reflection terms. It is approximately the product of the pre-filtered environment map and the environment BRDF map, and the approximate calculation formula is as follows: ,in, For ambient light, For surface roughness, For the environment BRDF map lighting, For normal line, The direction of the incident light.

9. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 8, characterized in that, The method of generating physically rendered images using a differentiable PBR renderer includes: Using geometric information, physical properties, and ambient lighting as inputs, a pre-filtered environment map and an environment BRDF map are generated, respectively. The pre-filtered environment map is obtained by performing differentiable convolution or filtering on the ambient lighting of the HDR environment map to generate a set of multi-level asymptotic textures to simulate specular reflection blur of different roughness. The environment BRDF map is calculated by integrating Fresnel terms and geometric terms, and is used to calculate and store an integral lookup table. In the real-time rendering stage, the specular reflection direction is calculated using the geometric information, physical properties, and ambient lighting of each sampling point as input. A multi-level asymptotic texture of the pre-filtered environment map is selected based on the roughness, and the pre-filtered environment map is sampled along the specular reflection direction to obtain the pre-filtered color. A two-dimensional vector is obtained based on the integral lookup table and the roughness sampling integral lookup table. The pre-filtered color, the two-dimensional vector, and the Fresnel term are multiplied to obtain the final specular reflection color. The final specular and diffuse colors are combined to generate a physically rendered image.

10. The physical property inversion and 3D reconstruction method based on Gaussian splashing and differentiable rendering according to claim 9, characterized in that, The constructed loss functions include volumetric rendering loss, geometric bidirectional consistency loss, and anti-corrosion regularization loss; Among them, volume rendering loss It provides gradient optimization at the appearance level to calculate the photometric error between the generated physically rendered image and the real image, and iteratively optimizes the implicit material network parameters and HDR ambient ray map parameters. This represents the mean absolute error of pixel color. Indicates structural difference loss. and These are weight parameters; Geometric bidirectional consistency loss It provides geometric gradient optimization to constrain the depth map obtained by explicit surface rasterization. Depth map obtained by Gaussian rendering The geometric shell remains consistent, ensuring that the explicit surface does not deviate from the visual shell of the Gaussian cloud. For loss weighting coefficients, It is an L1 norm; Corrosion resistance regularization loss It provides gradient optimization at the structural prior level to constrain the signed distance field value of all Gaussian spheres that serve as the center points of the Gaussian splash image sequence to be negative, thus preventing the thin-walled structure from disappearing during the optimization process. It is a function with maximum value. For the first The symbolic distance field value at the center point of a Gaussian sphere.

Images