Joint reconstruction method of refractive surface and underwater scene based on three-dimensional gaussian ray tracing

Abstract

This invention discloses a method for joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing, belonging to the field of computer vision and 3D reconstruction technology. By designing a hybrid water surface representation method, the water surface is represented as a neural height field; refraction calculations are performed to obtain the corresponding rays after refraction through the water surface; the underwater scene is modeled as a 3D Gaussian field for underwater scene rendering; a multi-view image supervision model is constructed, a 3D reconstruction loss function is designed, and end-to-end joint optimization is performed; thus, joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing is achieved. This invention achieves high-precision reconstruction while ensuring fast training speed and real-time new perspective rendering frame rate, solving the technical problems of high computational cost, multi-view geometric inconsistency, and inability to efficiently extract and separate water surface and underwater geometric structures in existing underwater scene 3D reconstruction methods.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, and relates to computer vision neural rendering and new perspective synthesis technology, especially to a method for joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing. Background Technology

[0002] Novel view synthesis (NVS) and 3D reconstruction of underwater scenes from above-water perspectives have important applications in shallow sea topographic mapping, environmental monitoring, and other fields. However, due to the nonlinear light refraction at the water-air interface and the complex wave geometry of the water surface, which often exhibits dynamic undulations, the linear propagation assumption of standard multi-view geometry is severely violated, making it difficult to achieve 3D reconstruction of scenes involving water surface refraction.

[0003] Conventional 3D reconstruction methods, such as Neural Radiation Field (NeRF) and 3D Gaussian Sputtering (3DGS), have achieved great success in synthesizing novel perspectives in air. However, these methods assume that light travels in straight lines and perform poorly in scenes involving water refraction. Current work on reconstruction methods designed for refracting objects (such as NU-NeRF and TransparentGS) lacks efficient modeling of water surfaces, making them difficult to apply to water refraction scenes. Existing methods specifically designed for water refraction scenes (such as NeRFrac) lack accurate water surface representations consistent across multiple perspectives and efficient, differentiable visual rendering strategies, resulting in poor performance in both speed and quality for 3D underwater scene reconstruction. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, this invention provides a method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing, which solves the technical problems of high computational cost, multi-view geometric inconsistency, and inability to efficiently extract and separate the geometric structures of the water surface and underwater scenes in existing three-dimensional underwater scene reconstruction methods.

[0005] According to Embodiment 1 of the present invention, a method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing is provided. The execution of this method includes four stages, including:

[0006] 1) Design a hybrid water surface representation method to model the water surface and represent it as a neural height field.

[0007] Representing the water surface as a neural height field The height field is parameterized by a multilayer perceptron (MLP), providing a continuous, smooth, and globally consistent representation of the surface geometry. To accelerate the calculation of ray intersections with this implicit height field, a proxy mesh is introduced as a piecewise linear approximation. Given a 2D triangular mesh... ,in A two-dimensional point set representing a grid. Represents the edge set. The height is obtained by querying the height field. The z-coordinate of each 2D point in the two-dimensional point set is obtained, and each 2D point is transformed into a 3D point, thereby promoting the 2D mesh into a 3D proxy mesh.

[0008] 2) Refraction calculation.

[0009] For each ray emitted from a pixel, calculate the corresponding ray (refracted ray) after the ray is refracted by the water surface.

[0010] To improve the modeling accuracy of complex interfaces while maintaining efficiency, a recursive subdivision tracing algorithm from coarse to fine is proposed to calculate the intersection depth *t* of a ray and the water surface. First, the ray performs a global intersection calculation with a coarse 3D proxy mesh, removes non-intersecting vertices, and subdivides the intersecting triangle into four sub-triangles. The depth *t* is then calculated by querying the height field. Update the position of the newly added vertex. Perform intersection calculations on the subdivided local triangles, and after multiple iterations, return the final intersection depth t. Then, calculate the world coordinates of the intersection as the starting point of the refracted ray. Next, use barycentric coordinate interpolation to calculate the smooth normal, and calculate the direction of the refracted ray according to Snell's law.

[0011] Refraction calculations include the following process:

[0012] 21) Extract explicit triangular meshes from parameterized water surface (neural height field) ,in Indicates by The resulting 3D point set;

[0013] 22) Construct a hierarchical bounding box (BVH) for the triangular mesh, and perform a global intersection test with the triangular mesh for each ray using the BVH;

[0014] Further local subdivision and intersection can be performed; local subdivision and intersection can be performed recursively multiple times until the required accuracy is achieved, and finally the intersection depth of each ray is obtained, representing the three-dimensional coordinates of the intersection point;

[0015] 23) Calculate the normal line at the intersection point using the barycentric coordinate interpolation method;

[0016] 24) Calculate the refracted ray, including the starting point and direction of the refracted ray;

[0017] 3) Underwater Scene Rendering. The underwater scene is modeled as a 3D Gaussian field, and the Gaussian ray tracing method is used for underwater scene rendering. First, the intersection of each refracted ray with the underwater Gaussian field is calculated. Then, alpha mixing and color accumulation are performed sequentially to obtain the final rendering result.

[0018] 4) Construct a multi-view image supervision model, design a 3D reconstruction loss function, and perform end-to-end joint optimization.

[0019] The system performs end-to-end joint optimization under multi-view image supervision. The gradient of the rendered color is backpropagated not only to the underwater Gaussian parameters (position, opacity, spherical harmonic coefficients, etc.) but also to the origin and direction of the refracted light rays, thereby updating the weights of the water surface height field MLP. Simultaneously, an opacity loss is used to penalize high-opacity floaters generated in sparse view regions, stabilizing the optimization process. The total loss function is a weighted sum of L1 loss, SSIM loss, and opacity loss.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] This invention provides a method for joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing. It reconstructs both the water surface and the underwater scene from multi-view images captured above the water. This invention achieves high-precision reconstruction while maintaining fast training speed and real-time new-view rendering frame rate. It addresses the technical problems of existing 3D underwater scene reconstruction methods, such as high computational cost, inconsistent geometry across multiple views, and the inability to efficiently extract and separate the surface and underwater geometric structures. Attached Figure Description

[0022] Figure 1 A flowchart is provided for the method of this invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] Example 1:

[0028] A method for joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing. It includes the following steps:

[0029] 1) Design a hybrid water surface representation method to model the water surface:

[0030] Accurate water surface modeling is crucial for reconstructing scenes involving refraction caused by water surfaces. This invention proposes a hybrid water surface representation method that combines the advantages of implicit neural representation (continuous, smooth, and differentiable) with the advantages of explicit triangular meshes (facilitating rapid intersection of rays).

[0031] Specifically, this scheme defines the water surface as a parameterized surface:

[0032]

[0033] in Represents the three-dimensional coordinates of any point on the surface. It is implemented using a multilayer perceptron (MLP).

[0034] Representing the water surface using an MLP (Mean Light Logic Pyramid) can fully utilize the smooth and continuous characteristics of the MLP; however, this implicit representation also brings difficulties to finding the intersection between rays and the water surface. To accelerate the ray-water surface intersection, this scheme extracts an explicit triangular mesh from the implicitly represented water surface before the ray-water surface intersection is determined.

[0035] Specifically, this solution will first generate a two-dimensional... Uniform grid The coordinates are then fed into the MLP to obtain the z-coordinate of each vertex on the mesh, thereby transforming the 2D mesh into a 3D mesh. This serves as an explicit piecewise linear approximation of the original implicit water surface representation.

[0036] 2) Refraction calculation

[0037] For rays emitted from each pixel of the target image that needs to be rendered ,in Indicates the starting point of the ray. To indicate the direction of rays, it is necessary to calculate the corresponding rays after refraction through the water surface. ,in Indicates the starting point of the refracted ray. This indicates the direction of the refracted ray. This scheme uses the following strategy to calculate the refracted ray:

[0038] First, as described in 1), an explicit triangular mesh is extracted from the parameterized water surface. Next, a Bounding Volume Hierarchy (BVH) is constructed for the triangular mesh, for each ray. All methods utilize BVH to perform global intersection tests with the triangular mesh. Due to the high computational cost of constructing a complete triangular mesh and its BVH, the triangular mesh extracted directly from the parametric water surface is set to a lower precision. To perform more accurate intersection calculations, this scheme performs local subdivision intersection calculations after the global intersection test. Specifically, for each triangle hit by a ray, midpoint subdivision is performed, dividing the triangle into four sub-triangles. The newly added vertices from the subdivision are then fed into the MLP water surface to correct the approximation error of the z-coordinate. For each ray intersecting the larger triangle before subdivision, intersection calculations are performed only on the four smaller triangles corresponding to the subdivision. Local subdivision intersection calculations can be performed recursively multiple times until the desired precision is achieved, ultimately obtaining the intersection depth t of each ray. Represents the three-dimensional coordinates of the intersection point.

[0039] Next, the normals at the intersection points need to be calculated. To avoid rendering quality degradation due to discontinuities in the normals at the triangle boundaries, this solution uses centroid interpolation to calculate the normals at the intersection points. :

[0040]

[0041] in, Let the coordinates of the centroid of the triangle satisfy... , The normals at the three vertices of the triangle are calculated by averaging the normals of adjacent faces.

[0042] Finally, the refracted rays were calculated according to Snell's law. The intersection of the ray and the triangular mesh is used as the starting point of the refracted ray:

[0043]

[0044] The direction of the refracted ray is calculated using the vector form of Snell's law:

[0045]

[0046] in

[0047] 3) Underwater scene rendering

[0048] This step utilizes the refracted rays generated in step 2) to perform ray tracing rendering on the 3D Gaussian primitives representing the underwater scene, thereby synthesizing the final pixel colors. The specific implementation is as follows:

[0049] The underwater scene is modeled as a set of anisotropic 3D Gaussian elements, each represented by an ellipsoid following a 3D Gaussian distribution. Along the calculated direction of the refracted ray... From the starting point of refraction Begin by allowing light to pass through and traverse the relevant Gaussian elements in the underwater scene. For each Gaussian element traversed by the refracted ray, accumulate color and opacity along the refracted ray path; these accumulated render colors will be used in subsequent joint loss calculations.

[0050] 4) Construct a multi-view image supervision model, design a 3D reconstruction loss function, and perform end-to-end joint optimization;

[0051] This step aims to calculate the loss function. The specific implementation is as follows:

[0052] First, a multi-view image supervision model is constructed, which compares the rendered image color C with the color of the ground truth (GT) image. Compare the results and calculate the reconstruction loss. The main monitoring signals include L1 loss. and structural similarity loss .

[0053] Secondly, opacity regularization is introduced. To address the issue of high-opacity floaters easily generated in sparse view regions, leading to occlusion and gradient vanishing, an additional opacity loss term is introduced. The loss is a global penalty applied to the high opacity values ​​of all N Gaussian elements:

[0054]

[0055] in Let represent the opacity attribute of the i-th Gaussian element, and N be the total number of Gaussian elements.

[0056] This regularization term effectively encourages transparency and stabilizes the optimization process. The total loss function of the system is defined as the weighted sum of the above terms:

[0057]

[0058] in The weight hyperparameters are set.

[0059] This scheme calculates the gradient of the loss function using a bidirectional backpropagation path. The first path is used for underwater scene updates, and the total loss is calculated using the path described above. The color gradient is propagated back along each refracted ray to all 3D Gaussian primitives that contribute to that pixel, thereby updating and optimizing the Gaussian parameters of the underwater scene. The second path is used for updating the refracted water surface. Unlike conventional radiative field rendering, this scheme further calculates the rendered color relative to the origin of the refracted ray. (i.e., intersection point p) and refraction direction The gradients of these ray levels are then backpropagated to the water surface model using Snell's law and the formula for finding the intersection of rays and the proxy mesh, ultimately updating the neural height field. The weights of the MLP network.

[0060] The above embodiment provides a method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing, which has broad application prospects.

[0061] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing, characterized in that, Includes the following steps: 1) Design a hybrid water surface representation method to model the water surface and represent it as a neural height field; The height field is parameterized by a multilayer perceptron, providing a continuous, smooth, and globally consistent representation of surface geometry; Before the intersection of light and water surfaces, a triangular mesh is extracted from the water surface represented by the neural height field and promoted from a two-dimensional mesh to a three-dimensional proxy mesh. 2) Perform refraction calculation: For each ray emitted from a pixel, calculate the corresponding ray after refraction through the water surface; Specifically, a recursive subdivision tracking algorithm from coarse to fine is designed to calculate the depth of the intersection point between the ray and the water surface; the world coordinates of the intersection point are calculated as the starting point of the refracted ray; then, the smooth normal is calculated using centroid coordinate interpolation, and the direction of the refracted ray is calculated according to Snell's law. 3) Model the underwater scene as a 3D Gaussian field and then render the underwater scene; 4) Construct a multi-view image supervision model, design a 3D reconstruction loss function, and perform end-to-end joint optimization; including: First, a multi-view image supervision model is constructed, and the colors of the rendered images are compared with the colors of the real ground images to calculate the reconstruction loss; the main supervision signals include L1 loss and structural similarity loss. Secondly, opacity regularization, or opacity loss, is introduced; the opacity loss globally penalizes the high opacity values ​​of all N high-level primitives. The total loss function is defined as the weighted sum of the losses of each individual item; The bidirectional backpropagation path for calculating the gradient of the loss function is used. The first path is used for underwater scene updating, where the color gradient of the total loss is backpropagated along each refracted ray to all 3D Gaussian primitives that contribute to that pixel, in order to update and optimize the Gaussian parameters of the underwater scene. The second path is used for refraction surface updating, further calculating the gradient of the rendered color relative to the starting point (intersection point) and refraction direction of the refracted ray. These gradients at the ray level are then backpropagated to the water surface model according to Snell's law and the formula for finding the intersection of rays and the 3D proxy mesh, updating the weights of the multilayer perceptron network in the neural height field. This enables the joint reconstruction of refractive surfaces and underwater scenes based on 3D Gaussian ray tracing.

2. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 1, characterized in that, In step 1), a two-dimensional uniform grid is first generated, and all coordinates are fed into a multilayer perceptron to obtain the third-dimensional coordinates of each vertex on the grid, thereby upgrading the two-dimensional grid into a three-dimensional grid as an explicit piecewise linear approximation of the original implicit water surface representation.

3. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 1, characterized in that, In step 2), the recursive subdivision tracking algorithm from coarse to fine includes: First, the ray performs a global intersection calculation with the 3D proxy mesh, removes non-intersecting vertices, and subdivides the intersecting triangles into four sub-triangles; the position of the newly added vertex is updated by querying the neural height field; cross calculation is performed on the subdivided local triangles, and the final intersection depth is returned after multiple iterations.

4. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 1, characterized in that, Refraction calculations include the following steps: 21) Extract explicit triangular meshes from the parameterized water surface, i.e., the neural height field. ,in Indicates by The resulting 3D point set; 22) Construct a hierarchical bounding box (BVH) for the triangular mesh, and perform a global intersection test with the triangular mesh for each ray using the BVH; 23) Calculate the normal line at the intersection point using the barycentric coordinate interpolation method; 24) Calculate the refracted ray, including the starting point and direction of the refracted ray.

5. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 4, characterized in that, In step 22), further local subdivision and intersection are performed; the local subdivision and intersection are recursively performed multiple times until the required accuracy is achieved, and the intersection depth of each ray is obtained, representing the three-dimensional coordinates of the intersection point.

6. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 4, characterized in that, Step 23) Calculate the normal at the intersection point using centroid interpolation. , is represented as: in, Let the coordinates of the centroid of the triangle satisfy... , The normals at the three vertices of the triangle are calculated by averaging the normals of adjacent faces.

7. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 6, characterized in that, Calculate the refracted rays according to Snell's law. ,in and The starting point and direction of the refracted ray are respectively indicated; the intersection of the ray and the triangular mesh is used as the starting point of the refracted ray. The direction of the refracted ray is calculated using the vector form of Snell's law, expressed as: in , Indicates the refractive index of water. This represents the direction vector of the incident ray.

8. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 1, characterized in that, Step 3) Render the underwater scene. First, find the intersection of each refracted ray with the underwater Gaussian. Then, perform α-mixing and color accumulation in sequence to obtain the final rendering result.

9. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 1, characterized in that, In step 4), the total loss function is expressed as: in, , , All are set weight hyperparameters; L1 loss is Structural similarity SSIM loss is C represents the color of the rendered image. The colors are those of a real-world GT image of the ground. This is due to loss of opacity.

10. The method for joint reconstruction of refractive surfaces and underwater scenes based on three-dimensional Gaussian ray tracing as described in claim 9, characterized in that, The loss of opacity is expressed as: in Let represent the opacity attribute of the i-th Gaussian element, and N be the total number of Gaussian elements.

Images