Method for transparent object reconstruction and novel view synthesis based on 3DGS

Abstract

The application relates to a 3DGS-based transparent object reconstruction and new view synthesis method, which comprises the following steps: S1. reconstructing a dense three-dimensional point cloud of a data set and camera pose information according to an image sequence; S2. using a three-dimensional point cloud and camera pose information to reconstruct a three-dimensional scene by using a 3DGS method, and generating a three-dimensional Gaussian kernel to represent an input scene; S3. introducing a Sobel edge detector, and respectively extracting edges of transparent objects in a rendered image generated by the three-dimensional Gaussian kernel and a real image, and calculating an edge loss function of a rendered edge map and a real edge map; S4. combining the edge loss function and an original 3DGS loss function to jointly optimize a three-dimensional scene reconstruction model; and S5. overall optimizing the scene and outputting a final three-dimensional reconstruction result. The application aims to improve the reconstruction quality of a 3DGS model, and further improve the three-dimensional reconstruction effect of transparent objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition processing, in particular to a transparent object reconstruction and new view synthesis method based on 3DGS. BACKGROUND

[0002] Three-dimensional reconstruction of transparent objects has long been a challenge in computer vision due to their unique optical properties such as reflection, refraction, and scattering. Traditional methods, such as those based on structured light, multi-view stereo matching, or depth sensors, cannot effectively handle the complexity of transparent objects. These techniques are often highly sensitive to lighting conditions and are easily disturbed by noise and errors, especially in dynamic scenes or highly reflective environments. In addition, their dependence on view changes further limits their practical applications.

[0003] In recent years, the 3D Gaussian Splatting (3DGS) method has shown significant potential in three-dimensional scene reconstruction. 3DGS represents point clouds in three-dimensional space using Gaussian distributions, achieving a good balance between model accuracy and computational efficiency. However, although 3DGS performs well in sparse data reconstruction and global modeling of complex scenes, its design is not optimized for the special optical properties of transparent objects. Therefore, when faced with transparent and semi-transparent objects, 3DGS still has limited performance and is difficult to solve the complex effects caused by refraction or multiple paths of light. SUMMARY

[0004] The purpose of the present application is to provide a transparent object reconstruction and new view synthesis method based on 3DGS, which can improve the reconstruction quality of 3DGS models and further improve the three-dimensional reconstruction effect of transparent objects.

[0005] The purpose of the present application is achieved by the following technical solution: a transparent object reconstruction and new view synthesis method based on 3DGS, comprising the following steps:

[0006] S1. Reconstruct a dense three-dimensional point cloud of the data set and camera pose information from the image sequence;

[0007] S2. Use the three-dimensional point cloud and camera pose information to reconstruct the three-dimensional scene using the 3D Gaussian Splatting method, generate a three-dimensional Gaussian kernel to represent the input scene;

[0008] S3. Introduce a Sobel edge detector and extract the edges of the transparent objects in the rendered image generated by the three-dimensional Gaussian kernel and the real image, respectively, and calculate the edge loss function of the rendered edge map and the real edge map;

[0009] S4. Combine the edge loss function and the original 3D Gaussian Splatting loss function to optimize the three-dimensional scene reconstruction model together;

[0010] S5. Overall optimization is performed on the scene, and the final three-dimensional reconstruction result is output.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] 1. The present application proposes an edge loss function L constructed based on a Sobel edge detector contour , which makes up for the shortcomings of the original 3DGS loss function in processing the edge features of transparent objects; by capturing the edge details of transparent objects and introducing edge constraints in loss optimization, the reconstruction effect of the rendered image and the real image in the transparent area is more accurate, significantly improving the edge expressiveness and reconstruction quality of transparent objects.

[0013] 2. The present application uses DetectorFreeSfM model instead of Colmap, which eliminates the dependence on sparse key point detection, thereby significantly improving the performance of transparent objects in the initial point cloud generation stage; the RoMa feature matcher is introduced to replace the LoFTR used by DetectorFreeSfM, through its more robust feature extraction and matching strategy, the feature matching accuracy of the transparent object area is improved, further optimizing the integrity and quality of the point cloud generation. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of a transparent object reconstruction and new view synthesis method based on 3DGS according to the present application.

[0015] Figure 2 is a comparison chart of point cloud files generated by traditional Colmap and improved DetectorFreeSfM.

[0016] Figure 3 is a result chart of edge recognition obtained by using Sobel edge detector on transparent objects.

[0017] Figure 4 is a three-dimensional reconstruction result of a transparent object reconstruction and new view synthesis method based on 3DGS according to the present application, and a comparison chart with existing methods. DETAILED DESCRIPTION

[0018] The content of the present application will be described in detail below in conjunction with the drawings and examples in the specification:

[0019] As Figures 1-4 shown is an embodiment schematic diagram of a transparent object reconstruction and new view synthesis method based on 3DGS according to the present application.

[0020] A transparent object reconstruction and new view synthesis method based on 3DGS, comprising the following steps:

[0021] Step S1. According to the image sequence, a dense three-dimensional point cloud of the data set and camera pose information are reconstructed.

[0022] The dense three-dimensional point cloud of the data set and the camera pose information are reconstructed according to the image sequence using the improved DetectorfreeSfM model. The feature detection-free SfM model DetectorFreeSfM can directly generate efficient dense point clouds without relying on feature point detection. The precision of the present application is optimized, and the LoFTR feature matcher in the original model is replaced with the RoMa feature matcher. The multi-scale feature enhancement and transformation invariance matching strategy of RoMa are used to improve the accuracy and robustness of the generated point cloud.

[0023] Compared with LoFTR, RoMa combines DINOv2 frozen coarse features and a dedicated fine feature pyramid structure, significantly improving the robustness and local accuracy of feature matching, especially under complex optical conditions of transparent objects. At the same time, the multi-view optimization and iterative refinement strategy of DetectorFreeSfM ensures the high integrity of the generated point cloud, providing solid support for accurate modeling of transparent objects. The generated point cloud diagram and comparison diagram with the traditional Colmap model are as shown in Figure 2 .

[0024] Step S2. Using the three-dimensional point cloud and camera pose information, a three-dimensional scene is reconstructed using the 3D Gaussian Splatting method, and a three-dimensional Gaussian kernel is generated to represent the input scene.

[0025] 3D Gaussian Splatting (3DGS) is an innovative three-dimensional reconstruction technology that represents the surface characteristics of an object through a Gaussian point cloud. Each point is represented as a Gaussian kernel containing position μ, covariance matrix ∑, transparency α, and spherical harmonic function information SH. During rendering, the Gaussian kernel can be projected as a two-dimensional ellipse, achieving fast and accurate color synthesis. The color calculation formula is:

[0026]

[0027] Where C represents color, c i represents the color information on each Gaussian kernel, and α i represents the transparency of each Gaussian kernel. The color of the overall image is obtained by calculating the color of each pixel and finally splicing these color values to form the rendered image I pred .

[0028] Step S3. Introduce Sobel edge detector, and respectively extract the edges of the transparent object in the rendered image generated by the three-dimensional Gaussian kernel and the real image, and calculate the edge loss function of the rendered edge map and the real edge map.

[0029] The present application applies Sobel operator to real image and 3D GS rendered image respectively, generates corresponding edge map, and the edge map of real image is used to represent the edge feature of transparent object, while the edge map of rendered image is used as the basis for comparison, for subsequent loss calculation and optimization.

[0030] Rendered image I pred and rendered edge map E pred The conversion formula is:

[0031]

[0032] Real image I gt and real edge map E gt The conversion formula is:

[0033]

[0034] Wherein, is the gradient of the real image in the x direction, is the gradient of the rendered image in the x direction;

[0035] Wherein, is the gradient of the real image in the y direction, is the gradient of the rendered image in the y direction.

[0036] The calculation formula of the edge loss function L contour of the rendered edge map and the real edge map is:

[0037]

[0038] Wherein, M represents the total number of pixels in the image, E pred (i) represents the edge intensity of the rendered edge map at the i th pixel, E gt (i) represents the edge intensity of the real edge map at the i th pixel.

[0039] Step S4. Combine the edge loss function and the original 3D Gaussian Splatting loss function to jointly optimize the three-dimensional scene reconstruction model.

[0040] The original 3D Gaussian Splatting loss function includes reconstruction consistency loss function L1 and SSIM loss function L SSIM , and the total loss function for jointly optimizing the three-dimensional scene reconstruction model is:

[0041] L total =λ1L1+λ2L ssim +λ3L contour .

[0042] The values ​​of λ1, λ2, and λ3 are 0.6, 0.2, and 0.2 respectively.

[0043] Margin loss and global loss (such as L1 and L SSIM ) complement each other, the global loss optimizes the overall consistency of the image, while the edge loss focuses on local details, thereby improving the reconstruction quality of transparent objects at different levels.

[0044]

[0045] L SSIM =1-SSIM(I pred , I gt )

[0046] I pred (i) I gt (i) represents the pixel value of the rendered image and the real image at the i-th pixel; SSIM(I pred , I gt ) represents the structural similarity index between the rendered image and the real image, and its specific calculation formula is:

[0047]

[0048] Where x and y are two images; μ x 、μ y are the average brightness of image x and image y respectively; and is the variance of images x and y, indicating contrast; σ xy is the covariance of image x and y, indicating structural information; C1 and C2 are constants added to stabilize the calculation. Usually for 8-bit images, C1 and C2 are 650.25 and 5808.75 respectively.

[0049] Step S5: Optimize the scene as a whole and output the final 3D reconstruction result.

[0050] Through the efficient rendering capabilities of 3DGS and the optimized 3D model, realistic image synthesis of transparent objects from new perspectives can be achieved.

[0051] The present application aims at the problem that the traditional 3D Gaussian Splatting method using Colmap to generate an initial point cloud performs poorly on low-texture transparent objects, and proposes an improved method based on the DetectorFreeSfM model. By introducing the RoMa feature matcher instead of LoFTR, the robustness of feature matching and the accuracy of point cloud generation of transparent objects are enhanced. At the same time, the present application designs an edge loss function based on the Sobel edge detector, which makes up for the deficiency of the original loss function in handling the edge details of transparent objects. Through the edge comparison of the real image and the rendered image, the model performance is optimized, and the reconstruction quality and overall effect of the transparent object region are significantly improved.

[0052] The above-described embodiments are merely preferred embodiments of the present application and are not intended to limit the concept and scope of the present application. Various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art without departing from the design concept of the present application shall fall within the protection scope of the present application, and the technical content claimed by the present application has been fully recorded in the claims.

Claims

1. A method for 3D GS based transparent object reconstruction and novel view synthesis, characterized in that, The method comprises the following steps: S1. Reconstructing a dense three-dimensional point cloud of the data set and camera pose information according to the image sequence; S2. Using the three-dimensional point cloud and the camera pose information to reconstruct a three-dimensional scene using a 3D Gaussian Splatting method, and generating a three-dimensional Gaussian kernel to represent the input scene; S3. Introduce Sobel edge detector, and extract the edges of the transparent object in the rendered image generated by the three-dimensional Gaussian kernel and the real image respectively, and calculate the edge loss function of the rendered edge map and the real edge map ; S4. Combining edge loss function and the original 3D Gaussian Splatting loss function, jointly optimize the three-dimensional scene reconstruction model; S5. Overall optimizing the scene and outputting a final three-dimensional reconstruction result; In step S3, the rendered image The conversion formula for the rendered edge map is ； real image with real edge map The conversion formula is: ； wherein is a gradient of the real image in the x direction, is a gradient of the rendered image in the x direction; is a gradient of the real image in the y direction, is a gradient of the rendered image in the y direction; Edge loss function of rendering edge map and real edge map The calculation formula is: ； where M represents the total number of pixels in the image, represents the edge strength of the rendered edge map at the i-th pixel, represents the edge strength of the true edge map at the i-th pixel.

2. The 3DGS-based transparent object reconstruction and new view synthesis method according to claim 1, wherein: In step S4, the original 3D Gaussian Splatting loss function includes a reconstruction consistency loss function and an SSIM loss function and the total loss function for jointly optimizing the three-dimensional scene reconstruction model is: 。 3. The 3DGS-based transparent object reconstruction and new view synthesis method according to claim 2, wherein: , , the values of a, b and c are respectively 0.6, 0.2, 0.

2.

4. The 3DGS-based transparent object reconstruction and new view synthesis method according to claim 1, wherein: In step S1, a DetectorfreeSfM model is used to reconstruct a dense three-dimensional point cloud of the data set and camera pose information according to the image sequence, and a RoMa feature matcher is used to replace the LoFTR feature matcher in the original model to generate an initial point cloud.

Images