Transparent object reconstruction and new view angle synthesis method based on 3DGS

Abstract

The invention relates to a transparent object reconstruction and new view angle synthesis method based on 3DGS. The transparent object reconstruction and new view angle synthesis method comprises the following steps: S1, reconstructing dense three-dimensional point cloud and camera pose information of a data set according to an image sequence; s2, reconstructing a three-dimensional scene by using the three-dimensional point cloud and the camera pose information and using a 3DGS method, and generating a three-dimensional Gaussian kernel to represent an input scene; s3, introducing a Sobe l edge detector, extracting edges of transparent objects in the rendered image and the real image generated by the three-dimensional Gaussian kernel, and calculating edge loss functions of a rendered edge image and a real edge image; s4, jointly optimizing the three-dimensional scene reconstruction model by combining the edge loss function and the original 3DGS loss function; and S5, performing overall optimization on the scene and outputting a final three-dimensional reconstruction result. The objective of the invention is to improve the reconstruction quality of the 3DGS model so as to improve the three-dimensional reconstruction effect of the transparent object.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and processing, and in particular to a transparent object reconstruction and new viewing angle synthesis method based on 3DGS. Background Art

[0002] 3D reconstruction of transparent objects has long been a challenge in computer vision due to their unique optical properties, such as reflection, refraction, and scattering. Traditional methods, such as those based on structured light, multi-view stereo matching, or depth sensors, cannot effectively cope with the complexity of transparent objects. These techniques are usually highly sensitive to lighting conditions, susceptible to noise and errors, and perform poorly in dynamic scenes or highly reflective environments. In addition, their reliance on viewpoint changes further limits their practical applications.

[0003] In recent years, the 3D Gaussian Splatting (3DGS) method has shown significant potential in 3D scene reconstruction. 3DGS can achieve a good balance between model accuracy and computational efficiency by representing point clouds in 3D space with Gaussian distribution. However, although 3DGS performs well in the reconstruction of sparse data and global modeling of complex scenes, its original design was not optimized for the special optical properties of transparent objects. Therefore, when facing transparent and translucent objects, 3DGS still faces the problem of limited performance and has difficulty in solving complex effects caused by refraction or multiple paths of light. Summary of the invention

[0004] The purpose of the present invention is to provide a transparent object reconstruction and new perspective synthesis method based on 3DGS, which is used to improve the reconstruction quality of the 3DGS model and further improve the three-dimensional reconstruction effect of transparent objects.

[0005] The object of the present invention is achieved through the following technical solution: A method for reconstructing transparent objects and synthesizing new perspectives based on 3DGS, comprising the following steps:

[0006] S1. Reconstruct the dense 3D point cloud and camera pose information of the dataset based on the image sequence;

[0007] S2. Using the 3D point cloud and camera pose information, the 3D scene is reconstructed using the 3D Gaussian Splatting method to generate a 3D Gaussian kernel to represent the input scene;

[0008] S3. Introduce the Sobel edge detector, extract the edges of the transparent objects in the rendered image generated by the three-dimensional Gaussian kernel and the real image, and calculate the edge loss function of the rendered edge map and the real edge map;

[0009] S4. Combine the edge loss function and the original 3D Gaussian Splatting loss function to jointly optimize the 3D scene reconstruction model;

[0010] S5. Optimize the scene as a whole and output the final 3D reconstruction result.

[0011] Compared with the prior art, the advantages of the present invention are:

[0012] 1. The present invention proposes an edge loss function L based on the Sobel edge detector contour , which makes up for the shortcomings of the original 3DGS loss function in processing edge features of transparent objects; by capturing the edge details of transparent objects and introducing edge constraints in loss optimization, the reconstruction effect of the rendered image and the real image in the transparent area is more accurate, significantly improving the edge expression and reconstruction quality of transparent objects.

[0013] 2. The present invention adopts the DetectorFreeSfM model to replace Colmap, gets rid of the dependence on sparse key point detection, and thus significantly improves the performance of transparent objects in the initial point cloud generation stage; introduces the RoMa feature matcher to replace the LoFTR used by DetectorFreeSfM, and through its more robust feature extraction and matching strategy, improves the feature matching accuracy of the transparent object area, and further optimizes the integrity and quality of point cloud generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of a transparent object reconstruction and new viewing angle synthesis method based on 3DGS of the present invention.

[0015] Figure 2 This is a comparison of the point cloud files generated by the traditional Colmap and the improved version of DetectorFreeSfM.

[0016] Figure 3 This is the result of edge recognition obtained by using the Sobel edge detector on a transparent object.

[0017] Figure 4 It is a three-dimensional reconstruction result of a transparent object reconstruction and new perspective synthesis method based on 3DGS of the present invention, as well as a comparison chart with the existing method. DETAILED DESCRIPTION

[0018] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0019] like Figure 1-4 Shown is a schematic diagram of an embodiment of a transparent object reconstruction and new perspective synthesis method based on 3DGS provided by the present invention.

[0020] A transparent object reconstruction and new perspective synthesis method based on 3DGS includes the following steps:

[0021] Step S1. Reconstruct the dense three-dimensional point cloud and camera pose information of the dataset based on the image sequence.

[0022] The improved DetectorfreeSfM model is used to reconstruct the dense three-dimensional point cloud and camera pose information of the data set according to the image sequence. The featureless detection SfM model DetectorFreeSfM does not rely on feature point detection and can directly generate efficient dense point clouds. The present invention optimizes its accuracy and replaces the LoFTR feature matcher in the original model with the RoMa feature matcher. The multi-scale feature enhancement and transformation invariance matching strategy of RoMa are used to improve the accuracy and robustness of point cloud generation.

[0023] Compared with LoFTR, RoMa combines DINOv2 frozen coarse features and a dedicated fine feature pyramid structure to significantly improve the robustness and local accuracy of feature matching, especially under the complex optical conditions of transparent objects. At the same time, DetectorFreeSfM's multi-view optimization and iterative refinement strategy ensures the high integrity of point cloud generation, providing solid support for the accurate modeling of transparent objects. The generated point cloud image and the comparison with the traditional Colmap model are shown in the figure below. Figure 2 shown.

[0024] Step S2. Utilize the 3D point cloud and camera pose information, use the 3D Gaussian Splatting method to reconstruct the 3D scene, and generate a 3D Gaussian kernel to represent the input scene.

[0025] 3D Gaussian Splatting (3DGS) is an innovative 3D reconstruction technology that uses Gaussian point clouds to characterize the surface characteristics of objects, and represents each point as a Gaussian kernel containing position μ, covariance matrix ∑, transparency α, and spherical harmonic function information SH. During rendering, the Gaussian kernel can be projected into a two-dimensional ellipse to achieve fast and accurate color synthesis. The color calculation formula is:

[0026]

[0027] Where C represents the color, c i Represents the color information on each Gaussian kernel, α i The color of the overall image is obtained by calculating the color of each pixel separately and finally concatenating these color values ​​to form a rendered image I pred .

[0028] Step S3. Introduce the Sobel edge detector, extract the edges of transparent objects in the rendered image generated by the three-dimensional Gaussian kernel and the real image, and calculate the edge loss function of the rendered edge map and the real edge map.

[0029] The present invention applies the Sobel operator to the real image and the 3DGS rendered image respectively to generate corresponding edge maps. The edge map of the real image is used to represent the edge features of the transparent object, while the edge map of the rendered image is used as a comparison basis for subsequent loss calculation and optimization.

[0030] Rendering Image I pred With the rendered edge graph E pred The conversion formula is:

[0031]

[0032] Real Image I gt With the real edge graph E gt The conversion formula is:

[0033]

[0034] in, is the gradient of the real image in the x direction, The gradient of the rendered image in the x direction;

[0035] in, is the gradient of the real image in the y direction, The gradient of the rendered image in the y direction.

[0036] Edge loss function L of rendered edge map and true edge map contour The calculation formula is:

[0037]

[0038] Where M represents the total number of pixels in the image, E pred (i) represents the edge strength of the rendered edge map at the i-th pixel, E gt (i) represents the edge strength of the true edge map at the i-th pixel.

[0039] Step S4. Combine the edge loss function and the original 3D Gaussian Splatting loss function to jointly optimize the 3D scene reconstruction model.

[0040] The original 3D Gaussian Splatting loss function includes the reconstruction consistency loss function L1 and the SSIM loss function L SSIM , and the total loss function of jointly optimizing the 3D scene reconstruction model is:

[0041] L total =λ1L1+λ2L ssim +λ3L contour .

[0042] The values ​​of λ1, λ2, and λ3 are 0.6, 0.2, and 0.2 respectively.

[0043] Marginal loss and global loss (such as L1 and L SSIM ) complement each other, the global loss optimizes the overall consistency of the image, while the edge loss focuses on local details, thereby improving the reconstruction quality of transparent objects at different levels.

[0044]

[0045] L SSIM =1-SSIM(I pred , I gt )

[0046] I pred (i) and (I) gt (i) represents the pixel value of the rendered image and the real image at the i-th pixel; SSIM (I pred , I gt ) represents the structural similarity index between the rendered image and the real image, and its specific calculation formula is:

[0047]

[0048] where x and y are two images; μ x , μ y are the average brightness of image x and image y respectively; and is the variance of images x and y, indicating contrast; σ xy is the covariance of image x and y, indicating structural information; C1 and C2 are constants added to stabilize the calculation. Usually for 8-bit images, C1 and C2 are 650.25 and 5808.75 respectively.

[0049] Step S5: Optimize the scene as a whole and output the final 3D reconstruction result.

[0050] Through the efficient rendering capabilities of 3DGS and the optimized 3D model, realistic image synthesis of transparent objects from a new perspective can be achieved.

[0051] Aiming at the problem that the traditional 3D Gaussian Splatting method uses Colmap to generate the initial point cloud, which performs poorly for low-texture transparent objects, the present invention proposes an improved method based on the DetectorFreeSfM model. By introducing the RoMa feature matcher to replace LoFTR, the robustness of feature matching and the accuracy of point cloud generation for transparent objects are enhanced. At the same time, the present invention designs an edge loss function based on the Sobel edge detector, which makes up for the deficiency of the original loss function in processing the edge details of transparent objects. By comparing the edges of the real image with the rendered image, the model performance is optimized, and the reconstruction quality and overall effect of the transparent object area are significantly improved.

[0052] The above-described embodiments are merely descriptions of preferred implementations of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in the art to the technical solution of the present invention should fall within the protection scope of the present invention, and the technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A transparent object reconstruction and new perspective synthesis method based on 3DGS, characterized in that: The following steps are involved: S1. Reconstruct the dense 3D point cloud and camera pose information of the dataset based on the image sequence; S2. Using the 3D point cloud and camera pose information, the 3D scene is reconstructed using the 3D Gaussian Splatting method to generate a 3D Gaussian kernel to represent the input scene; S3. Introduce the Sobel edge detector, extract the edges of transparent objects in the rendered image generated by the three-dimensional Gaussian kernel and the real image, and calculate the edge loss function L of the rendered edge map and the real edge map contour ; S4. Combined edge loss function L contour And the original 3D Gaussian Splatting loss function, jointly optimize the 3D scene reconstruction model; S5. Optimize the scene as a whole and output the final 3D reconstruction result.

2. The method for transparent object reconstruction and new viewpoint synthesis based on 3DGS according to claim 1, characterized in that: In step S3, the image I is rendered pred With the rendered edge graph E pred The conversion formula is: Real Image I gt With the real edge graph E gt The conversion formula is: in, is the gradient of the real image in the x direction, The gradient of the rendered image in the x direction; is the gradient of the real image in the y direction, The gradient of the rendered image in the y direction.

3. The method for transparent object reconstruction and new viewpoint synthesis based on 3DGS according to claim 2, characterized in that: Edge loss function L of rendered edge map and true edge map contour The calculation formula is: Where M represents the total number of pixels in the image, E pred (i) represents the edge strength of the rendered edge map at the i-th pixel, E gt (i) represents the edge strength of the true edge map at the i-th pixel.

4. The method for transparent object reconstruction and new viewpoint synthesis based on 3DGS according to claim 3, characterized in that: In step S4, the original 3D Gaussian Splatting loss function includes the reconstruction consistency loss function L1 and the SSIM loss function L SSIM , and the total loss function of jointly optimizing the 3D scene reconstruction model is: L total =λ1L1+λ2L ssim +λ3L contour 。 5. The method for transparent object reconstruction and new viewpoint synthesis based on 3DGS according to claim 4, characterized in that: The values ​​of λ1, λ2, and λ3 are 0.6, 0.2, and 0.2 respectively.

6. The method for transparent object reconstruction and new view synthesis based on 3DGS according to claim 1, characterized in that: In step S1, the DetectorfreeSfM model is used to reconstruct the dense three-dimensional point cloud and camera pose information of the data set according to the image sequence, and the LoFTR feature matcher in the original model is replaced by the RoMa feature matcher to generate the initial point cloud.

Images