Differential optimization-based camera attitude and depth joint optimization method in three-dimensional Gaussian sputtering

Abstract

The invention discloses a differential optimization-based camera attitude and depth joint optimization method in three-dimensional Gaussian sputtering, which comprises the following steps of: performing joint correction on camera attitude and depth by adopting a differential optimization strategy, decomposing a position vector and an Euler angle into independent parameters, introducing regularization constraint, and inhibiting initial attitude deviation; an edge perception depth regularization method is provided, an edge weight is generated through image gradient, and a depth distortion image is weighted and optimized to reserve details and smooth region errors; constructing a multi-scale depth pyramid, and enhancing the cross-scale consistency of depth estimation by using layered sampling and weighted error aggregation; the optimized camera position and depth estimation are used for a real-time rendering task, and an end-to-end 3D Gaussian sputtering frame is constructed; according to the method, the problem of geometric distortion caused by camera attitude estimation deviation and insufficient depth supervision in a complex scene is effectively solved, and the texture alignment precision and the far-field detail retention capability are remarkably improved while the real-time rendering performance is kept.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and three-dimensional reconstruction, and particularly relates to a camera pose and depth joint optimization method based on differentiable optimization in three-dimensional Gaussian splatting. BACKGROUND

[0002] In recent years, three-dimensional scene representation technology has made significant progress in the field of novel view synthesis, and its core goal is to reconstruct a three-dimensional scene from sparse input images and generate high-quality rendered images of any perspective. The implicit neural rendering method represented by Neural Radiance Fields (NeRF) models the scene radiance field through a multi-layer perceptron, and performs well in multi-view consistent rendering. However, NeRF relies on dense ray sampling and ray-by-ray neural network inference, which results in low training and inference efficiency, making it difficult to meet the needs of real-time interactive scenes. Although subsequent research has improved efficiency through methods such as hash encoding and tensor decomposition, these optimizations often come at the expense of rendering quality, especially in complex scenes, where details are easily lost and artifacts appear.

[0003] 3D Gaussian Splatting (3DGS) is a new explicit three-dimensional representation method that models the scene through parameterized Gaussian primitives, and combines differentiable projection and rasterization processes to significantly improve reconstruction quality while maintaining real-time rendering speed. Camera pose describes the position and orientation of the camera in three-dimensional space, which determines the transformation relationship of the camera coordinate system relative to the world coordinate system. In computer vision, depth usually refers to the distance from the camera's optical center to the point on the surface of the object, which reflects the front and back position relationship of the object in three-dimensional space relative to the camera. However, existing 3DGS technology still has two major bottlenecks: first, in complex scenes (such as repeated textures and weak texture areas), the initial camera pose obtained by relying on traditional Structure from Motion (SFM) has significant errors, leading to point cloud reconstruction misplacement and geometric distortion; second, the depth supervision mechanism is weak, and existing methods use a single scale of depth distortion constraint, which is difficult to effectively suppress smooth region artifacts and preserve edge details, resulting in local surface distortion and high-frequency noise. SUMMARY

[0004] The present application introduces differentiable optimization of camera pose and multi-scale consistency constraint of depth information to address the problems of point cloud reconstruction misplacement, geometric distortion, and rendering image alignment deviation caused by the error of the initial pose obtained by relying on traditional SFM in existing methods, as well as the inability to effectively balance edge detail preservation and smooth region artifact suppression caused by a single scale of depth distortion constraint, resulting in high-frequency noise and local surface distortion. The present application proposes a camera pose and depth joint optimization method based on differentiable optimization in three-dimensional Gaussian splatting, the specific steps are as follows: S1: Jointly correct camera poses using differentiable optimization strategy, decompose position vector and Euler angles into independent parameters and introduce regularization constraints to suppress initial pose bias; S2: Propose edge-aware depth regularization method, generate edge weight through image gradient, and optimize depth distortion map to preserve details and smooth regional error; S3: Construct multi-scale depth pyramid, use hierarchical sampling and weighted error aggregation to enhance cross-scale consistency of depth estimation; S4: Use step S1-S3 rendering optimization method for real-time rendering task, and construct end-to-end 3D Gaussian sputtering framework.

[0005] Further, step S1 is specifically: S11: Decompose camera pose into position vector And Euler angles ; S12: Use rotation matrix To represent the camera pose, directly optimize the pose parameters through differentiable parameterization scheme .

[0006] (1) (2) Where the primitive rotation matrix is: (3) Where: x, y, z are camera position coordinates.

[0007] S13: Introduce regularization in the optimization process to prevent pose parameters from deviating too much from the initial value. This helps maintain stability and avoid optimization process, avoid falling into local optimal solution: Position regularization loss: (4) Rotation regularization loss: (5) Where, And Respectively represent the optimized position vector and Euler angles, And Respectively represent the initial position vector and Euler angles, N represents the number of camera poses.

[0008] Further, step S2 is specifically: S21: Generate edge-aware weight by calculating the gradient amplitude of the real image: (6) Where And denotes x and y gradient in the direction.

[0009] S22: Generating edge-aware weights: (7) where is used to control the decay rate.

[0010] S23: Weighted depth distortion map: (8) where is a pre-computed depth distortion map, is the spatial resolution, i denotes the horizontal coordinate of the depth map, and j denotes the vertical coordinate of the depth map.

[0011] Further, step S3 is specifically: S31: Constructing a pyramid multi-scale depth map through bilinear interpolation: (9) (10) S32: Measuring the difference between different scales using a smooth L1 loss, and modulating the error through edge weights: (11) (12) where, is the down-sampling factor, is the scale decay coefficient, which is 0.5, is the smooth loss function, x and y are two independent variables in the equation, corresponding to the two parameters of formula (9) and formula (10) , according to the difference between the two parameters, different functions are selected, if , otherwise .

[0012] Further, step S4 is specifically: S41: When applied to real-time rendering tasks, a 3D Gaussian splatting network needs to be constructed, the input of the 3D Gaussian splatting network is the depth information extracted from the image and the camera pose, then the camera pose is decomposed into a position vector and Euler angles for optimization through S1; edge-aware depth regularization and multi-scale depth consistency constraints are applied through S2-S3; finally, the rendered image is obtained through rasterization of the optimized camera pose information and depth information.

[0013] The 3D Gaussian sputtering network loss function is defined as shown in formula 13: (13) wherein, , , The loss weights are 0.2, 0.05 and 0.1 respectively, is obtained from formula (11), is obtained from formula (8), and are obtained from formulas (4) and (5) respectively is the structural similarity loss, is the normal vector loss.

[0014] The present application comprises a device comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a camera pose and depth joint optimization method in three-dimensional Gaussian sputtering based on differentiable optimization.

[0015] The present application also comprises a computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements a camera pose and depth joint optimization method in three-dimensional Gaussian sputtering based on differentiable optimization.

[0016] The technical solution of the present application has the following technical effects: 1. Through differentiable joint optimization, the cumulative error of the initial pose of SFM is corrected, effectively eliminating the geometric misplacement and rendering artifacts caused by pose deviation.

[0017] 2. Through the edge-aware depth regularization strategy, structural details can be preserved and errors in texture and edge regions can be reduced. Combined with depth multi-scale consistency, the robustness of depth estimation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of a camera pose and depth joint optimization method in three-dimensional Gaussian sputtering based on differentiable optimization according to an embodiment of the present application.

[0019] Figure 2 is a network overall structure diagram according to an embodiment of the present application.

[0020] Figure 3 is a Mip-NeRF 360 dataset segmentation result visualization result diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific examples described herein are part of the embodiments of the present application, but not all the embodiments of the present application, and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0022] The present application is a 3D Gaussian sputtering method fusing camera pose optimization and depth consistency constraint, which is especially suitable for real-time high-precision three-dimensional modeling of complex scenes.

[0023] The method of the present application comprises the following steps: S1: jointly correcting the camera pose and the depth by using a differentiable optimization strategy, decomposing the position vector and the Euler angle into independent parameters and introducing a regularization constraint to suppress the initial pose deviation; Step S1 specifically comprises: S11: decomposing the camera pose into a position vector And Euler angle ; S12: using a rotation matrix To represent the camera pose, directly optimizing the pose parameters through a differentiable parameterization scheme .

[0024] (1) (2) Wherein the primitive rotation matrix is: (3) S13: introducing regularization in the optimization process to prevent the pose parameters from deviating too much from the initial value. This helps to maintain stability and avoid the optimization process from falling into a local optimal solution: Position regularization loss: (4) Rotation regularization loss: (5) Wherein, And Respectively represent the optimized position vector and Euler angle, And Respectively represent the initial position vector and Euler angle, and N represents the number of camera poses. Here, the average value of the deviation of all camera poses from the original pose in the optimization process is calculated to prevent the pose parameters from deviating too much from the initial value.

[0025] S2: propose edge-aware depth regularization method, generate edge weight through image gradient, and optimize depth distortion map to preserve details and smooth regional error; Step S2 specifically includes: S21: generate edge-aware weight by calculating the gradient amplitude of the real image: (6) Wherein represents the gradient in the direction, x represents the gradient in the direction. y

[0026] S22: generate edge-aware weight: (7) Wherein is used to control the decay rate.

[0027] S23: weight the depth distortion map: (8) Wherein is the pre-computed depth distortion map, H is high, W is wide, and their product is HW is the spatial resolution.

[0028] S3: build a multi-scale depth pyramid, use hierarchical sampling and weighted error aggregation to enhance the cross-scale consistency of depth estimation; Step S3 specifically includes: S31: build a pyramid multi-scale depth map through bilinear interpolation: (9) (10) S32: use smooth L1 loss to measure the difference between different scales, and modulate the error through edge weight: (11) (12) Wherein, is the down-sampling factor, is the scale decay coefficient, taking 0.5, is the smooth loss function, x and y are two independent variables in the equation, corresponding to the two parameters of formula (9) and formula (10) , according to the difference between the two parameters, different functions are selected, if , ​​ Other cases .

[0029] The overall framework of the network is shown in Figure 2 .

[0030] S4: The step S1-S3 rendering optimization method is used for real-time rendering task, and an end-to-end 3D Gaussian sputtering framework is constructed.

[0031] The step S4 specifically comprises: S41: When applied to real-time rendering task, a 3D Gaussian sputtering network needs to be constructed, the input of the 3D Gaussian sputtering network is the depth information and camera pose extracted from the image, then the camera pose is decomposed into position vector and Euler angle for optimization through S1; edge-aware depth regularization and multi-scale depth consistency constraint are applied through S2-S3; finally, the rendered image is obtained by rasterization with the optimized camera pose information and depth information.

[0032] The 3D Gaussian sputtering network loss function definition is shown in formula (13): (13) Wherein, , , is the loss weight, is an empirical value, and the present application , , 0.2, 0.05 and 0.1 respectively, is the structural similarity loss, is the normal vector loss.

[0033] is a general evaluation index in the art, usually, (14) Wherein, represents the local mean of the image x and the image y, that is, the brightness, indicates the local standard deviation, that is, the contrast, indicates the local covariance, that is, the structure information, indicates the stability coefficient, prevents the denominator from being zero, and usually , L is the maximum pixel value.

[0034] is obtained from formula (11), is obtained from formula (8), and are obtained from formula (4) and (5) respectively.

[0035] (15) For spatial resolution, i represents the horizontal pixel coordinate, j represents the vertical pixel coordinate, represents the normal of the reconstructed rendered image, represents the normal of the original image obtained from the depth map.

[0036] S42: Train and test in the Mip-Nerf 360, Tank and Teemple and DeepBlending dataset, calculate the peak signal-to-noise ratio PSNR, structural similarity SSIM and learned perceptual image patch similarity LPIPS of the rendered image of the model, and compare with PointNet, PointNet++, Kd-Network and other latest network model structures, obtain the comparison results shown in Tables 1, 2 and 3: Table 1 Comparison of rendering results of Mip-Nerf 360 dataset Table 2 Comparison of rendering results of Tank and Temple dataset Table 3 Comparison of rendering results of DeepBlending dataset S43: In order to observe the influence of the designed parts on the rendering results, an ablation experiment is designed to prove it. PR represents pose optimization and regularization, MS represents multi-scale consistency, and EA represents edge loss perception Table 3 Comparison of ablation experiment results of Mip-Nerf 360 dataset The present application can obtain better camera pose to reconstruct a high-precision three-dimensional scene, in the present application, formula (4) and formula (5) are used to prevent the pose parameters from deviating too much from the initial value during the optimization process, which helps to maintain stability and avoid falling into a local optimal solution during the optimization process. The function of formula (13) is similar to that of formula (4) and formula (5), and is also used to prevent the optimization from deviating too much.

[0037] The present application comprises a device comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method for joint optimization of camera pose and depth in three-dimensional Gaussian sputtering based on differentiable optimization.

[0038] The application also comprises a computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method for joint optimization of camera pose and depth in three-dimensional Gaussian sputtering based on differentiable optimization.

[0039] The above embodiments are merely examples for clearly illustrating the present application and are not intended to limit the embodiments. Based on the above description, other different forms of changes or modifications can be made by those skilled in the art. Here, all the embodiments are not required to be enumerated. The obvious changes or modifications derived from the above are still within the protection scope of the present application.

Claims

1. A method for joint camera pose and depth optimization in three-dimensional Gaussian sputtering based on differentiable optimization, characterized in that, The method comprises the following steps: S1: jointly correct the camera pose and depth using a differentiable optimization strategy, decompose the position vector and Euler angle into independent parameters and introduce regularization constraints to suppress the initial pose deviation of the camera; S2: propose an edge-aware depth regularization method, generate edge weights through image gradients, and optimize the depth distortion map to preserve details and smooth regional errors; S3: construct a multi-scale depth pyramid, use hierarchical sampling and weighted error aggregation to enhance the cross-scale consistency of depth estimation; S4: use the rendering optimization method of steps S1-S3 for real-time rendering tasks to build an end-to-end 3D Gaussian splatting framework.

2. The method of claim 1, wherein the method is based on differentiable optimization. The S1 step comprises: S11: decompose the camera pose into a position vector and Euler angles ; S12: using a rotation matrix representing the camera pose, by a differentiable parameterization scheme optimizing the pose parameters, (1) (2) Wherein the primitive rotation matrix is: (3) S13: introduce regularization in the optimization process: Wherein: position regularization loss: (4) Rotation regularization loss: (5) represents the optimized position vector, represents the optimized Euler angles, represents the initial position vector, represents the initial Euler angles, N represents the number of camera poses.

3. The method of claim 1, wherein, The S2 step comprises: S21: generating edge perception weight by calculating gradient amplitude of real image : (6) wherein represents x a gradient in the direction of represents y a gradient in the direction of S22: generating edge-aware weights : (7) wherein to control the rate of decay; S23: weighted depth distortion map: (8) wherein, is a pre-computed depth distortion map, is a spatial resolution.

4. The method of claim 1, wherein, The S3 step comprises: S31: construct a pyramid multi-scale depth map through bilinear interpolation: (9) (10) S32: use a smooth L1 loss to measure the difference between different scales, and modulate the error through edge weights: (11) (12) wherein, is a down-sampling factor, is a scale decay coefficient, and is 0.5, is a smoothing loss function.

5. The method of claim 1, wherein the method is based on differentiable optimization. In the S4 step, when applied to real-time rendering tasks, a 3D Gaussian splatting network is constructed, the input of the 3D Gaussian splatting network is the depth information and camera pose extracted from the image, the camera pose is decomposed into a position vector and Euler angle for optimization through the S1 step, edge-aware depth regularization and multi-scale depth consistency constraints are applied through the S2 and S3 steps, and the rendering image is obtained through rasterization based on the optimized camera pose information and depth information, The loss function of the 3D Gaussian splatting network is defined as formula (13): (13) 、 、 are 0.2, 0.05 and 0.1 respectively, from formula (11), from formula (8), and are obtained from formula (4), (5) respectively, is a structural similarity loss, is a normal vector loss.

6. An apparatus, comprising: It comprises: At least one processor; And a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-5.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-5. The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images