Improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision

Abstract

This paper discloses an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision. By combining the true depth of a sparse point cloud generated by SFM with the predicted depth generated by fast differentiable rasterization, the chamfer distance is calculated and used as the loss function. This optimization strengthens the geometric constraints on the position of Gaussians, ensuring their correct spatial placement, thereby avoiding incorrect occlusion relationships and rendering artifacts, and improving the realism of the scene. Compared with existing technologies, the advantage of this invention is that it overcomes the limitations of existing 3DGS technology in handling geometric constraints through joint supervision of depth and curvature, avoiding the incorrect occlusion and excessive number of small Gaussians that occur in traditional methods, and significantly improving the quality and efficiency of scene rendering.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional image reconstruction, and in particular relates to an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision. Background Art

[0002] Reconstructing complex surfaces and geometries, as well as synthesizing novel perspectives, has long been a focus of interest in computer graphics and vision, with significant applications in intelligent driving, virtual reality, and cultural heritage preservation. With the introduction of Neural Radiance Fields (NeRF), which utilizes multilayer perceptron (MLP) parameters to achieve continuous implicit encoding of scene parameters, NeRF has achieved impressive results in surface geometry reconstruction and novel perspective synthesis. While NeRF has opened up new avenues in geometry reconstruction and novel perspective synthesis, methods such as A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (Mip-NeRF) can achieve impressive rendering quality, but they incur significant time overhead. Instant neural graphics primitives with a multiresolution hash encoding (Instant-NGP) proposes a multiresolution hash encoding alternative to MLP, trading some quality for faster rendering speed. However, it is not well-suited for large-scale scenes. NeRF-based methods, however, lack the ability to effectively and efficiently address the challenges of fast and efficient real-time rendering.

[0003] Recently, 3D Gaussian Splatting (3DGS) has been proposed, building on NeRF's success by designing an explicit radiance field and rendering it to screen space via splatter-based rasterization. 3DGS, as a novel approach, builds upon NeRF and achieves state-of-the-art visual quality while maintaining competitive training time, bringing the state-of-the-art in scene representation and rendering to a new level. The innovation of 3DGS lies in its unique fusion of the strengths of a differentiable pipeline and point-based rendering techniques. By representing the scene using a learnable 3D Gaussian function, it retains the robust fitting capabilities of the continuum radiance field, which is crucial for high-quality image synthesis, while avoiding the computational overhead associated with NeRF-based approaches. During the optimization process, 3DGS manages points using a single Adaptive Density Control (ADC) algorithm, which leverages the average gradient to determine the regions to be processed. During the optimization process, the complexity of the scene geometry often varies greatly, making it difficult for a single ADC algorithm to identify and process all points, potentially leading to rendering errors. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision provided by the present invention solves the following problems:

[0005] (1) During rendering along the viewing direction, Gaussians with incorrect opacity or scale will occlude Gaussians behind them, eventually causing artifacts in the rendering results.

[0006] (2) The ADC algorithm of 3DGS uses the average gradient threshold based on the image gradient to adjust the splitting and cloning of Gaussians, which will result in a large number of small and dense Gaussians in high-gradient areas and ignore the Gaussians that contribute to rendering in areas with smaller gradients.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision, comprising the following steps:

[0008] S1. Obtain a sparse corresponding set of image features based on the real-time image captured by the camera, use the SfM technology to generate a sparse point cloud representation of the scene, and simultaneously estimate the camera's motion posture to obtain the camera extrinsic parameters;

[0009] S2. Using sparse point clouds, the scene is represented as a set of Gaussian distributions, where the center position is the spatial position of the point cloud, the constant of the spherical harmonic function is the color of the corresponding pixel of the point cloud, the high-order coefficient is set to zero, the three-axis scale is set to the average distance of the three point clouds adjacent to the center position, and the axis is represented by a quaternion and initialized as , opacity is set to 0.1 and the number of iterations is initialized;

[0010] S3. Combined with the camera external parameters, the perspective transformation matrix is used to project the Gaussians distribution onto the two-dimensional plane under the camera perspective, and the 2D Gaussians projected onto the image plane are obtained, from which the predicted depth is obtained;

[0011] S4. Sort the 2D Gaussians by depth, determine the color of each pixel to be rendered according to the specific point through which the viewing direction passes, and generate a rendered image;

[0012] S5. Determine whether the number of iterations reaches the preset iteration threshold. If so, proceed to S6. If not, calculate the loss function based on the true depth, predicted depth, rendered image, and RGB true image, update the optimized Gaussians distribution based on the calculation result, increase the number of iterations by 1, and return to S3.

[0013] S6. Output the final rendered image.

[0014] Furthermore, in S2, the method of representing the scene as a set of Gaussian distributions is specifically as follows:

[0015] The sparse point cloud is combined with the camera extrinsic parameters and converted into the screen space coordinate system. The distance from each point in the sparse point cloud to the camera center is obtained and used as the true depth of each point. The invisible points in the camera's perspective are filtered through the viewing cone, and the point with the smallest camera depth corresponding to the true depth of the remaining points is determined as the surface point. The surface points are initialized as a set of Gaussian distributions.

[0016] Among them, the Gaussians distribution includes several Gaussians, each of which includes 3D Gaussian mean, covariance matrix, opacity and spherical harmonics. The specific expression is:

[0017]

[0018] Where, is any point in 3D space, , is the mean vector, , R is a matrix, is the covariance matrix, , used to maintain during the optimization process The positive semidefinite property of .

[0019] Further: In the above S4, the color of the pixel is obtained by the Gaussians passed through by the viewing direction, and the pixel p Color The specific expression is:

[0020]

[0021] Where, is a learnable, view-dependent color representation using spherical harmonics, For the reason of evaluation from i Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point through which the viewing direction passes, For the reason of evaluation from j Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point that the viewing direction passes through. Represents the occlusion effect of the previous Gaussians on the current Gaussian.

[0022] Further: In S5, the loss function The specific expression is:

[0023]

[0024] Where, is the first hyperparameter, which is used to control the weight of SSIM structural similarity loss in the loss function. is the second hyperparameter, which is used to control the weight of depth loss in the loss function. is the mean absolute error loss based on the real image and the rendered image, is the SSIM structural similarity loss based on real images and rendered images, The depth of the camera corresponding to the center distance of the downsampled Gaussians and the true depth chamfer distance obtained from the sparse point cloud are expressed as follows:

[0025]

[0026] Where, is a set of predicted depth components, is the set of true depths of each point in the sparse point cloud, d i is the prediction depth of the specific Gaussian, d jis the true depth of a specific point in the sparse point cloud, is the L1 norm.

[0027] Furthermore, in S5, after the number of iterations reaches 500, adaptive density control is performed every 100 iterations to update and optimize the Gaussians distribution. The method of adaptive density control is specifically as follows:

[0028] For each Gaussian, track the magnitude of the position gradient of all rendered views and generate the average position gradient of all rendered views. In response to the gradient of the position being greater than the average gradient threshold during the iteration, a split or clone operation is performed. The specific method is as follows:

[0029] Determine whether the scale of the current Gaussian is less than or equal to the scale threshold. If so, perform a clone operation; if not, perform a split operation.

[0030] Furthermore, in S5, after the number of iterations reaches 500, a curvature-based Gaussians optimization is performed every 100 iterations to update and optimize the uniformity of the Gaussians distribution. The curvature-based Gaussians optimization method is specifically as follows:

[0031] During the iteration process, the number of Gaussians in each quadrant is counted, and the quadrant with the highest density is selected as the dense quadrant. A Gaussian is randomly selected in the dense quadrant as the center position, and the Euclidean distance from other Gaussians in the dense quadrant to the center is calculated. The Gaussian with a Euclidean distance less than the distance threshold is selected, and the selected Gaussian is optimized.

[0032] Further: The method for optimizing the selected Gaussian is as follows:

[0033] Calculate the curvature of the selected Gaussian and delete the Gaussian whose curvature is greater than the curvature threshold. The specific method for calculating the curvature is:

[0034] Use the KD-tree-based fast nearest neighbor search algorithm to efficiently find the neighbors of all Gaussians in the range , and calculate the covariance matrix at the center of the Gaussian, the covariance matrix The specific expression is:

[0035]

[0036] Where, n is the number of other Gaussians in the search range centered on this Gaussian, The coordinate value of the center position of the Gaussian, To find the coordinate values of the remaining Gaussians within the range by the proximity search algorithm, T is the transpose symbol;

[0037] Based on 3 3's covariance matrix, determine the first to third eigenvalues through the relationship between the eigenvalues and the corresponding eigenvectors, and then calculate the curvature;

[0038] The expression of the relationship between the eigenvalue and the corresponding eigenvector is as follows:

[0039]

[0040] Where, v is the eigenvector, is the characteristic value;

[0041] Calculating curvature The specific expression is:

[0042]

[0043] Where, is the first eigenvalue, is the second eigenvalue, is the third eigenvalue.

[0044] Further: in S5, a rescaling operation is performed every 300 iterations or every 1000 iterations, and the rescaling operation method is specifically as follows:

[0045] The scale of Gaussians is controlled by combining the predicted depth with the scaling factor, where the scaling factor The specific expression is:

[0046]

[0047] Where, For each Gaussian prediction depth, is the minimum predicted depth, is the maximum predicted depth, is a hyperparameter.

[0048] The beneficial effects of the present invention are as follows: the present invention provides an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision, which has the following effects compared with the prior art:

[0049] (1) Introduction of loss function: By combining the true depth of the sparse point cloud generated by SFM and the predicted depth generated by fast differentiable rasterization, the chamfer distance is calculated and used as the loss function. This optimization strengthens the geometric constraints on the position of Gaussians, ensuring the correct position of Gaussians in space, thereby avoiding incorrect occlusion relationships and rendering artifacts, and improving the realism of the scene.

[0050] (2) Rescaling: The scale of the Gaussians is adjusted based on the distance from their center to the camera. This operation ensures that the Gaussians have the correct occlusion relationship during the rendering process. In particular, when rendering in the view direction, the scale and occlusion relationship of the Gaussians are accurately maintained, avoiding the appearance of artifacts and improving rendering quality.

[0051] (3) Curvature as a constraint: To address the problem of small, dense Gaussians generated in high-gradient regions, the improved ADC algorithm incorporates curvature as a new constraint. This is especially true for high-curvature Gaussians that suddenly appear in smooth regions. These Gaussians often do not conform to the geometric characteristics of the region and therefore need to be removed. This operation reduces redundant small Gaussians, lowers computational complexity, and avoids negative impacts on model optimization.

[0052] (4) Improve rendering quality and performance: By performing more precise geometric constraints and scale adjustments on Gaussians, the occlusion relationship between Gaussians during the rendering process and their performance in three-dimensional space are improved, ultimately improving the accuracy of scene reconstruction, especially in the rendering of complex scenes, avoiding performance bottlenecks caused by unreasonable Gaussians distribution.

[0053] In summary, compared with the existing technology, the advantage of the present invention is that it solves the limitations of geometric constraint processing in the existing 3DGS technology through joint supervision of depth and curvature, avoids the false occlusion and excessive small Gaussians that occur in traditional methods, and significantly optimizes the quality and efficiency of scene rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision of the present invention. DETAILED DESCRIPTION

[0055] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0056] like Figure 1 As shown, in one embodiment of the present invention, an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision includes the following steps:

[0057] S1. Obtain a sparse corresponding set of image features based on the real-time image captured by the camera, use the SfM technology to generate a sparse point cloud representation of the scene, and simultaneously estimate the camera's motion posture to obtain the camera extrinsic parameters;

[0058] S2. Using sparse point clouds, the scene is represented as a set of Gaussian distributions, where the center position is the spatial position of the point cloud, the constant of the spherical harmonic function is the color of the corresponding pixel of the point cloud, the high-order coefficient is set to zero, the three-axis scale is set to the average distance of the three point clouds adjacent to the center position, and the axis is represented by a quaternion and initialized as , opacity is set to 0.1 and the number of iterations is initialized;

[0059] S3. Combined with the camera external parameters, the perspective transformation matrix is used to project the Gaussians distribution onto the two-dimensional plane under the camera perspective, and the 2D Gaussians projected onto the image plane are obtained, from which the predicted depth is obtained;

[0060] S4. Sort the 2D Gaussians by depth, determine the color of each pixel to be rendered according to the specific point through which the viewing direction passes, and generate a rendered image;

[0061] S5. Determine whether the number of iterations reaches the preset iteration threshold. If so, proceed to S6. If not, calculate the loss function based on the true depth, predicted depth, rendered image, and RGB true image, update the optimized Gaussians distribution based on the calculation result, increase the number of iterations by 1, and return to S3.

[0062] S6. Output the final rendered image.

[0063] In this embodiment, to address the limitations of geometric constraints in existing model optimization processes and improve rendering quality and performance, the present invention uses a sparse point cloud generated by SFM to obtain the true depth and the distance from the Gaussians generated by fast differentiable rasterization to the camera's depth as the predicted depth. The chamfer distance calculated from the predicted depth and the true depth is used as a loss function to strengthen the geometric constraints on the Gaussians' positions. At the same time, a rescaling operation is performed based on the distance between the center of the Gaussians and their relative camera center to ensure that when rendering along the viewing angle, the Gaussians have the correct occlusion relationship and a more accurate frequency representation. In dense areas after ADC algorithm optimization, the present invention introduces curvature as a new constraint. For example, the sudden appearance of a high-curvature Gaussian in a smooth area indicates that the Gaussian is not suitable for the geometric characteristics of the area. It is removed to prevent a large number of small and dense Gaussians from increasing computational complexity and hindering model optimization.

[0064] The S2 is specifically:

[0065] The sparse point cloud is combined with the camera extrinsic parameters and converted into the screen space coordinate system. The distance from each point in the sparse point cloud to the camera center is obtained and used as the true depth of each point. The invisible points in the camera's perspective are filtered through the viewing cone, and the point with the smallest camera depth corresponding to the true depth of the remaining points is determined as the surface point. The surface points are initialized as a set of Gaussian distributions.

[0066] Among them, the Gaussians distribution includes several Gaussians, each of which includes 3D Gaussian mean, covariance matrix, opacity and spherical harmonics. The specific expression is:

[0067]

[0068] Where, is any point in 3D space, , is the mean vector, , R is a matrix, is the covariance matrix, , used to maintain during the optimization process The positive semidefinite property of , which can be expressed as:

[0069]

[0070] Where, R is a matrix, specifically representing a 3 3's rotation matrix, which is converted from quaternion, S 3 3 scaling matrix, which describes the scaling transformation in 3D space.

[0071] In this embodiment, for Gaussian parameters, the covariance matrix is generated by the scale and the quaternion q, and the opacity and spherical harmonics are used to fit the appearance and color in the viewing direction.

[0072] In S3, the perspective transformation matrix is used in combination with the camera external parameters. Project the Gaussians distribution onto the two-dimensional plane under the camera's view:

[0073]

[0074] Where, is the Jacobian matrix of the projective transformation, which is used to calculate the covariance matrix in the two-dimensional image space.

[0075] In S4, the color of the pixel is obtained by the Gaussians passed through by the view direction. p Color The specific expression is:

[0076]

[0077] Where, is a learnable, view-dependent color representation using spherical harmonics, For the reason of evaluation from i Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point through which the viewing direction passes, For the reason of evaluation from j Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point through which the viewing direction passes, Used to indicate the occlusion effect of the previous Gaussians on the current Gaussian.

[0078] In this embodiment, It is not necessarily the center of the Gaussian, which is obtained based on the specific point through which the viewing direction passes, and is obtained by the Gaussian center point And the specific point through which the viewing direction passes distance , and The exponential decay coefficient is defined by , which is used to calculate the opacity of a specific point passing through Gaussians, which is equivalent to a decay index for the opacity of the center position , where cis the opacity corresponding to the center of the Gaussian, The definition is as follows:

[0079]

[0080] Where, yes x The variance of the coordinates, x The dispersion of the coordinates, for y The variance of the coordinates, y The dispersion of the coordinates, For x and y The covariance between the coordinates, expressed as x and y Linear relationship of coordinates.

[0081] In S5, the loss function The specific expression is:

[0082]

[0083] Where, is the first hyperparameter, which is used to control the weight of SSIM structural similarity loss in the loss function. is the second hyperparameter, which is used to control the weight of depth loss in the loss function. is the mean absolute error loss based on the real image and the rendered image, is the SSIM structural similarity loss based on real images and rendered images, The chamfer distance is the distance between the camera's depth and the real depth obtained from the sparse point cloud based on the downsampled Gaussians center distance. Its expression is as follows:

[0084]

[0085] Where, is a set of predicted depth components, is the set of true depths of each point in the sparse point cloud, d i is the prediction depth of the specific Gaussian, d j is the true depth of a specific point in the sparse point cloud, is the L1 norm.

[0086] In this embodiment, based on the obtained loss function result, backpropagation is performed to update and optimize each Gaussian parameter of the Gaussians distribution in S2, which includes the 3D Gaussian mean, the covariance matrix generated by the scale and the quaternion q, and each Gaussian also includes opacity and spherical harmonics.

[0087] In S5, after the number of iterations reaches 500, adaptive density control is performed every 100 iterations to update and optimize the Gaussians distribution. The method of adaptive density control is specifically as follows:

[0088] For each Gaussian, track the magnitude of the position gradient of all rendered views and generate the average position gradient of all rendered views. In response to the gradient of the position being greater than the average gradient threshold during the iteration, a split or clone operation is performed. The specific method is as follows:

[0089] Determine the current Gaussian scale s Is it less than or equal to the scale threshold? If so, it means that the coverage density of the Gaussian is low, and the cloning operation can be used to enhance the density. Perform a cloning operation. If not, it means that the coverage area density of the Gaussian is high, and a splitting operation can be performed to split the large Gaussian into multiple small Gaussians. Perform a splitting operation.

[0090] In this embodiment, if the average position gradient of each Gaussian exceeds a predefined average gradient threshold, the Gaussian is considered to be insufficiently representative of the corresponding 3D region. In regions with insufficient geometric features (i.e., under-reconstructed regions), density is increased by cloning existing 3D Gaussians. In over-reconstructed regions, a large 3D Gaussian may cover an area that should be represented by multiple small Gaussians. In this case, the large 3D Gaussian is split into two smaller Gaussians to better capture details.

[0091] In S5, after the number of iterations reaches 500, a curvature-based Gaussians optimization is performed every 100 iterations to update and optimize the uniformity of the Gaussians distribution. The curvature-based Gaussians optimization method is specifically as follows:

[0092] During the iteration process, the number of Gaussians in each quadrant is counted, and the quadrant with the highest density is selected as the dense quadrant. A Gaussian is randomly selected in the dense quadrant as the center position, and the Euclidean distance from other Gaussians in the dense quadrant to the center is calculated. The Gaussian with a Euclidean distance less than the distance threshold is selected, and the selected Gaussian is optimized.

[0093] The specific method for optimizing the selected Gaussian is:

[0094] Calculate the curvature of the selected Gaussian and delete the Gaussian whose curvature is greater than the curvature threshold. The specific method for calculating the curvature is:

[0095] Use the KD-tree-based fast nearest neighbor search algorithm to efficiently find the neighbors of all Gaussians in the range , and calculate the covariance matrix at the center of the Gaussian, the covariance matrix The specific expression is:

[0096]

[0097] Where, n is the number of other Gaussians in the search range centered on this Gaussian, The coordinate value of the center position of the Gaussian, To find the coordinate values of the remaining Gaussians within the range by the proximity search algorithm, T is the transpose symbol;

[0098] Based on 3 3's covariance matrix, determine the first to third eigenvalues through the relationship between the eigenvalues and the corresponding eigenvectors, and then calculate the curvature;

[0099] The expression of the relationship between the eigenvalue and the corresponding eigenvector is as follows:

[0100]

[0101] Where, v is the eigenvector, is the characteristic value;

[0102] Calculating curvature The specific expression is:

[0103]

[0104] Where, is the first eigenvalue, is the second eigenvalue, is the third eigenvalue.

[0105] In this embodiment, a fast nearest neighbor search algorithm based on a KD tree is used to efficiently find all neighboring points of all Gaussians within the range, and the covariance matrix of the Gaussian is calculated. The curvature of each Gaussian position is calculated using the eigenvalue obtained from the covariance matrix. When the curvature is greater than a given curvature threshold, the present invention will remove it, such as high curvature points on a smooth surface, to reduce the appearance of a large number of small and dense Gaussians in the dense quadrant.

[0106] In S5, a rescaling operation is performed every 300 iterations or every 1000 iterations. The specific method of the rescaling operation is:

[0107] The scale of Gaussians is controlled by combining the predicted depth with the scaling factor, where the scaling factor The specific expression is:

[0108]

[0109] Where, For each Gaussian prediction depth, is the minimum predicted depth, is the maximum predicted depth, is a hyperparameter.

[0110] In this embodiment, a scaling factor is defined To optimize the Gaussians scale, the distance from the center of the Gaussians to the center of the camera is used as the control basis and normalized. In order to further control the scaling factor, the present invention introduces a hyperparameter , so that the scaling factor is limited to .

[0111] Small Gaussians represent high-frequency information, while large Gaussians do the opposite. This scaling factor can reduce the scale of Gaussians closer to the camera plane and increase the scale of Gaussians farther from the camera plane, making their frequency representation more accurate.

[0112] The beneficial effects of the present invention are as follows: the present invention provides an improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision, which has the following effects compared with the prior art:

[0113] (1) Introduction of loss function: By combining the true depth of the sparse point cloud generated by SFM and the predicted depth generated by fast differentiable rasterization, the chamfer distance is calculated and used as the loss function. This optimization strengthens the geometric constraints on the position of Gaussians, ensuring the correct position of Gaussians in space, thereby avoiding incorrect occlusion relationships and rendering artifacts, and improving the realism of the scene.

[0114] (2) Rescaling: The scale of the Gaussians is adjusted based on the distance from their center to the camera. This operation ensures that the Gaussians have the correct occlusion relationship during the rendering process. In particular, when rendering in the view direction, the scale and occlusion relationship of the Gaussians are accurately maintained, avoiding the appearance of artifacts and improving rendering quality.

[0115] (3) Curvature as a constraint: To address the problem of small, dense Gaussians generated in high-gradient regions, the improved ADC algorithm incorporates curvature as a new constraint. This is especially true for high-curvature Gaussians that suddenly appear in smooth regions. These Gaussians often do not conform to the geometric characteristics of the region and therefore need to be removed. This operation reduces redundant small Gaussians, lowers computational complexity, and avoids negative impacts on model optimization.

[0116] (4) Improve rendering quality and performance: By performing more precise geometric constraints and scale adjustments on Gaussians, the occlusion relationship between Gaussians during the rendering process and their performance in three-dimensional space are improved, ultimately improving the accuracy of scene reconstruction, especially in the rendering of complex scenes, avoiding performance bottlenecks caused by unreasonable Gaussians distribution.

[0117] In summary, compared with the existing technology, the advantage of the present invention is that it solves the limitations of geometric constraint processing in the existing 3DGS technology through joint supervision of depth and curvature, avoids the false occlusion and excessive small Gaussians that occur in traditional methods, and significantly optimizes the quality and efficiency of scene rendering.

[0118] In the description of the present invention, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of such features.

Claims

1. Improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision, characterized by: The following steps are involved: S1. Obtain a sparse corresponding set of image features based on the real-time image captured by the camera, use the SfM technology to generate a sparse point cloud representation of the scene, and simultaneously estimate the camera's motion posture to obtain the camera extrinsic parameters; S2. Using sparse point clouds, the scene is represented as a set of Gaussian distributions, where the center position is the spatial position of the point cloud, the constant of the spherical harmonic function is the color of the corresponding pixel of the point cloud, the high-order coefficient is set to zero, the three-axis scale is set to the average distance of the three point clouds adjacent to the center position, and the axis is represented by a quaternion and initialized as , opacity is set to 0.1 and the number of iterations is initialized; S3. Combined with the camera external parameters, the perspective transformation matrix is used to project the Gaussians distribution onto the two-dimensional plane under the camera perspective, and the 2D Gaussians projected onto the image plane are obtained, from which the predicted depth is obtained; S4. Sort the 2D Gaussians by depth, determine the color of each pixel to be rendered according to the specific point through which the viewing direction passes, and generate a rendered image; S5. Determine whether the number of iterations reaches the preset iteration threshold. If so, proceed to S6. If not, calculate the loss function based on the true depth, predicted depth, rendered image, and RGB true image, update the optimized Gaussians distribution based on the calculation result, increase the number of iterations by 1, and return to S3. S6. Output the final rendered image.

2. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S2, the method of representing the scene as a set of Gaussian distributions is specifically as follows: The sparse point cloud is combined with the camera extrinsic parameters and converted into the screen space coordinate system. The distance from each point in the sparse point cloud to the camera center is obtained and used as the true depth of each point. The invisible points in the camera's perspective are filtered through the viewing cone, and the point with the smallest camera depth corresponding to the true depth of the remaining points is determined as the surface point. The surface points are initialized as a set of Gaussian distributions. Among them, the Gaussians distribution includes several Gaussians, each of which includes 3D Gaussian mean, covariance matrix, opacity and spherical harmonics. The specific expression is: Where, is any point in 3D space, , is the mean vector, , R is a matrix, is the covariance matrix, , used to maintain during the optimization process The positive semidefinite property of .

3. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S4, the color of the pixel is obtained by the Gaussians passed through by the view direction. p Color The specific expression is: Where, is a learnable, view-dependent color representation using spherical harmonics, For the reason of evaluation from i Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point through which the viewing direction passes, For the reason of evaluation from j Gaussian projection to pixel p The 2D Gaussian gets the opacity of the specific point that the viewing direction passes through.

4. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S5, the loss function The specific expression is: Where, is the first hyperparameter, which is used to control the weight of SSIM structural similarity loss in the loss function. is the second hyperparameter, which is used to control the weight of depth loss in the loss function. is the mean absolute error loss based on the real image and the rendered image, is the SSIM structural similarity loss based on real images and rendered images, The depth of the camera corresponding to the center distance of the downsampled Gaussians and the true depth chamfer distance obtained from the sparse point cloud are expressed as follows: Where, is a set of predicted depth components, is the set of true depths of each point in the sparse point cloud, d i is the prediction depth of the specific Gaussian, d j is the true depth of a specific point in the sparse point cloud, is the L1 norm.

5. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S5, after the number of iterations reaches 500, adaptive density control is performed every 100 iterations to update and optimize the Gaussians distribution. The method of adaptive density control is specifically as follows: For each Gaussian, track the magnitude of the position gradient of all rendered views and generate the average position gradient of all rendered views. In response to the gradient of the position being greater than the average gradient threshold during the iteration, a split or clone operation is performed. The specific method is as follows: Determine whether the scale of the current Gaussian is less than or equal to the scale threshold. If so, perform a clone operation; if not, perform a split operation.

6. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S5, after the number of iterations reaches 500, a curvature-based Gaussians optimization is performed every 100 iterations to update and optimize the uniformity of the Gaussians distribution. The curvature-based Gaussians optimization method is specifically as follows: During the iteration process, the number of Gaussians in each quadrant is counted, and the quadrant with the highest density is selected as the dense quadrant. A Gaussian is randomly selected in the dense quadrant as the center position, and the Euclidean distance from other Gaussians in the dense quadrant to the center is calculated. The Gaussian with a Euclidean distance less than the distance threshold is selected, and the selected Gaussian is optimized.

7. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 6 is characterized in that: The specific method for optimizing the selected Gaussian is: Calculate the curvature of the selected Gaussian and delete the Gaussian whose curvature is greater than the curvature threshold. The specific method for calculating the curvature is: Use the KD-tree-based fast nearest neighbor search algorithm to efficiently find the neighbors of all Gaussians in the range , and calculate the covariance matrix at the center of the Gaussian, the covariance matrix The specific expression is: Where, n is the number of other Gaussians in the search range centered on this Gaussian, The coordinate value of the center position of the Gaussian, To find the coordinate values of the remaining Gaussians within the range by the proximity search algorithm, T is the transpose symbol; Based on 3 3's covariance matrix, determine the first to third eigenvalues through the relationship between the eigenvalues and the corresponding eigenvectors, and then calculate the curvature; The expression of the relationship between the eigenvalue and the corresponding eigenvector is as follows: Where, v is the eigenvector, is the eigenvalue; Calculating curvature The specific expression is: Where, is the first eigenvalue, is the second eigenvalue, is the third eigenvalue.

8. The improved 3DGS scene reconstruction and rendering method based on depth and curvature supervision according to claim 1 is characterized in that: In S5, a rescaling operation is performed every 300 iterations or every 1000 iterations. The specific method of the rescaling operation is: The scale of Gaussians is controlled by combining the predicted depth with the scaling factor, where the scaling factor The specific expression is: Where, For each Gaussian prediction depth, is the minimum predicted depth, is the maximum predicted depth, is a hyperparameter.

Images