A method and system for rendering a three-dimensional scene reconstruction

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a three-dimensional scene reconstruction rendering method and system. The present application generates a corresponding 3D Gaussian ellipsoid for each sparse point cloud of a scene to be reconstructed; obtains a re-projection area of each 3D Gaussian ellipsoid on a two-dimensional image at different viewing angles; takes each re-projection area as the center, and sets all re-projection areas within a certain pixel distance range as adjacent re-projection areas of the re-projection area; based on the sum of the Gaussian kernel function values from the center of the re-projection area of each 3D Gaussian ellipsoid on the two-dimensional image at different viewing angles to the centers of each adjacent re-projection area of the area, the edge intensity, the local variance, the normalized gradient amplitude, and the absolute value of the dot product of the gradient vector and the normal vector, calculates the split control function value of each re-projection area; compares the split control function value of each re-projection area with the size of a set threshold, and controls the 3D Gaussian ellipsoid to split. The present application improves the three-dimensional scene reconstruction rendering precision.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a three-dimensional scene reconstruction and rendering method and system. Background Technology

[0002] 3D scene reconstruction rendering aims to recover the 3D geometric structure and appearance information of a scene from 2D image data. It achieves 3D modeling of the scene through geometric analysis, feature matching, and data fusion of multi-view images. Rendering techniques then transform the 3D model into a 2D image that reflects the scene's geometric structure, lighting effects, material properties, and other 3D information, intuitively presenting scene details. This technology is widely used in film and television production, game development, virtual reality, and other fields. Compared to the traditional Neural Radiation Field (NeRF) method, 3D Gaussian Sputtering (3DGS) significantly improves rendering efficiency while maintaining high-fidelity rendering quality. 3DGS utilizes Structure from Motion (SfM) technology to process a series of 2D images of the 3D scene to be reconstructed. SfM technology estimates the camera pose and the 3D position of the sparse point cloud in the scene by matching and tracking feature points in multi-view images, generating sparse point clouds and camera parameters. An initial 3D Gaussian ellipsoid is created for each sparse point cloud. Each 3D Gaussian ellipsoid consists of position, covariance matrix, color, and transparency. By optimizing the parameters of the 3D Gaussian ellipsoid, the geometry and texture information of the scene can be accurately modeled, so that the Gaussian ellipsoid can better approximate the real scene and obtain a 3D scene model. Then, using differentiable tile rasterization technology, the optimized 3D Gaussian ellipsoid is projected onto a 2D plane to generate a 2D rendering map to display the 3D scene model.

[0003] Despite significant advancements in rendering efficiency and scene representation, 3DGS methods still face the challenge of jagged artifacts when handling complex scenes. These artifacts are typically caused by the aliasing of low-frequency sampling with high-frequency information, particularly noticeable in high-contrast regions and detailed edges. When the sampling frequency fails to match the high-frequency details in the scene (such as sharp object edges and complex textures), information loss or mismatch results in stepped or flickering visual defects. This not only degrades image visual quality but also impacts the accuracy and realism of rendering algorithms. To address this issue, some research utilizes edge information to perform splitting decisions on the 3D Gaussian ellipsoid. This involves splitting the 3D Gaussian ellipsoid at the edges, increasing the number of Gaussian splits at the edges with multiple new Gaussian ellipsoids. This refines the sampling density of local areas, making the sampling frequency more closely match the distribution of high-frequency information, thereby reducing the aliasing of low-frequency sampling with high-frequency information and mitigating jagged artifacts to some extent. However, current techniques typically rely solely on the edge response intensity of the image (such as gradient magnitude or the output of edge detection algorithms) to determine the splitting of the 3D Gaussian ellipsoid, ignoring other important features such as local texture variations, scale variations, and spatial distribution relationships within the image. For example, while some regions may exhibit strong edge features in edge detection, excessive 3D Gaussian ellipsoid splitting due to low texture complexity or density can lead to overfitting, failing to further improve rendering accuracy and increasing unnecessary computational overhead. Conversely, for regions with rich texture details but weak edge gradients, relying solely on edge intensity for splitting decisions can result in jagged edges due to insufficient sampling. Therefore, a 3D Gaussian ellipsoid splitting strategy is still needed that can effectively balance rendering quality and computational efficiency while addressing jagged edges. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the existing 3D Gaussian ellipsoid splitting strategy, which only relies on the edge response intensity of the image and ignores other important features such as local texture changes, scale changes and spatial distribution relationships in the image. This makes it difficult to deal with jagged artifacts and leads to a decrease in the accuracy of 3D scene reconstruction and rendering.

[0005] To address the aforementioned technical problems, this invention provides a three-dimensional scene reconstruction and rendering method, comprising:

[0006] Step S1: Construct an initial set of 3D Gaussian ellipsoids based on the 3D Gaussian ellipsoids corresponding to each sparse point cloud in the scene to be reconstructed;

[0007] Step S2: Obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; take each reprojection region as the center, and take all reprojection regions within a set pixel distance range as the adjacent reprojection regions of that reprojection region.

[0008] Step S3: Based on the sum of the Gaussian kernel function values ​​from the center of each reprojection region to the centers of all adjacent reprojection regions, as well as the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector at the center of the reprojection region, calculate the split control function value for each reprojection region.

[0009] Step S4: Determine whether the split control function values ​​of all reprojected regions are less than or equal to the set threshold; if there is a reprojected region with a split control function value greater than the set threshold, split the 3D Gaussian ellipsoid corresponding to the reprojected region, iteratively update the 3D Gaussian ellipsoid set, and return to execute step S2; if all are less than or equal to the set threshold, use the current 3D Gaussian ellipsoid set as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

[0010] Preferably, the step of using the current 3D Gaussian ellipsoid set as the scene densification result to generate a 3D model and a 2D rendering of the scene to be reconstructed includes:

[0011] Based on the size of the reprojection region of each 3D Gaussian ellipsoid on the 2D image from different viewpoints, the gradient magnitude and transparency of each pixel, the 3D Gaussian ellipsoid set is pruned, and the 3D model of the scene to be reconstructed is generated using the pruned 3D Gaussian ellipsoid set.

[0012] Based on the camera extrinsics, the pruned 3D Gaussian ellipsoid set is projected onto the target 2D image plane. After processing each reprojection region on the target 2D image plane with a Gaussian low-pass filter, the parameters of each processed reprojection region are rasterized with differentiable tiles to generate a 2D rendering of the scene to be reconstructed.

[0013] Preferably, each reprojection region on the target two-dimensional image plane is processed using a Gaussian low-pass filter to obtain parameters for each processed reprojection region, including:

[0014] Based on the width, height, and number of pruned 3D Gaussian ellipsoids of the target 2D image, calculate the parameters of the Gaussian low-pass filter.

[0015] By adding Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix of each reprojection region on the target two-dimensional image plane, the parameters of each processed reprojection region are obtained.

[0016] The formula for calculating the parameters of the Gaussian low-pass filter is as follows:

[0017]

[0018] In the formula, s is the Gaussian low-pass filter parameter, H is the height of the target 2D image, W is the width of the target 2D image, and K is the number of pruned 3D Gaussian ellipsoids.

[0019] Preferably, the step of obtaining the parameters of each processed reprojection region by adding Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix of each reprojection region on the target two-dimensional image plane includes:

[0020]

[0021] in, The position of the k-th reprojection region on the target two-dimensional image plane The reprojected region parameters are exp(·), which is an exponential function with the natural constant as the base. Let μ′ be any position in the k-th reprojection region on the target two-dimensional image plane. k Let be the center coordinates of the k-th reprojected region on the target two-dimensional image plane, (·). T For the transpose operation, Σ′ k Let be the covariance matrix of the k-th reprojection region, s be the parameters of the Gaussian low-pass filter, and I be the identity matrix.

[0022] Preferably, the edge intensity at the center of the current reprojection region is obtained by weighted summation of the magnitudes of the gradients of the three RGB channels at the center of the current reprojection region.

[0023] Calculate the absolute values ​​of the differences between the pixel values ​​at the center of the current reprojection region and the pixel values ​​of other pixels in the region, and take the average value as the local variance of the center of the current reprojection region.

[0024] The gradient magnitude at the center of the current reprojection region is normalized to obtain the normalized gradient magnitude at the center of the current reprojection region.

[0025] Preferably, the formula for calculating the normalized gradient magnitude at the center of the current reprojection region is:

[0026]

[0027] Among them, S gauss (x, y) represents the normalized gradient magnitude at the center of the current reprojection region. Let be the gradient vector at the center of the current reprojection region, ∈ be the regularization term, ‖·‖ be the Euclidean norm, and represent the magnitude. Let (x, y) be the gradient vector of the center of the current reprojection region, where (x, y) are the coordinates of the center of the current reprojection region, x is the x-coordinate of the center of the current reprojection region, and y is the y-coordinate of the center of the current reprojection region.

[0028] Preferably, the formula for calculating the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions is as follows:

[0029]

[0030] Among them, C space (x, y) represents the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions, N is the number of adjacent reprojection regions to the center of the current reprojection region, exp(·) is an exponential function with the natural constant as the base, (x, y) represents the coordinates of the center of the current reprojection region, x is the x-coordinate of the center of the current reprojection region, y is the y-coordinate of the center of the current reprojection region, (x i y i Let be the coordinates of the i-th adjacent reprojection region center of the current reprojection region center, ‖·‖ 2 σ is the square of the L2 norm, i is the index of the adjacent reprojection region, and σ is a parameter that controls the influence range of the 3D Gaussian ellipsoid.

[0031] Preferably, the splitting control function value of the current reprojection region is calculated based on the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions, the edge strength, the local variance, the normalized gradient magnitude, and the absolute value of the dot product of the gradient vector and the normal vector. The formula is as follows:

[0032]

[0033] Where S(x, y) is the splitting control function value of the current reprojection region, and E img (x, y) represents the edge intensity at the center of the current reprojection region, and α1 is the local variance weight. S is the local variance of the center of the current reprojection region, α2 is the normalized gradient magnitude weight, and S gauss (x, y) represents the normalized gradient magnitude at the center of the current reprojection region, α3 is the gradient direction matching weight, and D direction (x, y) is the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region, α4 ​​is the weight of the spatial coupling effect, and C space (x, y) is the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions of this region.

[0034] Preferably, the process of acquiring the sparse point cloud of the scene to be reconstructed includes:

[0035] A set of two-dimensional images of the scene to be reconstructed from different perspectives are used to generate a sparse point cloud of the scene to be reconstructed using the SfM algorithm.

[0036] The present invention also provides a three-dimensional scene reconstruction and rendering system, comprising:

[0037] The initial generation module is used to construct an initial set of 3D Gaussian ellipsoids based on the 3D Gaussian ellipsoids corresponding to each sparse point cloud of the scene to be reconstructed.

[0038] The reprojection region mapping module is used to obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; with each reprojection region as the center, all reprojection regions within a set pixel distance range are taken as the adjacent reprojection regions of that reprojection region.

[0039] The split control function calculation module is used to calculate the split control function value of each reprojection region based on the sum of the Gaussian kernel function values ​​from the center of each reprojection region to the centers of all adjacent reprojection regions of that region, as well as the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector of the reprojection region center.

[0040] The generation module is used to determine whether the split control function values ​​of all reprojected regions are less than or equal to a set threshold. If there is a reprojected region with a split control function value greater than the set threshold, the 3D Gaussian ellipsoid corresponding to the reprojected region is split, the 3D Gaussian ellipsoid set is iteratively updated, and the reprojected region mapping module is returned to be executed. If all values ​​are less than or equal to the set threshold, the current 3D Gaussian ellipsoid set is used as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

[0041] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0042] This invention discloses a 3D scene reconstruction and rendering method and system. The invention constructs a splitting control function, comparing the splitting control function value of the reprojected region of each 3D Gaussian ellipsoid on a 2D image from different viewpoints with a set threshold to determine whether to split the 3D Gaussian ellipsoid. This function calculates the edge intensity at the center of the reprojected region, enabling more accurate identification of real and critical edges in the 2D image, enhancing sampling of high-frequency details, and avoiding insufficient or excessive splitting due to edge misjudgment. Introducing the local variance at the center of the reprojected region reflects the texture complexity of the region, guiding the Gaussian ellipsoid to increase its density in areas rich in detail, thus improving splitting accuracy. The normalized gradient magnitude at the center of the reprojected region is closely related to the boundary and structural dimensions of objects in the image, allowing for adaptive adjustment of the Gaussian ellipsoid size according to different image regions, such as detail regions and edge regions. This effectively avoids over-refinement in large-scale regions while increasing the density of the Gaussian ellipsoid in detail regions, improving reconstruction accuracy. Edges in images typically possess a clear directionality, especially at object contours or structural boundaries. By calculating the absolute value of the dot product of the gradient vector and normal vector at the center of the reprojected region, directional changes in edge regions can be accurately identified. This enhances the splitting intensity at edges, reducing false splits caused by insufficient directional changes. Considering that the 3D Gaussian ellipsoid does not exist independently in space, by calculating the sum of the Gaussian kernel function values ​​from the center of each reprojected region to the centers of all adjacent reprojected regions, information from the surrounding Gaussian sphere can be fully considered during the splitting process. This avoids over- or under-splitting caused by isolated decisions, maintaining the consistency and stability of the scene representation. Through multi-layered perception of high-frequency information, the splitting decisions of the 3D Gaussian ellipsoid match the actual high-frequency distribution in the scene, effectively mitigating jagged artifacts in 2D renderings and improving the accuracy of 3D scene reconstruction rendering.

[0043] This invention calculates Gaussian low-pass filter parameters based on the width, height, and number of pruned 3D Gaussian ellipsoids of the target 2D image. It precisely controls the coverage area of ​​the processed reprojection region parameters on the target 2D image plane. By adding the calculated Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix, it ensures that each reprojection region covers the necessary minimum pixel area within the effective gradient propagation range. Compared to traditional fixed Gaussian low-pass filter parameter values, this invention adaptively sets the Gaussian low-pass filter parameter values ​​according to the width, height, and number of pruned 3D Gaussian ellipsoids of the target 2D image. It automatically increases the Gaussian low-pass filter parameters in high-resolution or sparse Gaussian regions to enhance the smoothness of the processed reprojection region parameters and suppress jagged artifacts, and automatically decreases the Gaussian low-pass filter parameters in low-resolution or dense Gaussian regions to preserve the detailed features of the reprojection region and avoid excessive blurring, effectively improving the rendering accuracy of the 2D rendered image. Attached Figure Description

[0044] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart of the steps of a three-dimensional scene reconstruction and rendering method according to the present invention.

[0046] Figure 2 This is a flowchart illustrating a three-dimensional scene reconstruction and rendering method according to the present invention.

[0047] Figure 3 This is a schematic diagram comparing the process of the present invention with that of traditional pruning methods. Figure 3 (a) in the diagram is a flowchart of the traditional pruning method. Figure 3 (b) in the figure is a flowchart of the pruning method of the present invention.

[0048] Figure 4 This is a schematic diagram comparing the performance of the 3D scene reconstruction and rendering method of this invention with other methods on the MipNerf360 dataset.

[0049] Figure 5 This is a schematic diagram comparing the effects of the three-dimensional scene reconstruction and rendering method of this invention with other methods in low-frequency and high-frequency mixed scenes. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0051] Reference Figure 1 , Figure 2 As shown, this embodiment provides a three-dimensional scene reconstruction and rendering method, including the following steps:

[0052] Step S1: Based on the 3D Gaussian ellipsoid corresponding to each sparse point cloud of the scene to be reconstructed, construct an initial set of 3D Gaussian ellipsoids; each 3D Gaussian ellipsoid is defined by position (mean), covariance matrix and opacity α, while the directional appearance component (color) of the radiation field is represented by the spherical harmonic function SH (Spherical Harmonics);

[0053] In this embodiment, specifically, the process of acquiring the sparse point cloud of the scene to be reconstructed includes:

[0054] A set of two-dimensional images of the scene to be reconstructed from different perspectives are used to generate a sparse point cloud of the scene to be reconstructed using the SfM algorithm.

[0055] Step S2: Obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; take each reprojection region as the center, and take all reprojection regions within a set pixel distance range as the adjacent reprojection regions of that reprojection region.

[0056] Among them, taking the center position of each reprojection area as the center, the distance between all pixels in the adjacent reprojection areas of the current reprojection area and the center of the current reprojection area is less than or equal to the set pixel distance.

[0057] In this embodiment, specifically, each 3D Gaussian ellipsoid is mapped onto two-dimensional images from different perspectives through multi-view projection, resulting in the reprojection region of each 3D Gaussian ellipsoid on two-dimensional images from different perspectives. Each reprojection region on a two-dimensional image corresponds to a 3D Gaussian ellipsoid.

[0058] Step S3: Based on the sum of the Gaussian kernel function values ​​from the center of each reprojection region to the centers of all adjacent reprojection regions, as well as the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector at the center of the reprojection region, calculate the split control function value for each reprojection region.

[0059] In this embodiment, specifically, the edge intensity of the center of the current reprojection region is obtained by weighted summation of the magnitudes of the gradients of the three RGB channels at the center of the current reprojection region.

[0060] The formula for calculating the edge intensity at the center of the current reprojection region is:

[0061]

[0062] Among them, E img (x, y) represents the edge intensity at the center of the current reprojection region, α is the gradient weight of the R channel, β is the gradient weight of the G channel, and γ is the gradient weight of the B channel. Let R be the gradient vector of the channel at (x, y). Let G be the gradient vector of the channel at (x, y). Let be the gradient vector of channel B at (x, y), where (x, y) are the coordinates of the center of the current reprojection region, x is the x-coordinate of the center of the current reprojection region, y is the y-coordinate of the center of the current reprojection region, and ‖·‖ is the Euclidean norm, representing the magnitude.

[0063] By calculating the edge intensity at the center of the reprojection region, we can more accurately identify the real and critical edge parts in the two-dimensional image, enhance the sampling of high-frequency details, and avoid insufficient or excessive splitting caused by edge misjudgment.

[0064] Calculate the absolute values ​​of the differences between the pixel values ​​at the center of the current reprojection region and the pixel values ​​of other pixels in the region, and take the average value as the local variance of the center of the current reprojection region.

[0065] The formula for calculating the local variance at the center of the current reprojection region is:

[0066]

[0067] in, Let M be the local variance of the center of the current reprojection region, M be the number of pixels in the current reprojection region excluding the center, and I(x, y) be the pixel value at the center of the current reprojection region. j y j ) represents the pixel value of the j-th pixel in the current reprojection region, excluding the center.

[0068] The local variance at the center of the reprojection region reflects the texture complexity of the region in the image. In regions with complex textures, even if the edge detection algorithm fails to produce a strong response, changes in local density can still indicate that the region needs more 3D Gaussian ellipsoids for fine modeling. By introducing this factor, the density of 3D Gaussian ellipsoids can be increased in regions with rich details, thereby improving splitting accuracy.

[0069] The gradient magnitude at the center of the current reprojection region is normalized to obtain the normalized gradient magnitude at the center of the current reprojection region.

[0070] The formula for calculating the normalized gradient magnitude at the center of the current reprojection region is:

[0071]

[0072] Among them, S gauss (x, y) represents the normalized gradient magnitude at the center of the current reprojection region. Let be the gradient vector at the center of the current reprojection region, ∈ be the regularization term to avoid division by zero error, and ‖·‖ be the Euclidean norm, representing the magnitude. Let (x, y) be the gradient vector of the center of the current reprojection region, where (x, y) are the coordinates of the center of the current reprojection region, x is the x-coordinate of the center of the current reprojection region, and y is the y-coordinate of the center of the current reprojection region.

[0073] The normalized gradient magnitude at the center of the reprojection region reflects the close relationship between the scale variation of the 3D Gaussian ellipsoid and the boundary and structural dimensions of objects in the image. By considering the scale variation of the Gaussian ellipsoid, the size of the Gaussian ellipsoid can be adaptively adjusted according to different image regions (such as detail regions, edge regions, etc.). This approach can effectively avoid over-refinement in large-scale regions, while increasing the density of the 3D Gaussian ellipsoid in detail regions, thus improving the accuracy of 3D model reconstruction.

[0074] The formula for calculating the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region is:

[0075]

[0076] Among them, D direction (x, y) is the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region. The gradient vector at the center of the current reprojection region. Let |·| be the normal vector of the center of the current reprojection region, and |·| be its absolute value.

[0077] Edges in an image typically have a clear directionality, especially at the contours or structural boundaries of objects. By calculating the matching degree between the gradient direction at the center of the reprojection region and the normal vector, the directional changes in the edge region can be accurately identified, thereby enhancing the splitting intensity in the edge region, effectively improving splitting accuracy, and reducing false splitting caused by insufficient directional changes.

[0078] The formula for calculating the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions is as follows:

[0079]

[0080] Among them, C space (x, y) represents the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions, N is the number of adjacent reprojection regions to the center of the current reprojection region, exp(·) is an exponential function with the natural constant as the base, (x, y) represents the coordinates of the center of the current reprojection region, x is the x-coordinate of the center of the current reprojection region, y is the y-coordinate of the center of the current reprojection region, (x i y i Let be the coordinates of the i-th adjacent reprojection region center of the current reprojection region center, ‖·‖ 2 σ is the square of the L2 norm, i is the index of the adjacent reprojection region, and σ is a parameter that controls the influence range of the 3D Gaussian ellipsoid.

[0081] Gaussian spheres do not exist independently in space; the interaction between adjacent Gaussian spheres may affect the splitting decision. The sum of the Gaussian kernel function values ​​from the center of the reprojection region to the centers of all adjacent reprojection regions in that region can reflect the spatial coupling effect of the 3D Gaussian ellipsoid, so that the information of the surrounding 3D Gaussian ellipsoids can be taken into account during the splitting decision process, avoiding excessive or insufficient splitting. This avoids the problem of local over-refinement or coarsening that occurs in traditional methods, and improves the coordination and consistency of 3D model reconstruction.

[0082] In this embodiment, preferably, the splitting control function value of the current reprojection region is calculated based on the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions, the edge strength, the local variance, the normalized gradient magnitude, and the absolute value of the dot product of the gradient vector and the normal vector. The formula is as follows:

[0083]

[0084] Where S(x, y) is the splitting control function value of the current reprojection region, and E img (x, y) represents the edge intensity at the center of the current reprojection region, and α1 is the local variance weight. S is the local variance of the center of the current reprojection region, α2 is the normalized gradient magnitude weight, and S gauss (x, y) represents the normalized gradient magnitude at the center of the current reprojection region, α3 is the gradient direction matching weight, and D direction (x, y) is the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region, α4 ​​is the weight of the spatial coupling effect, and C space (x, y) is the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions of this region.

[0085] This invention uses the edge intensity at the center of the current reprojection region as the basic weight to ensure that splitting occurs preferentially in high-contrast edge regions, directly targeting the high-frequency sources of jagged artifacts and ensuring the stability of the basic edge detection mechanism. At the same time, it allows features of each dimension to provide positive or negative corrections when necessary. This design enables the splitting strategy to inherit the advantages of traditional edge detection and accurately locate high-frequency information regions through the synergistic effect of multi-dimensional features.

[0086] This invention designs a splitting control function that considers not only edge strength, but also local density variations, scale variations, directional information, and the spatial coupling effect between 3D Gaussian ellipsoids. This multi-dimensional comprehensive approach overcomes the limitations of traditional methods that rely solely on edge detection, providing a more comprehensive and accurate basis for splitting decisions.

[0087] Step S4: Determine whether the split control function values ​​of all reprojected regions are less than or equal to the set threshold; if there is a reprojected region with a split control function value greater than the set threshold, split the 3D Gaussian ellipsoid corresponding to the reprojected region, iteratively update the 3D Gaussian ellipsoid set, and return to execute step S2; if all are less than or equal to the set threshold, use the current 3D Gaussian ellipsoid set as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

[0088] like Figure 3 As shown, Figure 3This is a schematic diagram comparing the process of the present invention with that of traditional pruning methods. Figure 3 (a) in the diagram is a flowchart of the traditional pruning method. Figure 3 (b) in the figure is a flowchart of the pruning method of the present invention.

[0089] In this embodiment, preferably, the step of using the current 3D Gaussian ellipsoid set as the scene densification result to generate a 3D model and a 2D rendering of the scene to be reconstructed includes:

[0090] Based on the size of the reprojection region of each 3D Gaussian ellipsoid on the 2D image from different viewpoints, the gradient magnitude and transparency of each pixel, the 3D Gaussian ellipsoid set is pruned, and the 3D model of the scene to be reconstructed is generated using the pruned 3D Gaussian ellipsoid set.

[0091] Based on the camera extrinsics, the pruned 3D Gaussian ellipsoid set is projected onto the target 2D image plane (i.e., splatting). After processing each reprojection region on the target 2D image plane with a Gaussian low-pass filter, the parameters of each processed reprojection region are rasterized with differentiable tiles to generate a 2D rendering of the scene to be reconstructed.

[0092] In this embodiment, optionally, the pruning of the 3D Gaussian ellipsoid set based on the size of the reprojection region of each 3D Gaussian ellipsoid on the two-dimensional image from different viewpoints, the gradient magnitude of each pixel, and the transparency includes:

[0093] Determine whether the average gradient magnitude of the current 3D Gaussian ellipsoid reprojection region is less than the set gradient threshold. If it is less, then remove the current 3D Gaussian ellipsoid.

[0094] Determine whether the average transparency of the current 3D Gaussian ellipsoid reprojection region is less than the set transparency threshold. If it is less, then remove the current 3D Gaussian ellipsoid.

[0095] Determine if the size of the current 3D Gaussian ellipsoid reprojection region is smaller than a set size threshold. If it is smaller, then remove the current 3D Gaussian ellipsoid.

[0096] Compared to traditional pruning methods that rely solely on a single pruning condition, such as Gaussian weights below a threshold, which can easily lead to the accidental deletion of Gaussian ellipses in critical detail areas or the retention of redundant Gaussian ellipses, this invention prunes the 3D Gaussian ellipsoid set based on the size of the reprojection region of each 3D Gaussian ellipsoid on the 2D image from different viewpoints, the gradient magnitude of each pixel, and its transparency. This achieves refined selection of Gaussian ellipsoids, suppressing 3D Gaussian ellipsoids with low transparency, reducing redundant model elements that contribute little to the rendering result, retaining more Gaussian ellipsoids in areas with high gradient magnitudes to ensure sampling density of high-frequency details, eliminating Gaussian ellipsoids with excessively small reprojection regions to avoid invalid calculations, and retaining large-sized Gaussian ellipsoids to maintain the macroscopic representation of the scene structure. This effectively avoids the jagged edges caused by the aliasing of high-frequency information and low-frequency sampling.

[0097] The 3D Gaussian ellipsoid is determined by a complete 3D covariance matrix defined in world space, centered at a point (mean) μ, and its spatial distribution is as follows:

[0098]

[0099] During rendering blending, Gaussian radiance values ​​are modulated using weighting factors. To achieve rendering, a 3D Gaussian ellipsoid needs to be projected onto a 2D image plane. Given the view transformation matrix W, the covariance matrix Σ in the camera coordinate system... ′ The affine approximation Jacobian matrix J can be derived through projection transformation, with the formula: Σ ′ =JWΣW T J T Ignore Σ ′ After obtaining the third-dimensional component, a two-dimensional variance matrix can be obtained, the structure of which is consistent with the projection method based on the plane point normal.

[0100] Another approach is to optimize the covariance matrix Σ to obtain a 3DGS representing the radiation field. However, the covariance matrix only has physical meaning when it is positive semi-definite. Gradient descent is typically used for optimization of all parameters, but this method is difficult to constrain to generate an efficient matrix, and update steps and gradients can easily produce invalid covariance matrices.

[0101] Therefore, this invention selects a more intuitive yet equally expressive representation method for optimization. The covariance matrix of 3DGS is analogous to describing the shape of an ellipsoid; given a scaling matrix S and a rotation matrix R, the corresponding Σ can be found using the formula: Σ = RSS. T R TIn order to optimize these two factors independently, the scaling and rotation components are stored separately. Scaling is represented by a 3D vector s and rotation by a quaternion q. They can be easily converted into their respective matrices and combined together. Normalization of q can effectively avoid the problem of generating invalid matrices and obtain an effective unit quaternion.

[0102] To avoid the significant computational overhead caused by automatic differentiation during training, the gradients of each parameter are explicitly derived, thus avoiding the computational overhead of automatic differentiation. This allows the 3D Gaussian ellipsoid to adaptively optimize into a compact representation of complex geometric structures (such as planes, curved surfaces, and anisotropic objects) in the scene.

[0103] In this embodiment, preferably, each reprojection region on the target two-dimensional image plane is processed by a Gaussian low-pass filter to obtain parameters for each processed reprojection region, including:

[0104] Based on the width, height, and number of pruned 3D Gaussian ellipsoids of the target 2D image, calculate the parameters of the Gaussian low-pass filter.

[0105] By adding Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix of each reprojection region on the target two-dimensional image plane, the parameters of each processed reprojection region are obtained.

[0106] The formula for calculating the parameters of the Gaussian low-pass filter is as follows:

[0107]

[0108] In the formula, s is the Gaussian low-pass filter parameter, H is the height of the target 2D image, W is the width of the target 2D image, and K is the number of pruned 3D Gaussian ellipsoids.

[0109] A Gaussian low-pass filter is a common image processing tool primarily used to suppress high-frequency components, thereby reducing noise and artifacts caused by over-sharpening. Its core idea is to perform a locally weighted average in the image, thus achieving smoothing. Unlike traditional mean filters, Gaussian filters weight pixel neighborhoods according to a normal distribution, resulting in smaller weights for pixels farther from the center, thus smoothing image details more naturally.

[0110] The jagged edges arise from high-frequency aliasing, especially noticeable when the image sampling rate is insufficient. Since edge regions often contain rich high-frequency information, and low-frequency sampling cannot effectively capture these details, information folding occurs, forming artifacts.

[0111] Studies have found that when the reprojected region in the projected 2D image plane is smaller than a single pixel, directly using it will lead to visual artifacts. Therefore, the scale of the reprojected region in the projected 2D image plane can be enlarged by adding a small value to the diagonal elements of the covariance matrix, as shown in the following formula:

[0112]

[0113] in, The position of the k-th reprojection region on the target two-dimensional image plane The reprojected region parameters are exp(·), which is an exponential function with the natural constant as the base. Let μ′ be any position in the k-th reprojection region on the target two-dimensional image plane. k Let be the center coordinates of the k-th reprojected region on the target two-dimensional image plane, (·). T For the transpose operation, Σ′ k Let be the covariance matrix of the k-th reprojection region, s be the parameters of the Gaussian low-pass filter, and I be the identity matrix.

[0114] This process can also be understood as the reprojection region in the target two-dimensional image plane obtained by projection. With a mean μ = 0 and a variance Convolution is performed on a Gaussian low-pass filter h, i.e. This has proven to be a crucial step in preventing aliasing. After convolution with a low-pass filter, the reprojected region on the target 2D image plane approximates a circle, the radius of which is determined by the 2D covariance matrix Σ′. k +sI is defined as three times the larger eigenvalue in the I-value.

[0115] In the previous study, the Gaussian low-pass filter parameter s was a predefined value, typically set to 0.3. However, this invention notes that the Gaussian low-pass filter parameter s ensures that each reprojection region must cover a minimum area in the target 2D image plane. Since Gaussians only receive gradients within a few standard deviations, learning from a wider region is crucial for Gaussians to learn the coarse structural information of the scene. Therefore, this invention calculates the Gaussian low-pass filter parameter s to allow Gaussians to cover a wider region early in training and gradually learn from more local regions, ensuring that the minimum area each reprojection region must cover in the target 2D image plane is greater than 9πs. 2 s represents the area of ​​the target two-dimensional image. A Gaussian low-pass filter effectively mitigates jagged edges by suppressing high-frequency components, resulting in a smoother image.

[0116] This invention differs from traditional global filtering strategies by combining a 3D Gaussian ellipsoid splitting strategy, Gaussian low-pass filtering, and a dynamic pruning mechanism. It adaptively adjusts the filtering scope, applying it only to the areas requiring processing, thus avoiding unnecessary blurring of non-aliased regions. Specifically, traditional Gaussian filtering typically applies uniform smoothing to the entire image, while the 3D Gaussian ellipsoid splitting strategy dynamically splits Gaussians based on features of the 2D projected image, such as edge strength, texture complexity, and edge directionality (e.g., the dot product of gradient and normal vectors). This generates more sub-Gaussians in high-frequency regions (e.g., sharp edges, complex textures), increasing local sampling density and reducing the potential risk of aliasing at its source. The Gaussian dynamic pruning method adaptively evaluates multiple dimensions, including gradient changes, transparency information, and screen space size, applying filtering only to high-contrast edge regions or other areas susceptible to aliasing. In this way, the method effectively reduces aliasing while preserving key image details, thereby improving the quality of the final rendering result.

[0117] This embodiment compares the 2D rendered image with the ground truth image, calculates the loss function, updates the parameters of 3DGS through backpropagation, and sends them to the adaptive density control module to further optimize the distribution of the point cloud.

[0118] To validate the proposed 3D scene reconstruction and rendering method, the NeRF-Synthetic dataset was selected as the primary dataset for the experiments. This dataset, proposed by Mildenhall et al. in their pioneering work on NeRF, is widely used in research on view synthesis and 3D scene reconstruction tasks. The NeRF-Synthetic dataset contains eight high-quality synthetic scenes (such as Chair, Drums, Ficus, Hotdog, Lego, Materials, Mic, and Ship), each consisting of 100 RGB images taken from different perspectives at a resolution of 800x800. Each image is equipped with precise camera parameters (including position, rotation, and focal length), providing a reliable foundation for model training and evaluation. The dataset's scene design is highly professional and complex, containing rich lighting variations, complex material reflections, and intricate geometric structures, effectively testing the model's ability to handle high-frequency details, lighting variations, and complex geometry. Furthermore, the dataset's images have transparent backgrounds (alpha channel), allowing researchers to flexibly set backgrounds (such as solid color backgrounds or realistic backgrounds) to better simulate real-world application scenarios.

[0119] Although the NeRF-Synthetic dataset was originally designed for NeRF, its high-quality multi-view images and accurate camera parameters are equally suitable for 3DGS research. 3DGS, by explicitly representing scenes as a series of learnable 3D Gaussian distributions, can efficiently render complex scenes and has achieved significant progress in both rendering speed and quality. The complex lighting and geometry in the NeRF-Synthetic dataset provide an ideal testing environment for 3DGS, enabling the validation of model performance in high-frequency details and edge smoothing. Furthermore, the dataset's transparent background design allows 3DGS to handle background information more flexibly, avoiding the influence of background interference on rendering results. The NeRF-Synthetic dataset was chosen because of its widespread acceptance, high-quality data, diversity, and transparent background design. This dataset is one of the benchmark datasets in the field of 3D rendering and has been used in numerous research works, ensuring the reliability and comparability of experimental results. Its high-resolution images and accurate camera parameters provide a solid foundation for the optimization and evaluation of 3DGS, while the diverse scene types (such as objects, buildings, and natural scenes) can validate the model's generalization ability across different scenarios. Experiments on this dataset allow for a comprehensive evaluation of 3DGS's rendering capabilities in complex scenes and a fair comparison with existing methods.

[0120] This invention selects Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as evaluation metrics to comprehensively assess the performance of the proposed 3D scene reconstruction rendering method in the deblurring task. These three evaluation metrics have different advantages and can reflect the quality of the image restored by the model from multiple dimensions.

[0121] PSNR is a classic evaluation metric widely used to measure the restoration quality in image reconstruction tasks. It reflects the error level of an image by calculating the mean squared error (MSE) between the restored image and the original image. Due to its ubiquity and historical background, PSNR remains an indispensable basic metric in the field of image deblurring. SSIM, as a perceptual structural similarity measure, can evaluate the similarity of images from three aspects: brightness, contrast, and structure. Compared to PSNR, SSIM can better simulate the human eye's perception of image quality. SSIM is considered to be closer to the cognitive mode of the human visual system than PSNR, therefore it can more accurately reflect the preservation of image details and structure when evaluating image quality. For deblurring tasks, SSIM can effectively evaluate the structural consistency of the restored image, avoiding the limitations of relying solely on pixel-level errors. LPIPS is a novel metric proposed in recent years for evaluating image perceptual quality. LPIPS can better reflect the subjective perception of the human visual system on images, especially when dealing with complex image content, where it outperforms traditional PSNR and SSIM. LPIPS has higher sensitivity to the ability to preserve image details and textures, making it particularly suitable for evaluating subtle perceptual differences in image restoration. The combination of these three elements enables this embodiment to provide a more comprehensive and accurate evaluation of the performance of LPD-3DGS (LowPass Dynamic Prune-3D Gaussian Splatting).

[0122] As shown in the figure Figure 4 This diagram illustrates the performance comparison between the proposed 3D scene reconstruction and rendering method of this invention and other methods on the MipNerf360 dataset. The proposed 3D scene reconstruction and rendering method (Ours) is compared with GroundTruth, Plenoxels, INCP, and Mip-NeRF.

[0123] As shown in Table 1, Table 1 is a comparison of the PSNR values ​​of the three-dimensional scene reconstruction and rendering method of the present invention with those of Plenoxels, INGP-Base, Mip-NeRF, Point-NeRF, and 3DGS.

[0124] Table 1

[0125]

[0126]

[0127] As shown in Table 1, the present invention achieved the highest or second-highest values ​​in all 8 test scenarios, with an average PSNR of 33.79, surpassing cutting-edge methods such as 3DGS (33.64) and Point-NeRF (33.30).

[0128] like Figure 5 As shown, Figure 5 This diagram illustrates a comparison of the effects of the proposed 3D scene reconstruction and rendering method with other methods in mixed low-frequency and high-frequency scenes. The proposed 3D scene reconstruction and rendering method (Ours) is compared with GroundTruth and MSGS in mixed low-frequency and high-frequency scenes.

[0129] As shown in Table 2, Table 2 is a schematic diagram comparing the PSNR values ​​of the three-dimensional scene reconstruction and rendering method of the present invention with the experimental results of MSGS.

[0130] Table 2

[0131]

[0132] according to Figure 5 Table 2 demonstrates the superiority of the proposed 3D scene reconstruction and rendering method in low-frequency and high-frequency mixed scenes.

[0133] Plenoxels and INGP are the fastest NeRF methods recently. Plenoxels is a sparse voxel mesh-based method for efficiently rendering and optimizing implicit representations of 3D scenes. It encodes the scene as a voxel mesh, storing density and color information with spherical harmonics in each voxel. INGP, proposed by NVIDIA, is a method for fast training of neural scene representations, employing multiresolution hash encoding and a small MLP network. Mip-NeRF is an improved NeRF method that better handles anti-aliasing and scale variations by introducing the concept of MipMapping (multi-level texture sampling). Experimental results verify that the method proposed in this invention outperforms the 3DGS method.

[0134] To evaluate the performance contribution of each module in the Gaussian low-pass filter, 3D Gaussian ellipsoid splitting strategy, and dynamic pruning strategy of this invention, the improvement of network performance by the Gaussian low-pass filter, 3D Gaussian ellipsoid splitting strategy, and dynamic pruning strategy was tested separately, as shown in Table 3. Table 3 shows the results of the ablation experiments for each module.

[0135] Table 3

[0136]

[0137] This invention makes several key improvements to the traditional 3DGS method to enhance its performance in image anti-aliasing, especially in handling high-contrast edges and complex scenes. Compared with the classic 3DGS method, the method proposed in this invention shows significant advantages in several core metrics. Subsequently, experimental comparisons were conducted with the existing MS-GS method, proving that the 3D scene reconstruction and rendering method proposed in this invention still has advantages in scenes with a mixture of high and low frequencies.

[0138] This invention introduces a low-pass filter to suppress high-frequency noise, thereby smoothing image edges more naturally during the rendering process. Experimental results show that after introducing the low-pass filter, the PSNR and SSIM indices of the image are significantly improved, especially in the detail recovery of complex edge areas. Compared with the original 3DGS method, the addition of the low-pass filter effectively alleviates the jaggedness in high-contrast areas, while improving the smoothness of the image.

[0139] This invention also introduces a dynamic pruning strategy to more flexibly control the application area of ​​Gaussian filtering. Dynamic pruning adaptively determines whether filtering is needed for specific regions by deeply analyzing multi-dimensional features such as pixel gradients, transparency, and screen space size. This adaptive strategy allows the Gaussian filter to maintain global smoothness while avoiding excessive blurring of detailed areas. Experiments show that compared to the original 3DGS method, the introduction of dynamic pruning significantly enhances the model's generalization ability in different scenes, especially demonstrating higher robustness in rendering high-contrast details and complex backgrounds.

[0140] This invention significantly improves detail preservation by integrating a Gaussian low-pass filter, a 3D Gaussian ellipsoid splitting strategy, and a dynamic pruning strategy into the reconstruction rendering process, thereby effectively solving the detail loss problem caused by the uniformity of Gaussian filtering in traditional methods. Compared with the original 3DGS method, the joint optimization framework proposed in this invention exhibits higher accuracy when processing images with rich details, especially in the rendering effect of edge transition regions, achieving better visual quality and significantly improving the overall rendering detail and visual fidelity.

[0141] This second embodiment provides a three-dimensional scene reconstruction and rendering system, including:

[0142] The initial generation module is used to construct an initial set of 3D Gaussian ellipsoids based on the 3D Gaussian ellipsoids corresponding to each sparse point cloud of the scene to be reconstructed.

[0143] The reprojection region mapping module is used to obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; with each reprojection region as the center, all reprojection regions within a set pixel distance range are taken as the adjacent reprojection regions of that reprojection region.

[0144] The split control function calculation module is used to calculate the split control function value of each reprojection region based on the sum of the Gaussian kernel function values ​​from the center of each reprojection region to the centers of all adjacent reprojection regions of that region, as well as the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector of the reprojection region center.

[0145] The generation module is used to determine whether the split control function values ​​of all reprojected regions are less than or equal to a set threshold. If there is a reprojected region with a split control function value greater than the set threshold, the 3D Gaussian ellipsoid corresponding to the reprojected region is split, the 3D Gaussian ellipsoid set is iteratively updated, and the reprojected region mapping module is returned to be executed. If all values ​​are less than or equal to the set threshold, the current 3D Gaussian ellipsoid set is used as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method of rendering a three-dimensional scene reconstruction, the method comprising: include: Step S1: Construct an initial set of 3D Gaussian ellipsoids based on the 3D Gaussian ellipsoids corresponding to each sparse point cloud in the scene to be reconstructed; Step S2: Obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; take each reprojection region as the center, and take all reprojection regions within a set pixel distance range as the adjacent reprojection regions of that reprojection region. Step S3: Based on the sum of the Gaussian kernel function values ​​from the center of each reprojected region to the centers of all its adjacent reprojected regions, and the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector at the center of the reprojected region, calculate the splitting control function value for each reprojected region. The formula is as follows: ， in, This represents the splitting control function value for the current reprojected region. The edge intensity at the center of the current reprojection region. For local variance weights, The local variance of the center of the current reprojection region. To normalize the gradient magnitude weights, The normalized gradient magnitude at the center of the current reprojection region. The gradient direction matching degree weights, Let be the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region. For spatial coupling effect weights, It is the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions in that region; Step S4: Determine whether the split control function values ​​of all reprojected regions are less than or equal to the set threshold; if there is a reprojected region with a split control function value greater than the set threshold, split the 3D Gaussian ellipsoid corresponding to the reprojected region, iteratively update the 3D Gaussian ellipsoid set, and return to execute step S2; if all are less than or equal to the set threshold, use the current 3D Gaussian ellipsoid set as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

2. The three-dimensional scene reconstruction and rendering method according to claim 1, characterized in that, The step of using the current 3D Gaussian ellipsoid set as the scene densification result to generate a 3D model and a 2D rendering of the scene to be reconstructed includes: Based on the size of the reprojection region of each 3D Gaussian ellipsoid on the 2D image from different viewpoints, the gradient magnitude and transparency of each pixel, the 3D Gaussian ellipsoid set is pruned, and the 3D model of the scene to be reconstructed is generated using the pruned 3D Gaussian ellipsoid set. Based on the camera extrinsics, the pruned 3D Gaussian ellipsoid set is projected onto the target 2D image plane. After processing each reprojection region on the target 2D image plane with a Gaussian low-pass filter, the parameters of each processed reprojection region are rasterized with differentiable tiles to generate a 2D rendering of the scene to be reconstructed.

3. The three-dimensional scene reconstruction and rendering method according to claim 2, characterized in that, Each reprojection region on the target 2D image plane is processed using a Gaussian low-pass filter to obtain parameters for each processed reprojection region, including: Based on the width, height, and number of pruned 3D Gaussian ellipsoids of the target 2D image, calculate the parameters of the Gaussian low-pass filter. By adding Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix of each reprojection region on the target two-dimensional image plane, the parameters of each processed reprojection region are obtained. The formula for calculating the parameters of the Gaussian low-pass filter is as follows: ， In the formula, These are the parameters of a Gaussian low-pass filter. The height of the target 2D image, The width of the target 2D image. This represents the number of 3D Gaussian ellipsoids after pruning.

4. The three-dimensional scene reconstruction and rendering method according to claim 3, characterized in that, The process involves adding Gaussian low-pass filter parameters to the diagonal elements of the covariance matrix of each reprojection region on the target two-dimensional image plane to obtain the parameters of each processed reprojection region, including: ， in, For the target two-dimensional image plane, the first Location in each projection region Processed reprojection region parameters It is an exponential function with the natural constant as its base. For the target two-dimensional image plane, the first Any position within a reprojection region For the target two-dimensional image plane, the first The center coordinates of each reprojection region For transpose operation, For the first Covariance matrix of each reprojected region These are the parameters of a Gaussian low-pass filter. It is an identity matrix.

5. The three-dimensional scene reconstruction and rendering method according to claim 1, characterized in that, The edge intensity at the center of the current reprojection region is obtained by weighted summing of the magnitudes of the gradients of the three RGB channels at the center of the current reprojection region. Calculate the absolute values ​​of the differences between the pixel values ​​at the center of the current reprojection region and the pixel values ​​of other pixels in the region, and take the average value as the local variance of the center of the current reprojection region. The gradient magnitude at the center of the current reprojection region is normalized to obtain the normalized gradient magnitude at the center of the current reprojection region.

6. The three-dimensional scene reconstruction and rendering method according to claim 1, characterized in that, The formula for calculating the normalized gradient magnitude at the center of the current reprojection region is: ， in, The normalized gradient magnitude at the center of the current reprojection region. The gradient vector at the center of the current reprojection region. For regularization terms, Let Euclidean norm represent the magnitude. The gradient vector at the center of the current reprojection region. These are the coordinates of the center of the current reprojection region. The x-coordinate of the center of the current reprojection region. This is the ordinate of the center of the current reprojection region.

7. The three-dimensional scene reconstruction and rendering method according to claim 1, characterized in that, The formula for calculating the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions is as follows: ， in, It is the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all its adjacent reprojection regions. This represents the number of adjacent reprojection regions to the center of the current reprojection region. It is an exponential function with the natural constant as its base. These are the coordinates of the center of the current reprojection region. The x-coordinate of the center of the current reprojection region. The ordinate of the center of the current reprojection region. The center of the current reprojection region The coordinates of the centers of adjacent reprojected regions The square of the L2 norm, Index for adjacent reprojection regions. Parameters for controlling the influence range of the 3D Gaussian ellipsoid.

8. The three-dimensional scene reconstruction and rendering method according to claim 1, characterized in that, The process of acquiring the sparse point cloud of the scene to be reconstructed includes: A set of two-dimensional images of the scene to be reconstructed from different perspectives are used to generate a sparse point cloud of the scene to be reconstructed using the SfM algorithm.

9. A three-dimensional scene reconstruction and rendering system, characterized in that, include: The initial generation module is used to construct an initial set of 3D Gaussian ellipsoids based on the 3D Gaussian ellipsoids corresponding to each sparse point cloud of the scene to be reconstructed. The reprojection region mapping module is used to obtain the reprojection region of each 3D Gaussian ellipsoid in the 3D Gaussian ellipsoid set on the two-dimensional image from different viewpoints; with each reprojection region as the center, all reprojection regions within a set pixel distance range are taken as the adjacent reprojection regions of that reprojection region. The split control function calculation module is used to calculate the split control function value for each reprojection region based on the sum of the Gaussian kernel function values ​​from the center of each reprojection region to the centers of all adjacent reprojection regions, as well as the edge strength, local variance, normalized gradient magnitude, and absolute value of the dot product of the gradient vector and the normal vector at the center of the reprojection region. The formula is as follows: ， in, This represents the splitting control function value for the current reprojected region. The edge intensity at the center of the current reprojection region. For local variance weights, The local variance of the center of the current reprojection region. To normalize the gradient magnitude weights, The normalized gradient magnitude at the center of the current reprojection region. The gradient direction matching degree weights, Let be the absolute value of the dot product of the gradient vector and the normal vector at the center of the current reprojection region. For spatial coupling effect weights, It is the sum of the Gaussian kernel function values ​​from the center of the current reprojection region to the centers of all adjacent reprojection regions in that region; The generation module is used to determine whether the split control function values ​​of all reprojected regions are less than or equal to a set threshold. If there is a reprojected region with a split control function value greater than the set threshold, the 3D Gaussian ellipsoid corresponding to the reprojected region is split, the 3D Gaussian ellipsoid set is iteratively updated, and the reprojected region mapping module is returned to be executed. If all values ​​are less than or equal to the set threshold, the current 3D Gaussian ellipsoid set is used as the scene densification result to generate the 3D model and 2D rendering map of the scene to be reconstructed.

Images