Optimization method and device for sparse-view three-dimensional gaussian blur

Abstract

The application relates to a sparse-view three-dimensional Gaussian splatting optimization method and device, and belongs to the three-dimensional reconstruction technical field in computer vision. The method comprises the following steps: sparse-view images are collected; a multi-view stereo vision model based on deep learning is used to generate a geometrically consistent depth map from the sparse-view images, the depth map is converted into a point cloud, and dense point clouds are obtained by fusion; a voxel-guided farthest point sampling method is used to sample the dense point clouds, initialization point clouds are obtained, and a three-dimensional Gaussian field is constructed; a geometry renderer is used to render the three-dimensional Gaussian field, rendering depth and rendering normals are obtained; a multilevel geometric regularization loss function is constructed, and the three-dimensional Gaussian field is optimized; and the three-dimensional Gaussian field is optimized and adjusted based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy, so that an optimized three-dimensional Gaussian field is obtained. The application solves the problems of initialization failure, insufficient geometric supervision and primitive out-of-control of 3D Gaussian splatting under sparse views.

Description

TECHNICAL FIELD

[0001] The present application relates to a sparse-view three-dimensional Gaussian splash optimization method and device, belonging to the field of three-dimensional reconstruction in computer vision. BACKGROUND

[0002] Three-dimensional reconstruction and novel view synthesis technology is a core supporting technology in the fields of computer vision, virtual reality, autonomous driving, etc., and its goal is to recover the three-dimensional geometric structure of the scene from the input images and generate realistic rendering images of any view point. Traditional three-dimensional reconstruction methods (such as motion recovery structure SfM, multi-view stereo matching MVS) and early neural rendering technology (such as neural radiation field NeRF) often rely on a large number of dense input views (usually tens to hundreds of images) to ensure the reconstruction accuracy and rendering quality. However, in practical applications, the acquisition of dense view data often faces significant bottlenecks: on the one hand, professional equipment is expensive to shoot, and there are operational restrictions on the shooting environment (such as precious cultural relics, large outdoor scenes); on the other hand, due to objective conditions (such as dynamic scenes, inaccessible areas), some scenes cannot obtain enough views, making it difficult for traditional methods to be applicable.

[0003] To overcome the above limitations, sparse-view novel view synthesis technology has emerged, which aims to achieve high-quality scene three-dimensional reconstruction and novel view rendering using only a very limited number (usually 2-8) of input images. Although this technology has important practical value, existing methods still have many inherent defects under sparse input conditions: (1) insufficient geometric constraints: due to the limited input views, the geometric constraint information between multiple views is scarce, resulting in a blurred and incomplete scene geometric structure, which is difficult to accurately reflect the details of the real scene; (2) poor generalization ability: the model is prone to overfitting to the limited input views, and there is a significant deviation when generating images of unobserved views, which cannot be generalized to new view points stably.

[0004] As a new three-dimensional representation and rendering technology in recent years, 3D Gaussian Splatting (3DGS) represents the scene by a set of 3D Gaussian primitives, and has the advantages of rendering efficiency and quality, and is widely used in novel view synthesis tasks. However, when the 3D Gaussian Splatting technology is applied to sparse view scenes, there are still three key technical challenges: (1) initialization failure problem: existing 3D Gaussian Splatting usually relies on the point cloud generated by SfM technology (such as COLMAP) to initialize Gaussian primitives, but in sparse input, the point cloud generated by SfM is extremely sparse or even completely missing, which leads to that the Gaussian field cannot cover the complete scene, and there is serious information loss in the initialization stage; (2) insufficient supervision problem: existing methods mostly rely on the photometric loss between the rendered image and the input image for optimization, but in sparse view, this single supervision method cannot effectively constrain the scene geometry, resulting in significant deviation between the rendered depth and the real geometry, and phenomena such as surface fragmentation, floating objects and artifacts; (3) Gaussian primitive out-of-control problem: existing Gaussian primitive constraint methods fail in sparse view: on the one hand, the original scale constraint controls the size of the primitive by directly clipping large-scale Gaussians, but in sparse scenes, large-scale Gaussians may be the key to maintaining the integrity of the scene, and blind clipping may lead to structural collapse; on the other hand, fixed opacity threshold cannot effectively remove semi-transparent Gaussians, resulting in residual artifacts, affecting rendering quality and causing redundant calculations.

[0005] In summary, in the application of 3D Gaussian Splatting in sparse view scenes, the existing technology still has problems such as insufficient initialization robustness, incomplete geometric supervision, and failure of Gaussian primitive control, and a systematic optimization method is needed to break through the bottleneck of three-dimensional reconstruction and novel view synthesis in sparse view scenes. SUMMARY

[0006] To solve the above problems, the present application provides an optimization method and device for sparse view three-dimensional Gaussian Splatting, which can enhance the generalization ability of sparse input scenes and provide efficient and reliable technical support for virtual reality, cultural heritage and other fields.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] In a first aspect, the present application provides an optimization method for sparse view three-dimensional Gaussian Splatting, comprising the following steps:

[0009] Step S1, collecting sparse view images, the sparse view images are target scene images collected in actual application scenarios, and the number of the target scene images is extremely limited (usually 2-8);

[0010] Step S2, generating a geometrically consistent depth map based on a deep learning-based multi-view stereo vision model, converting the depth map into a point cloud and fusing to obtain a dense point cloud;

[0011] Step S3, sampling the dense point cloud by using voxel-guided farthest point sampling to obtain an initialized point cloud, and constructing a three-dimensional Gaussian field based on the initialized point cloud;

[0012] Step S4, rendering the three-dimensional Gaussian field by using an enhanced geometry renderer to obtain rendered depth and rendered normal, constructing a multi-level geometry regularization loss function, and optimizing the three-dimensional Gaussian field, the multi-level geometry regularization loss function including MVS depth loss, monocular depth loss, monocular normal loss and depth-normal consistency loss;

[0013] Step S5, optimizing and adjusting the three-dimensional Gaussian field based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field.

[0014] As a possible implementation manner of the embodiment, the step S1 includes the following steps:

[0015] Step S11, adjusting the shooting position and angle of the shooting device according to the actual shootable range of the scene, preferentially covering the core structure of the scene, but without pursuing a "dense coverage" layout to simulate a passive sparse scene with limited view angle;

[0016] Step S12, collecting 1-2 images at a random view angle at each selected position, without forcibly requiring the overlap ratio of adjacent view angle images, and storing the collected images in a general format at a conventional resolution;

[0017] Step S13, directly taking all the 2-8 collected images as subsequent processing input.

[0018] As a possible implementation manner of the embodiment, the step S2 includes the following steps:

[0019] Step S21, processing the sparse view angle images by using a deep learning-based multi-view stereo vision model to generate a depth map with geometric consistency, the deep learning-based multi-view stereo vision model being a pre-trained MVS model, and the generated depth map having geometric consistency ensured by multi-view geometric constraint, the geometric consistency being that the spatial position matching error of the multi-view depth map in the overlapping area is lower than a preset threshold;

[0020] Step S22, converting the generated depth map into a corresponding point cloud, and fusing the point cloud to obtain a dense point cloud covering the complete scene.

[0021] As a possible implementation manner of the embodiment, in the step S22, the process of converting the generated depth map into a corresponding point cloud comprises: converting the depth value of each pixel in the depth map into a three-dimensional space coordinate based on the internal parameter and the external parameter of the camera, to form a point cloud corresponding to the view angle; the internal parameter of the camera comprises a focal length and a principal point coordinate, and the external parameter comprises a rotation matrix and a translation vector, which are used to convert the depth value in a pixel coordinate system into a three-dimensional coordinate in a world coordinate system; the fusion processing comprises a de-duplication, registration and splicing operation on the point clouds generated in multiple view angles, wherein: the de-duplication is to remove redundant points with spatial positions coinciding or distances less than a preset threshold; the registration is to align the point clouds in multiple view angles to the same coordinate system through a spatial transformation; and the splicing is to integrate the registered point clouds to form a dense point cloud covering a complete scene.

[0022] As a possible implementation manner of the embodiment, the step S3 comprises the following steps:

[0023] In the step S31, the dense point cloud is sampled by using a voxel-guided farthest point sampling method to obtain an initialized point cloud.

[0024] In the step S32, a three-dimensional Gaussian field is constructed based on the initialized point cloud, wherein the three-dimensional Gaussian field is composed of a plurality of Gaussian primitives, and the parameters of each Gaussian primitive comprise a center position, a scaling factor, a rotation quaternion, an opacity value and a color feature.

[0025] As a possible implementation manner of the embodiment, in the step S31, the voxel-guided farthest point sampling method comprises:

[0026] In the step S311, the scene where the dense point cloud is located is voxelized to divide a plurality of voxel units.

[0027] In the step S312, the point density in each voxel unit is calculated, and the point density is a ratio of the number of point clouds contained in the voxel unit to the volume of the voxel unit.

[0028] In the step S313, each voxel unit is assigned with an inverse sampling rate according to the point density, wherein the voxel unit with a high point density is assigned with a low sampling rate, and the voxel unit with a low point density is assigned with a high sampling rate.

[0029] In the step S314, in each voxel unit, the farthest point sampling algorithm is executed according to the assigned inverse sampling rate to obtain the initialized point cloud.

[0030] As a possible implementation manner of the embodiment, the step S4 comprises the following steps:

[0031] In the step S41, the three-dimensional Gaussian field is rendered by using an enhanced geometry renderer to generate a rendered depth and a rendered normal.

[0032] Step S42, constructing a multi-level geometric regularization loss function;

[0033] Step S43, optimizing the parameters of the three-dimensional Gaussian field based on the multi-level geometric regularization loss function to improve the geometric accuracy and surface smoothness of the scene reconstruction.

[0034] As a possible implementation manner of the embodiment, in step S41, the process of calculating the rendered depth by the enhanced geometry renderer includes:

[0035] Projecting the center of the Gaussian primitive to the camera coordinate system to obtain a projected center coordinate (x', y', z');

[0036] Calculating the depth value at the pixel based on the projected center coordinate and the relative pixel position:

[0037] ,

[0038] wherein, is the depth of the projected center of the i-th Gaussian primitive, Δx = x- , Δy = y- are the deviations of the pixel position (x, y) and the projected center , respectively, and p is a vector determined based on the Gaussian primitive parameters .

[0039] As a possible implementation manner of the embodiment, in step S41, the process of calculating the rendered normal by the enhanced geometry renderer includes:

[0040] Converting the normal vector from the ray space to the camera space:

[0041] ,

[0042] wherein, J represents a local affine matrix, is the depth of the projected center of the Gaussian primitive, is the depth of the center of the Gaussian primitive, and p is a vector determined based on the Gaussian primitive parameters.

[0043] As a possible implementation manner of the embodiment, in step S42, the MVS depth loss is an L1 loss between the rendered depth and the depth generated by the MVS model, and the formula is:

[0044] ,

[0045] wherein, is the rendered depth, is the MVS depth.

[0046] As a possible implementation manner of the embodiment, in step S42, the monocular depth loss is a Pearson correlation loss between the rendered depth and the monocular predicted depth, and the formula is:

[0047] ,

[0048] wherein represents the monocular depth estimated by the prior estimator, wherein Cov is a covariance function, and Var is a variance function; the monocular depth loss is realized by a global-local hierarchical strategy, including calculating the correlation loss for the global depth map and the randomly sampled local depth blocks respectively.

[0049] As a possible implementation manner of the embodiment, in step S42, the monocular normal loss is an L1 loss between the rendered normal and the monocular predicted normal:

[0050] ,

[0051] wherein represents the Gaussian rendered normal, represents the monocular normal.

[0052] As a possible implementation manner of the embodiment, in step S42, the depth-normal consistency loss is a cosine similarity loss between the rendered normal and the pseudo normal calculated based on the rendered depth:

[0053] ,

[0054] wherein represents the pseudo normal map, which is calculated by finite difference based on the rendered depth:

[0055] ,

[0056] wherein , are gradients of the 3D point in x and y directions respectively.

[0057] As a possible implementation manner of the embodiment, in step S42, the composite form of the multi-level geometric regularization loss function is:

[0058] ,

[0059] wherein respectively represent the weights corresponding to each loss, and .

[0060] As a possible implementation manner of the embodiment, the weight coefficient has a value range of: .

[0061] As a possible implementation manner of the embodiment, in step S42, the parameters of the three-dimensional Gaussian field include the center position of the Gaussian cell, the scaling factor, the rotation quaternion, the opacity value and the color feature, and the parameter optimization is achieved by minimizing the multi-level geometric regularization loss function through the back propagation algorithm.

[0062] As a possible implementation manner of the embodiment, the step S5 includes the following steps:

[0063] In step S51, the needle-shaped abnormal Gaussian cell in the three-dimensional Gaussian field is identified and removed based on a shape-size constraint criterion.

[0064] In step S52, the opacity value of the Gaussian cell in the three-dimensional Gaussian field is adjusted by using a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field.

[0065] As a possible implementation manner of the embodiment, the shape-size constraint criterion is that if the ratio of the longest axis to the second longest axis of the Gaussian cell is greater than a preset ratio threshold and the length of the longest axis is greater than a preset size threshold, the Gaussian cell is removed; the two-stage adaptive opacity constraint strategy includes a pruning stage and a fine-tuning stage, the pruning stage gradually increases the opacity threshold at a fixed interval to remove the translucent Gaussian cell, and the fine-tuning stage adaptively adjusts the final opacity threshold based on the viewpoint coverage range of the input image.

[0066] In a second aspect, an optimization device for sparse-view three-dimensional Gaussian splashing is provided, and the device includes:

[0067] A sparse-view image acquisition module is configured to acquire sparse-view images, wherein the sparse-view images refer to a very limited number (usually 2-8) of target scene images acquired in an actual application scenario.

[0068] A sparse-view image processing module is configured to generate a geometrically consistent depth map based on a multi-view stereo vision model of deep learning, convert the depth map into a point cloud, and fuse the point cloud to obtain a dense point cloud.

[0069] A three-dimensional Gaussian field construction module is configured to sample the dense point cloud by using a voxel-guided farthest point sampling to obtain an initialized point cloud, and construct a three-dimensional Gaussian field based on the initialized point cloud.

[0070] A geometric regularization module is configured to render the three-dimensional Gaussian field by using an enhanced geometry renderer to obtain a rendered depth and a rendered normal, construct a multi-level geometric regularization loss function, and optimize the three-dimensional Gaussian field, wherein the multi-level geometric regularization loss function includes an MVS depth loss, a monocular depth loss, a monocular normal loss, and a depth-normal consistency loss.

[0071] An optimization adjustment module is configured to perform optimization adjustment on the three-dimensional Gaussian field based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy, to obtain an optimized three-dimensional Gaussian field.

[0072] The technical scheme of the embodiment of the present application has the following beneficial effects:

[0073] The present application solves the problems of initialization failure, insufficient geometric supervision and primitive out of control of 3D Gaussian splatting under sparse view by robust initialization, multi-level geometric regularization and dynamic Gaussian primitive constraint, improves the initial point cloud integrity, geometric reconstruction accuracy and rendering quality, reduces the abnormal primitives and artifacts, enhances the generalization ability of sparse input scene, and provides efficient and reliable technical support for virtual reality, cultural relic protection and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is an optimization method flowchart of sparse view three-dimensional Gaussian splatting according to an exemplary embodiment;

[0075] Figure 2 is a structural schematic diagram of an optimization device of sparse view three-dimensional Gaussian splatting according to an exemplary embodiment;

[0076] Figure 3 is a 3D Gaussian splatting optimization flowchart;

[0077] Figure 4 is a 3D Gaussian splatting adaptive density control strategy diagram;

[0078] Figure 5 is a robust initialization flowchart based on Voxel-FPS;

[0079] Figure 6 is a multi-level geometric regularization schematic diagram;

[0080] Figure 7 is a shape-scale constraint schematic diagram;

[0081] Figure 8 is an opacity constraint schematic diagram. DETAILED DESCRIPTION

[0082] To make the technical features of the present application clearer, the present application will be described in detail below with reference to the specific embodiments and the accompanying drawings.

[0083] As shown in Figure 1 , the optimization method of sparse view three-dimensional Gaussian splatting provided by the embodiment of the present application includes the following steps:

[0084] Step S1, sparse view images are collected, which are extremely limited (usually 2-8) target scene images collected in an actual application scenario;

[0085] Step S2, a multi-view stereo vision model based on deep learning is used to generate a geometrically consistent depth map from the sparse view images, the depth map is converted into a point cloud, and a dense point cloud is obtained by fusion;

[0086] Step S3, the dense point cloud is sampled by using a voxel-guided farthest point sampling to obtain an initialized point cloud, and a three-dimensional Gaussian field is constructed based on the initialized point cloud;

[0087] Step S4, the three-dimensional Gaussian field is rendered by an enhanced geometry renderer to obtain rendered depth and rendered normal; a multi-level geometric regularization loss function is constructed, and the three-dimensional Gaussian field is optimized, the multi-level geometric regularization loss function includes MVS depth loss, monocular depth loss, monocular normal loss, and depth-normal consistency loss;

[0088] Step S5, the three-dimensional Gaussian field is optimized and adjusted based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field.

[0089] As a possible implementation manner of the embodiment, the step S1 includes the following steps:

[0090] Step S11, the shooting position and angle of the shooting device are adjusted according to the actual shootable range of the scene, the core structure of the scene (such as cultural relic details, rescue key areas, and subject scenes of user attention) is preferentially covered, but a “dense coverage” layout is not required, to simulate a passive sparse scene with limited view angle; the shooting device uses a conventional shooting device such as a normal camera, a smart phone, or a video camera, and basic shooting parameters (focal length, aperture, exposure time, etc.) are configured to adapt to the lighting conditions of the target scene;

[0091] Step S12, 1-2 images are collected at each selected position under a random view angle (to reduce the shooting operation burden and be suitable for personal users to quickly record), and there is no need to strictly require the overlap ratio of adjacent view images (because the overlap may be limited by shooting conditions in an actual scene and cannot be guaranteed), and the collected images are stored in a conventional resolution as a general format such as JPEG or PNG;

[0092] Step S13, all 2-8 collected images are directly used as subsequent processing inputs (because only these images may be available in an actual scene, and there is no need to perform screening and elimination).

[0093] As a possible implementation manner of the embodiment, the step S2 includes the following steps:

[0094] In step S21, a multi-view stereo vision model based on deep learning is used to process the sparse view images to generate a depth map with geometric consistency. The multi-view stereo vision model based on deep learning is a pre-trained MVS model, and the generated depth map has geometric consistency ensured by multi-view geometric constraints. The geometric consistency is manifested as a spatial position matching error of the multi-view depth map in the overlapping area being lower than a preset threshold.

[0095] In step S22, the generated depth map is converted into a corresponding point cloud, and the point cloud is fused to obtain a dense point cloud covering a complete scene.

[0096] As a possible implementation manner of the embodiment, in step S22, the process of converting the generated depth map into a corresponding point cloud includes: converting the depth value of each pixel in the depth map into a three-dimensional space coordinate based on the intrinsic and extrinsic parameters of the camera, to form a point cloud of the corresponding view. The intrinsic parameters of the camera include focal length and principal point coordinates, and the extrinsic parameters include a rotation matrix and a translation vector, which are used to convert the depth value in the pixel coordinate system into a three-dimensional coordinate in the world coordinate system. The fusion processing includes de-duplication, registration and splicing operations on the multi-view generated point cloud. The de-duplication is to remove redundant points with spatial position coincidence or distance less than a preset threshold. The registration is to align the multi-view point clouds to the same coordinate system through spatial transformation. The splicing is to integrate the registered point clouds to form a dense point cloud covering a complete scene.

[0097] As a possible implementation manner of the embodiment, step S3 includes the following steps:

[0098] In step S31, a voxel-guided farthest point sampling method is used to sample the dense point cloud to obtain an initialized point cloud.

[0099] In step S32, a three-dimensional Gaussian field is constructed based on the initialized point cloud. The three-dimensional Gaussian field is composed of a plurality of Gaussian primitives, and the parameters of each Gaussian primitive include a center position, a scaling factor, a rotation quaternion, an opacity value and a color feature.

[0100] As a possible implementation manner of the embodiment, in step S31, the voxel-guided farthest point sampling method includes:

[0101] In step S311, the scene where the dense point cloud is located is voxelized to divide a plurality of voxel units.

[0102] In step S312, the point density in each voxel unit is calculated. The point density is the ratio of the number of point clouds contained in the voxel unit to the volume of the voxel unit.

[0103] Step S313, assigning an inverse sampling rate to each voxel unit according to the point density, wherein a voxel unit with high point density is assigned a lower sampling rate, and a voxel unit with low point density is assigned a higher sampling rate.

[0104] Step S314, performing a furthest point sampling algorithm in each voxel unit according to the assigned inverse sampling rate to obtain the initial point cloud.

[0105] As a possible implementation manner of the embodiment, the step S4 comprises the following steps:

[0106] Step S41, generating a rendered depth and a rendered normal by rendering the three-dimensional Gaussian field through an enhanced geometry renderer;

[0107] Step S42, constructing a multi-level geometry regularization loss function;

[0108] Step S43, optimizing parameters of the three-dimensional Gaussian field based on the multi-level geometry regularization loss function to improve the geometric accuracy and surface smoothness of scene reconstruction.

[0109] As a possible implementation manner of the embodiment, in step S41, the process of the enhanced geometry renderer calculating the rendered depth comprises:

[0110] Projecting the center of the Gaussian primitive to the camera coordinate system to obtain a projected center coordinate (x', y', z');

[0111] Calculating a depth value at the pixel based on the projected center coordinate and the relative pixel position:

[0112] ,

[0113] wherein, is the depth of the projected center of the i-th Gaussian primitive, Δx=x- , Δy=y- are deviations of the pixel position (x, y) and the projected center , and p is a vector determined based on Gaussian primitive parameters .

[0114] As a possible implementation manner of the embodiment, in step S41, the process of the enhanced geometry renderer calculating the rendered normal comprises:

[0115] Converting the normal vector from the ray space to the camera space:

[0116] ,

[0117] wherein, J represents a local affine matrix, depth of the projection center of the Gaussian cell, depth of the center of the Gaussian cell, p is a vector determined based on parameters of the Gaussian cell.

[0118] As a possible implementation manner of the embodiment, in step S42, the MVS depth loss is an L1 loss between the rendering depth and the depth generated by the MVS model, and the formula is as follows:

[0119] ,

[0120] wherein, is the rendering depth, is the MVS depth.

[0121] As a possible implementation manner of the embodiment, in step S42, the monocular depth loss is a Pearson correlation loss between the rendering depth and the monocular predicted depth, and the formula is as follows:

[0122] ,

[0123] wherein monocular depth estimated by the prior estimator, wherein Cov is a covariance function, and Var is a variance function; the monocular depth loss is realized by a global-local hierarchical strategy, including calculating the correlation loss for the global depth map and the randomly sampled local depth blocks respectively.

[0124] As a possible implementation manner of the embodiment, in step S42, the monocular normal loss is an L1 loss between the rendering normal and the monocular predicted normal:

[0125] ,

[0126] wherein is the Gaussian rendering normal, is the monocular normal.

[0127] As a possible implementation manner of the embodiment, in step S42, the depth-normal consistency loss is a cosine similarity loss between the rendering normal and the pseudo normal calculated based on the rendering depth:

[0128] ,

[0129] wherein, is a pseudo normal map, which is calculated by finite difference based on the rendering depth:

[0130] ,

[0131] wherein, , are gradients of the 3D point in x and y directions respectively.

[0132] As a possible implementation manner of the embodiment, in step S42, the composite form of the multi-level geometric regularization loss function is:

[0133]

[0134] wherein, w1, w2, w3, w4 and w5 respectively represent the weight corresponding to each loss, and .

[0135] As a possible implementation manner of the embodiment, the weight coefficient is in the range of: .

[0136] As a possible implementation manner of the embodiment, in step S42, the parameters of the three-dimensional Gaussian field include the center position, the scaling factor, the rotation quaternion, the opacity value and the color feature of the Gaussian primitive, and the parameter optimization is realized by minimizing the multi-level geometric regularization loss function through the back propagation algorithm.

[0137] As a possible implementation manner of the embodiment, the step S5 comprises the following steps:

[0138] Step S51, based on the shape-size constraint criterion, identify and remove the needle-shaped abnormal Gaussian primitive in the three-dimensional Gaussian field;

[0139] Step S52, adopt a two-stage adaptive opacity constraint strategy to adjust the opacity value of the Gaussian primitive in the three-dimensional Gaussian field, and obtain the optimized three-dimensional Gaussian field.

[0140] As a possible implementation manner of the embodiment, the shape-size constraint criterion is that if the ratio of the longest axis to the second longest axis of the Gaussian primitive is greater than a preset ratio threshold and the length of the longest axis is greater than a preset size threshold, the Gaussian primitive is removed; the two-stage adaptive opacity constraint strategy comprises a pruning stage and a fine-tuning stage, the pruning stage gradually increases the opacity threshold at a fixed interval to remove the translucent Gaussian primitive, and the fine-tuning stage adaptively adjusts the final opacity threshold based on the viewpoint coverage range of the input image.

[0141] As shown in Figure 2 , the embodiment of the application provides an optimization device of sparse-view three-dimensional Gaussian splatter, which comprises:

[0142] A sparse-view image acquisition module is configured to acquire sparse-view images, wherein the sparse-view images are target scene images with extremely limited quantity (usually 2-8) acquired in an actual application scenario.

[0143] ​​The sparse view image processing module is configured to generate a geometrically consistent depth map from sparse view images based on a deep learning-based multi-view stereo vision model, convert the depth map into a point cloud, and fuse the point cloud to obtain a dense point cloud.

[0144] The three-dimensional Gaussian field construction module is configured to sample the dense point cloud to obtain an initialized point cloud by using voxel-guided farthest point sampling, and construct a three-dimensional Gaussian field based on the initialized point cloud.

[0145] The geometric regularization module is configured to render the three-dimensional Gaussian field by using an enhanced geometry renderer to obtain rendered depth and rendered normal, construct a multi-level geometric regularization loss function, and optimize the three-dimensional Gaussian field, wherein the multi-level geometric regularization loss function comprises an MVS depth loss, a monocular depth loss, a monocular normal loss, and a depth-normal consistency loss.

[0146] The optimization adjustment module is configured to optimize and adjust the three-dimensional Gaussian field based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field.

[0147] The technical scheme of the present application can be implemented in three modules: in order to achieve robust initialization, the present application proposes a dense point cloud initialization strategy. In the optimization process, multi-level geometric regularization is introduced to achieve more accurate geometric supervision. Finally, targeted attribute constraints (shape scale and opacity) are realized on the Gaussian primitives to prevent overfitting in the sparse view scene.

[0148] The specific implementation of the present application mainly involves the following key technologies.

[0149] I. Basic knowledge of 3D Gaussian splatting technology.

[0150] 3D Gaussian splatting uses a set of 3D Gaussian functions to represent a 3D scene. Each Gaussian primitive can be described by a center position , a scale factor , and a rotation quaternion . The base function of the i-th Gaussian primitive is defined as , and the specific definition formula is as follows:

[0151] (1),

[0152] where the covariance matrix Σ is calculated from the scale and the rotation quaternion . In order to realize rendering, the Gaussian primitive also retains an opacity value and a K-dimensional color feature . In summary, the complete parameter definition of the i-th Gaussian is When rendering images from a given viewpoint, 3D Gaussian Splash uses a differentiable The rendering method of the hybrid point is as follows:

[0153] (2),

[0154] where is the color computed from the feature is obtained by evaluating a 2D Gaussian with covariance and multiplying by the opacity . The 2D covariance matrix is computed by which projects the 3D covariance into the camera coordinate system. denotes the Jacobian matrix of the projective transformation affine approximation, is the view transformation matrix.

[0155] 3D Gaussian Splash uses the point cloud generated from COLMAP or other SfM methods to initialize the Gaussian function. During the optimization process, the parameters of all Gaussian primitives are optimized by photometric supervision, while the density and distribution of Gaussian primitives are controlled by an adaptive densification strategy. The optimization process of 3D Gaussian Splash is shown in Figure 3 .

[0156] 3D Gaussian Radiance Field optimization specifically depends on the photometric error and structural similarity error between the rendered image and the real photo. Its loss function can be expressed as:

[0157] (3),

[0158] where, is a scale coefficient, usually set to 0.2. is the photometric error between the rendered image and the real photo; is the structural similarity error. The structural similarity error is used to detect the similarity between the reconstructed image and the original image, which measures the similarity of the image from three dimensions of brightness, contrast and structure, and can effectively express the structural information of the three-dimensional scene.

[0159] ​Further, since the initial 3D Gaussian radiance field is optimized based on a sparse point cloud model, the sparse point cloud cannot fully represent the scene, resulting in areas of under-reconstruction or over-reconstruction. Therefore, the size and number of 3D Gaussian radiance fields need to be adjusted during the optimization process. Specifically, 3D Gaussian splatting triggers Gaussian cloning in under-reconstructed areas (such as complex geometric surfaces or high-frequency detail areas) by analyzing gradient information and geometric complexity, and enhances local details by copying and fine-tuning Gaussian parameters; while in over-reconstructed areas (such as smooth surfaces or low-variation areas), Gaussian splitting is used to split a single Gaussian into multiple small-scale Gaussians to alleviate the risk of overfitting. The adaptive density control strategy of 3D Gaussian splatting is shown in Figure 4

[0160] II. Robust Initialization Based on Voxel-FPS

[0161] In traditional application scenarios, the initialization process of 3D GS consists of two core stages. First, a sparse point cloud is generated using SfM technology. Then, an adaptive densification strategy is used to dynamically adjust the Gaussian distribution and density. This strategy monitors the gradient changes of the Gaussian position in the view space, and triggers local point cloud densification when the average gradient exceeds a predefined threshold, thereby achieving progressive reconstruction of the scene geometry. However, under sparse input conditions, this technology faces challenges: first, the initial point cloud generated by the SfM method is extremely limited, and in extreme cases it may not be able to reconstruct any point cloud. Second, the adaptive encryption mechanism has defects in sparse scenarios, and when the initial point coverage is insufficient or completely missing, it cannot generate new Gaussian distributions at the correct location.

[0162] 2.1 Learning-based MVS enhanced initialization:

[0163] Compared to sparse point clouds, dense point clouds provide more complete initial information and have a higher coverage of the scene. Some existing research has explored using MVS methods to generate dense point clouds for initialization under sparse conditions and has shown promising results.

[0164] Inspired by these works, to achieve more stable initialization under sparse view conditions, the present invention proposes a robust initialization method, as shown in Figure 5

[0165] 2.2 Voxel-guided farthest point sampling:

[0166] ​​While MVS-generated point clouds are denser, the resulting compact point distribution often introduces redundancy for Gaussian field initialization. Excessive initial points increase memory and computational cost, and can even limit the representational capacity of the Gaussian field. Previous work usually applies random down-sampling to obtain manageable point numbers, but this approach often leads to the loss of critical structural details. Specifically, areas with low image overlap produce sparse initial points, where insufficient supervision further limits effective scene reconstruction through density-enhancing strategies. Therefore, points in these areas are crucial during initialization. However, random point cloud down-sampling can fail to preserve these critical areas, leading to the loss of geometric details and a decrease in initialization quality.

[0167] Accordingly, the present invention introduces a voxel-guided farthest point sampling method. This method employs a three-step strategy to optimize point cloud sampling: first, the scene is voxelized and the point density within each voxel is calculated. Second, based on the density, an inverse sampling rate is adaptively assigned to each voxel. Dense areas employ a lower sampling rate, while sparse areas employ a higher sampling rate. Finally, a farthest point sampling algorithm is applied within each voxel to ensure uniform spatial distribution of the sampled points, effectively preserving the overall structure of the scene. This hierarchical sampling method ensures both the representational accuracy of weakly supervised areas and the overall sampling efficiency.

[0168] III. Multi-level geometry regularization.

[0169] The multi-level geometry regularization framework of the present invention contains three parts. First, the present invention utilizes the highly geometrically consistent regions in MVS depth as basic supervision to preserve the basic geometric structure of the scene. Second, the present invention introduces depth and normal priors, providing richer geometric information to further optimize scene geometry. Finally, the present invention implements depth-normal consistency to reduce the ambiguity of supervision and strengthen geometry regularization. Prior to this process, the present invention adds a higher-level renderer to achieve more accurate depth and normal rendering.

[0170] 3.1 Enhanced geometry renderer.

[0171] Unlike traditional 3D representations, Gaussian fields decompose the scene into 3D Gaussians, which inherently lack explicit depth and surface normal attributes. Previous work usually employs alpha blending rendering in 3D GS for depth rasterization, where the depth value is estimated by aggregating the contributions of Gaussians along each ray in sorted order. This can be represented as:

[0172] (4),

[0173] where is the depth of the th Gaussian distribution, The same as the value in equation (2). However, this estimation method only considers the centroid position of each Gaussian distribution, while ignoring their volume range. This simplification introduces errors. In addition, to approximate the surface normal, most existing methods make the 3D Gaussian distribution present a flat disc shape by constraining one scaling axis to be much smaller than the other two axes. Then the direction of this minimized axis is taken as the normal vector. While this method is effective for normal extraction, this geometric regularization essentially limits the shape diversity of Gaussian distributions, thereby limiting their ability to represent complex geometric shapes. In summary, existing depth and normal rendering methods have inherent limitations that limit the effectiveness of geometry supervision.

[0174] To enhance the supervision of depth and normal in Gaussian fields, the present invention employs a renderer in RaDe-GS. This renderer incorporates Gaussian volume influence during rasterization, resulting in more accurate rendered depth and normal values. Specifically, the center of a Gaussian G is first projected into the camera coordinate system, denoted as After obtaining the projected center coordinates (x', y', z') of each Gaussian, the depth value at each pixel is calculated as follows:

[0175] (5),

[0176] where, is the depth of the projected center of the i-th Gaussian cell, Δx = x , Δy = y , and Δz = z are the deviations of the pixel position (x, y) and the projected center , p is the vector determined based on the Gaussian cell parameters .

[0177] The vector p is determined by the Gaussian parameters . The normal direction of the projected Gaussian is consistent with the surface plane. To calculate the normal map, the present invention converts the normal vector from "ray space" to "camera space" as follows:

[0178] (6),

[0179] where J represents the local affine matrix, is the depth of the projected center of the Gaussian cell, is the depth of the Gaussian cell center, and p is the vector determined based on the Gaussian cell parameters.

[0180] 3.2 MVS depth geometry regularization.

[0181] Deep supervision is proven to be crucial for sparse view scene reconstruction, and existing works have demonstrated its effectiveness in constraining the Gaussian field. Specifically, the depth prior regularization is able to correct the geometric shape of the Gaussian distribution and mitigate the overfitting risk inherent in sparse constraints. It is essential to achieve robust novel view synthesis under limited observation conditions.

[0182] In the geometric regularization framework of the present application, the present application first adopts multi-view Figure 1 depth as the base supervision to maintain scene geometry consistency and prevent overfitting. Specifically, the MVS depth information is generated in the MVS-based point cloud initialization process. Although this depth has been converted into an initial point cloud and does not provide additional geometric information, its multi-view geometric consistency can reduce overfitting caused by insufficient supervision. Given the scale consistency between MVS depth and rendered depth, the present application directly applies L1 regularization:

[0183] (7),

[0184] wherein denotes the Gaussian rendered depth calculated according to formula (5), denotes the MVS depth.

[0185] 3.3 Monocular geometric prior regularization.

[0186] Monocular depth regularization: Since MVS depth regularization only provides supervision for areas with high geometric consistency, it cannot cover the entire scene. To achieve complete coverage, the present application introduces a monocular depth prior to further guide the Gaussian distribution to converge to a more accurate geometric position.

[0187] In the optimization process, considering the scale inconsistency between monocular depth prediction and Gaussian field depth, direct application of L1 loss will destroy the scene geometry. The present application uses Pearson correlation similarity for soft depth supervision. The Pearson correlation coefficient is a basic statistical measure used to quantify the linear correlation between two data sets, essentially evaluating the similarity between distributions X and Y. The formula for depth Pearson supervision is:

[0188] (8),

[0189] wherein denotes the monocular depth estimated by the prior estimator. This supervision strategy effectively avoids depth scale misplacement by taking advantage of inter-image similarity. To enhance the comprehensiveness of supervision, the present application also measures the similarity of randomly sampled local depth blocks, which can reduce local errors. Through a hierarchical global-local strategy, the method of the present application achieves more robust depth supervision effect.

[0190] Monocular normal regularization: Normal supervision is also crucial for scene reconstruction, which shows significant effect in improving the accuracy of geometry alignment and the smoothness of reconstructed scene surface. Similar to depth, the invention also adopts monocular predicted normals to supervise the scene. Normal information encourages Gaussian ellipsoid deformation to better align with the real surface geometry, thus enhancing the smoothness and geometric details of the scene. Considering the relative smoothness of predicted normals, the invention directly adopts L1 loss during the training process:

[0191] (9),

[0192] where represents the Gaussian rendered normal calculated according to formula (6), represents the monocular normal estimated by the prior estimator.

[0193] 3.4 Depth-normal consistency regularization.

[0194] Finally, the invention enhances the supervision effect through joint depth-normal consistency regularization. Existing research shows that independent optimization of depth and normal cannot obtain the best results and may introduce ambiguity (for example, the normal does not match the direction of depth gradient). For this purpose, the invention introduces a depth-normal consistency loss, which enhances the surface smoothness and geometric consistency by enforcing the intrinsic geometric relationship between depth and normal. Specifically, the invention calculates a pseudo-normal map from the rendered depth map and aligns it with the Gaussian rendered normal map. Due to the inherent noise in the rendered depth, the pseudo-normal map exhibits non-smoothness. Therefore, the invention adopts a cosine similarity loss during the training process, whose formula is as follows:

[0195] (10),

[0196] where represents the pseudo-normal map. Specifically, the invention converts each pixel and its depth value in the depth map into a 3D point, and then calculates the normal using adjacent depth points through finite difference. The formula is as follows:

[0197] (11),

[0198] Overall, the geometry regularization framework of the invention integrates four components: MVS depth regularization, monocular depth regularization, monocular normal regularization, and depth-normal consistency regularization, as shown in Figure 6 The composite loss function is formally defined as follows:

[0199] (12),

[0200] where represent the weights corresponding to each loss, respectively.

[0201] IV. Gaussian cell constraints.

[0202] This invention will discuss the impact of Gaussian primitives under sparse input conditions and re-consider the constraint strategy of Gaussian primitive's scale parameter s and opacity value a to further reduce the difficulty of optimization under insufficient supervision.

[0203] 4.1 Shape-scale constraint:

[0204] The shape and scale properties of Gaussian ellipsoid have significant impact on sparse view synthesis. In the original Gaussian framework, the scale constraint of Gaussian is enforced by directly pruning those Gaussian ellipsoids whose screen-space projection size or world-space dimension exceeds a pre-defined threshold after a certain number of iterations. However, this strategy is unreliable in sparse view scenarios.

[0205] Specifically, the initial size of a Gaussian is determined by the average distance from its three nearest neighbors. In areas with low image overlap, sparse point clouds often result in oversized Gaussian ellipsoids. Moreover, due to the lack of sufficient supervision in these areas, the densification strategy cannot effectively segment these oversized Gaussians even after multiple iterations. Therefore, in low overlap areas such as scene edges or backgrounds, oversized Gaussians persist, but this invention finds that these Gaussians are crucial for maintaining the integrity of scene reconstruction. Directly pruning these oversized Gaussians would cause irreversible damage to the scene, manifesting as holes or missing areas. More importantly, the densification strategy cannot generate new Gaussians in the correct location to compensate for the removed Gaussians. Therefore, most previous work abandoned this constraint strategy, choosing not to impose a direct limit on Gaussian size.

[0206] However, this approach still has serious flaws. First, abandoning the scale constraint inevitably introduces redundancy. More importantly, the lack of supervision in sparse view scenarios can lead to severe overfitting, causing certain Gaussian distributions to exhibit extreme elongation along specific axes, forming needle-like ellipsoids. These elongated needle-like ellipsoids can disrupt the geometric structure and severely degrade the rendering quality of new views.

[0207] To address this issue, this invention proposes a targeted pruning strategy that goes beyond the simple longest axis threshold method. This invention additionally evaluates the ratio between the longest axis and the second longest axis to identify needle-like Gaussian distributions. This criterion is expressed as follows:

[0208] (13),

[0209] where and denote the length of the longest axis and the second longest axis of the Gaussian ellipsoid, respectively, T represents the threshold value of the size of the super-gaussian, and T represents the threshold value of the ratio for identifying the needle-like gaussian.

[0210] By this strategy, the present application can accurately eliminate these abnormal gaussians caused by insufficient constraints while maintaining the integrity of the scene, such as Figure 7 During the training process, when the scene structure is relatively stable and overfitting occurs, this strategy will be implemented after a certain number of iterations.

[0211] 4.2 Opacity constraint:

[0212] The traditional Gaussian method periodically resets all Gaussian opacities to a minimum value during the training process to enhance domain learning and avoid falling into local optima. In addition, a minimum opacity threshold is enforced during training to prune gaussians with insufficient opacity. This method effectively removes artifacts and eliminates false or unnecessary gaussians.

[0213] However, in sparse view scenes, this strategy shows limitations. Due to insufficient training images and minimal changes in shooting angles (e.g., capturing a small number of images from similar directions), the limited availability of cross-view supervision causes certain gaussians to fall into local optima during optimization, preventing them from falling below a predefined threshold. These semi-transparent gaussians falsely persist near real surfaces, reducing rendering quality by introducing ghosting artifacts and generating blurred depth maps, thereby disrupting geometric regularization.

[0214] To solve this problem, the present application proposes a two-stage adaptive opacity constraint strategy. First, in the pruning stage, the present application gradually increases the opacity threshold at fixed intervals during training iterations. Semi-transparent gaussian distributions are systematically removed. Second, in the fine-tuning stage, the present application adaptively adjusts the final opacity threshold based on the viewpoint coverage of the input images. The scene is optimized stably at this threshold, maintaining its integrity. The update rule is expressed as follows:

[0215] (14),

[0216] where represents the opacity threshold, represents the opacity value increased after each iteration interval.

[0217] By adopting this strategy during optimization, the present application not only eliminates artifacts and floating debris in the scene, but also removes redundant semi-transparent gaussian distributions near the surface, thereby promoting "surface solidification" and enhancing geometric regularization, as shown in Figure 8 .

[0218] Five. Training loss.

[0219] The final loss function consists of two parts: photometric loss commonly used in 3D GS and geometric regularization loss The present application defines the overall loss function:

[0220] (15),

[0221] wherein consists of one and one structural similarity loss , denoted as:

[0222] (16),

[0223] wherein denotes the weight. Following the original Gaussian formula, the is set to 0.2.

[0224] The present application aims at the core pain points of sparse-view 3D Gaussian splatting technology, and realizes multi-dimensional technical breakthrough through a three-stage optimization architecture of "robust initialization → geometric regularization → Gaussian primitive constraint", as follows:

[0225] 1. Solving the initialization failure problem, improving the robustness and integrity of the initial point cloud:

[0226] The prior art relies on sparse point clouds or random point clouds generated by SfM for initialization, which is prone to problems such as point cloud missing, excessive noise, or loss of key structures under sparse input. The present application realizes breakthrough through the following improvements:

[0227] A geometry-consistent depth map is generated using a deep learning-based MVS model, and a dense point cloud is obtained by fusion, solving the problem of extremely sparse point clouds under sparse views by SfM;

[0228] Combined with voxel-guided farthest point sampling, the sampling weight is dynamically allocated according to the voxel density, which preserves the structural integrity of weakly supervised areas (areas with low image overlap) while reducing redundancy, ensuring that the initialized point cloud covers the complete scene and avoids the introduction of noise and invalid points.

[0229] The present application can still generate robust and high-precision initial point clouds under 2-8 sparse input images, laying a reliable foundation for subsequent Gaussian field optimization, overcoming the defects of traditional initialization methods such as "incomplete scene coverage" and "noise sensitivity".

[0230] 2. Strengthening the completeness of geometric supervision, significantly improving the scene reconstruction accuracy and rendering quality:

[0231] The prior art only relies on monocular depth or photometric loss constraints, resulting in sparse view geometry blur, surface fragmentation and many artifacts. The present application realizes optimization through a multi-level geometric regularization framework:

[0232] The multiplex MVS high-confidence depth provides basic geometric constraints, avoids scale differences by combining the global-local Pearson correlation loss of monocular depth, introduces monocular normal constraints to enhance surface smoothness, and finally enforces geometric internal correlation through depth-normal consistency loss;

[0233] The multi-level loss works together to form comprehensive geometric supervision, solving the problem of "insufficient information" and "weak constraints" of single supervision.

[0234] The present application significantly improves the scene geometry reconstruction accuracy, greatly reduces the deviation between rendered depth and real geometry, significantly reduces surface fragmentation, floating objects and other artifacts, and the geometry consistency and detail restoration of new view synthesis are significantly better than existing methods.

[0235] 3. Solve the problem of Gauss primitive out of control, reduce the abnormal primitive and redundant artifacts:

[0236] The constraint strategy of the prior art for Gauss primitives is invalid under sparse view: either directly clipping large-size Gauss to cause scene structure collapse, or indiscriminately retaining all Gauss to cause needle-shaped deformity and semi-transparent artifact residues. The present application realizes breakthrough through a dynamic constraint mechanism:

[0237] An innovative shape-scale constraint criterion is used to accurately remove needle-shaped deformed Gauss by setting a threshold for the ratio of the longest axis to the second longest axis, while retaining large-size Gauss of key backgrounds to avoid scene structure damage;

[0238] Two-stage adaptive opacity constraints are used: in the pruning stage, the threshold is gradually increased to remove semi-transparent artifacts, and in the fine-tuning stage, the threshold is adaptively adjusted according to the viewpoint coverage to reduce redundant primitives and reduce memory occupancy.

[0239] The present application effectively eliminates needle-shaped deformed Gauss and semi-transparent floating artifacts, the Gauss primitive distribution is more consistent with the real scene structure, the clarity and realism of the rendered image are significantly improved, and the redundant calculation is reduced, optimizing the memory efficiency.

[0240] 4. Improve the generalization ability under sparse view, expand the application scene of the technology:

[0241] The prior art is prone to overfitting to limited view under sparse input, resulting in poor generalization ability of new view synthesis. The present application, through the above three-stage optimization, can still robustly reconstruct scene geometry and generate high-quality new view images with only 2-8 input images, breaking through the dependence of traditional methods on "dense input view".

[0242] The application provides a reliable three-dimensional reconstruction and new view synthesis scheme for scenes such as virtual reality, autonomous driving and cultural relic protection, which have high data acquisition cost or limited operation, and solves the core bottleneck of 'difficult to obtain dense view data' in practical applications.

[0243] The application systematically solves the three core problems of initialization robustness, geometric supervision completeness and Gaussian primitive control of sparse view 3D Gaussian splashing, and provides an efficient and high-precision technical scheme for three-dimensional reconstruction and new view synthesis of sparse input scenes, which has important practical value and industrial application prospect.

[0244] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the protection scope of the claims of the present application.

Claims

1. A method of optimizing sparse view angle three-dimensional Gaussian splash, characterized in that, Comprising the following steps: Step S1, sparse view images are collected, which are extremely limited in number and collected in actual application scenarios; Step S2, a multi-view stereo vision model based on deep learning is used to generate a geometrically consistent depth map from the sparse view images, and the depth map is converted into a point cloud and fused to obtain a dense point cloud; Step S3, the dense point cloud is sampled using a voxel-guided farthest point sampling method to obtain an initialized point cloud, and a three-dimensional Gaussian field is constructed based on the initialized point cloud; Step S4, the three-dimensional Gaussian field is rendered by an enhanced geometry renderer to obtain rendered depth and rendered normal; a multi-level geometric regularization loss function is constructed, and the three-dimensional Gaussian field is optimized, wherein the multi-level geometric regularization loss function includes MVS depth loss, monocular depth loss, monocular normal loss, and depth-normal consistency loss; Step S5, the three-dimensional Gaussian field is optimized and adjusted based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field; The step S3 comprises the following steps: Step S31, the dense point cloud is sampled using a voxel-guided farthest point sampling method to obtain an initialized point cloud; Step S32, a three-dimensional Gaussian field is constructed based on the initialized point cloud, wherein the three-dimensional Gaussian field is composed of multiple Gaussian primitives, and the parameters of each Gaussian primitive include a center position, a scaling factor, a rotation quaternion, an opacity value, and a color feature; In step S31, the voxel-guided farthest point sampling method comprises: Step S311, the scene where the dense point cloud is located is voxelized to divide multiple voxel units; Step S312, the point density in each voxel unit is calculated, which is the ratio of the number of point clouds contained in the voxel unit to the volume of the voxel unit; Step S313, each voxel unit is assigned an inverse sampling rate according to the point density, wherein the voxel unit with high point density is assigned a lower sampling rate, and the voxel unit with low point density is assigned a higher sampling rate; Step S314, in each voxel unit, the farthest point sampling algorithm is executed according to the assigned inverse sampling rate to obtain the initialized point cloud.

2. The method of claim 1, wherein, The step S1 comprises the following steps: Step S11, the shooting position and angle of the shooting device are adjusted according to the actual shootable range of the scene, and the core structure of the scene is preferentially covered without pursuing a "dense coverage" layout to simulate a passive sparse scene with limited view angles; Step S12, 1-2 images are collected at each selected position under random view angles without forcibly requiring an overlap ratio of adjacent view images, and the collected images are stored in a general format at a regular resolution; Step S13, all 2-8 collected images are directly used as input for subsequent processing.

3. The method of claim 1, wherein, The step S2 comprises the following steps: In step S21, a multi-view stereo vision model based on deep learning is used to process the sparse view images to generate a depth map with geometric consistency. The multi-view stereo vision model based on deep learning is a pre-trained MVS model, and the generated depth map has geometric consistency ensured by multi-view geometric constraints. The geometric consistency is that the spatial position matching error of the multi-view depth map in the overlapping area is less than a preset threshold. In step S22, the generated depth map is converted into a corresponding point cloud, and the point cloud is fused to obtain a dense point cloud covering the complete scene.

4. The method of optimizing sparse view angle three-dimensional Gaussian splash of claim 3, wherein, In step S22, the process of converting the generated depth map into a corresponding point cloud includes: converting the depth value of each pixel in the depth map into a three-dimensional space coordinate based on the intrinsic and extrinsic parameters of the camera to form a point cloud corresponding to the view. The intrinsic parameters of the camera include focal length and principal point coordinates, and the extrinsic parameters include a rotation matrix and a translation vector, which are used to convert the depth value in the pixel coordinate system into a three-dimensional coordinate in the world coordinate system. The fusion process includes de-duplication, registration and splicing operations on the multi-view generated point cloud. The de-duplication is to remove redundant points with spatial position coincidence or distance less than a preset threshold. The registration is to align the multi-view point clouds to the same coordinate system through spatial transformation. The splicing is to integrate the registered point clouds to form a dense point cloud covering the complete scene.

5. The method of claim 1, wherein, The step S4 includes the following steps: In step S41, a three-dimensional Gaussian field is rendered by an enhanced geometric renderer to generate a rendered depth and a rendered normal; In step S42, a multi-level geometric regularization loss function is constructed; In step S43, the parameters of the three-dimensional Gaussian field are optimized based on the multi-level geometric regularization loss function to improve the geometric accuracy and surface smoothness of scene reconstruction.

6. The method of optimizing sparse view three-dimensional Gaussian spill of any of claims 1-5, wherein, The step S5 includes the following steps: In step S51, a shape-scale constraint criterion is used to identify and remove needle-shaped abnormal Gaussian elements in the three-dimensional Gaussian field; In step S52, a two-stage adaptive opacity constraint strategy is used to adjust the opacity values of the Gaussian elements in the three-dimensional Gaussian field to obtain an optimized three-dimensional Gaussian field.

7. The method of Claim 1-5, wherein, The shape-scale constraint criterion is that if the ratio of the longest axis to the second longest axis of a Gaussian element is greater than a preset ratio threshold and the length of the longest axis is greater than a preset size threshold, the Gaussian element is removed. The two-stage adaptive opacity constraint strategy includes a pruning stage and a fine-tuning stage. In the pruning stage, the opacity threshold is gradually increased at a fixed interval to remove semi-transparent Gaussian elements. In the fine-tuning stage, the final opacity threshold is adaptively adjusted based on the viewpoint coverage range of the input image.

8. An apparatus for optimizing sparse view angle three-dimensional Gaussian splash, characterized in that, It includes: A sparse view image acquisition module is configured to acquire sparse view images. The sparse view images refer to a small number of target scene images collected in actual application scenarios. A sparse view image processing module is configured to generate a geometric consistent depth map from the sparse view images based on a multi-view stereo vision model based on deep learning, convert the depth map into a point cloud, and fuse the point cloud to obtain a dense point cloud. a three-dimensional Gaussian field construction module, configured to sample the dense point cloud by voxel-guided farthest point sampling to obtain an initial point cloud, and construct a three-dimensional Gaussian field based on the initial point cloud; a geometry regularization module, configured to render the three-dimensional Gaussian field by an enhanced geometry renderer to obtain rendered depth and rendered normal, construct a multi-level geometry regularization loss function, and optimize the three-dimensional Gaussian field, the multi-level geometry regularization loss function including MVS depth loss, monocular depth loss, monocular normal loss, and depth-normal consistency loss; an optimization adjustment module, configured to optimize and adjust the three-dimensional Gaussian field based on a shape-scale constraint criterion and a two-stage adaptive opacity constraint strategy to obtain an optimized three-dimensional Gaussian field; the three-dimensional Gaussian field construction module is configured to sample the dense point cloud by voxel-guided farthest point sampling to obtain an initial point cloud, and construct a three-dimensional Gaussian field based on the initial point cloud, and the specific process of constructing the three-dimensional Gaussian field based on the initial point cloud includes the following steps: In step S31, the voxel-guided farthest point sampling method includes: In step S311, the scene in which the dense point cloud is located is voxelized to divide a plurality of voxel units; In step S312, the point density in each voxel unit is calculated, the point density being a ratio of the number of point clouds contained in the voxel unit to the volume of the voxel unit; In step S313, each voxel unit is assigned an inverse sampling rate according to the point density, wherein the voxel unit with a high point density is assigned a low sampling rate, and the voxel unit with a low point density is assigned a high sampling rate; In step S314, in each voxel unit, the farthest point sampling algorithm is executed according to the assigned inverse sampling rate to obtain the initial point cloud. ​ ​

Images