New view synthesis method based on Gaussian probability distribution and feature regularization

Abstract

The invention provides a new view synthesis method based on Gaussian probability distribution and feature regularization, and relates to the technical field of new view synthesis, and the method comprises the steps: extracting a feature tensor from a preprocessing input image, and carrying out the Gaussian probability calculation based on the mean value and standard deviation of the feature tensor; and introducing elastic network regularization loss to restrain the feature tensor, carrying out mean value calculation on the Gaussian probability of the feature tensor, substituting a calculation result into a loss function, minimizing the loss function, obtaining an optimized Gaussian point set, carrying out multi-scale densification and pruning, and generating a new view. Through the method and the device, the technical problem that the view synthesis accuracy is further influenced due to too few Gaussian points and low transparency caused by overfitting in a sparse scene is solved, and by introducing Gaussian probability mapping and feature regularization, the modeling effect and the view synthesis accuracy in a sparse view angle are improved, the efficiency is increased, and the precision is improved.

Description

Technical Field

[0001] This application relates to the field of novel view synthesis technology, and in particular to a novel view synthesis method based on Gaussian probability distribution and feature regularization. Background Technology

[0002] Novel View Synthesis (NVS) is an important field in computer vision and graphics, aiming to capture and render realistic 3D representations of physical scenes. It generates new 2D views from existing 3D models or images, thus achieving realistic 3D scene rendering. In NVS, advanced techniques such as 3D Gaussian Splatting (3DGS) and Neural Radiation Fields (NNF) are widely used. 3DGS represents the scene by iteratively optimizing a set of Gaussian distributions with parameters such as position, color, and transparency, forming a point cloud-based volume representation method that can provide color and density for any point in 3D space, thus producing relatively realistic rendering results. In contrast, NSF focuses on implicit representations, synthesizing new views by learning the latent representation of the scene. While both methods achieve good rendering results, they both face the problem of overfitting under sparse scene data, especially when the number of images is limited or the camera's field of view is small, leading to incorrect geometric structures or poor rendering quality. Due to the limited number of camera viewpoints and the small scene radius, the number of generated Gaussian ellipsoids is insufficient, and their transparency is too low. This causes Gaussian points to be filtered out during the trimming process, resulting in blank models and an inability to complete effective modeling. This not only leads to a lack of detail in the generated views but also causes errors in gradient calculations during training, further affecting the model's performance and thus impacting the accuracy of view synthesis.

[0003] In summary, existing technologies suffer from the technical problem that overfitting is prone to occur in sparse scenes, resulting in too few Gaussian points and low transparency, which further affects the accuracy of view synthesis. Summary of the Invention

[0004] The purpose of this application is to provide a new view synthesis method based on Gaussian probability distribution and feature regularization, in order to solve the technical problem in the prior art that overfitting is prone to occur in sparse scenes, resulting in too few Gaussian points and low transparency, which further affects the accuracy of view synthesis.

[0005] In view of the above problems, this application provides a novel view synthesis method based on Gaussian probability distribution and feature regularization. The method includes: extracting feature tensors from a preprocessed input image to obtain a first feature tensor and a second feature tensor, where the feature tensors represent the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the higher-order spherical harmonic function coefficients of the Gaussian point; calculating the Gaussian probability of the feature tensors based on the mean and standard deviation of the first and second feature tensors to obtain the Gaussian probability of the first and second feature tensors; introducing an elastic mesh regularization loss to constrain the first and second feature tensors, and calculating the mean of the Gaussian probabilities of the first and second feature tensors, substituting the calculation results into a loss function, minimizing the loss function, and obtaining an optimized Gaussian point set; and performing multi-scale densification and pruning on the Gaussian point set using a threshold group to generate a new view.

[0006] Optionally, the mean and standard deviation of the first and second feature tensors are calculated using the formulas for calculating the mean, standard deviation, and standard deviation of the first and second feature tensors, respectively, to obtain the mean, standard deviation, and standard deviation of the first and second feature tensors. Then, the Gaussian probability of the first feature tensor is calculated using the first Gaussian probability formula, combined with the mean and standard deviation of the first feature tensor. Finally, the Gaussian probability of the second feature tensor is calculated using the second Gaussian probability formula, combined with the mean and standard deviation of the second feature tensor.

[0007] Optionally, the formula for calculating the mean of the first feature tensor is obtained, wherein the formula for calculating the mean of the first feature tensor is:

[0008] Where, μ dc The first feature tensor is the mean, N is the total number of points in the point cloud, and features_dc is the total number of points in the point cloud. i Let be the first feature tensor of the i-th point in the point cloud.

[0009] Obtain the formula for calculating the standard deviation of the first feature tensor, where the formula for calculating the standard deviation of the first feature tensor is:

[0010] Where, σ dc The standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. i Let μ be the first feature tensor of the i-th point in the point cloud. dc The mean of the first characteristic tensor.

[0011] Obtain the formula for calculating the mean of the second feature tensor, where the formula for calculating the mean of the second feature tensor is:

[0012] Where, μ rest The second feature tensor mean, where N is the total number of points in the point cloud, and features_rest is the mean of the second feature tensor. i Let be the second feature tensor of the i-th point in the point cloud.

[0013] Obtain the formula for calculating the standard deviation of the second feature tensor, where the formula for calculating the standard deviation of the second feature tensor is:

[0014] Where, σ rest The standard deviation of the second feature tensor, N, is the total number of point clouds, and features_rest i Let μ be the second feature tensor of the i-th point in the point cloud. rest The mean of the second characteristic tensor.

[0015] Optionally, the formula for calculating the first Gaussian probability is obtained, wherein the formula for calculating the first Gaussian probability is:

[0016] Wherein, P(features) dc ) represents the Gaussian probability of the first feature tensor, features dc Let μ be the first characteristic tensor. dc Let σ be the mean of the first characteristic tensor. dc Let be the standard deviation of the first feature tensor, and ∈ be the noise level constant.

[0017] Obtain the formula for calculating the second Gaussian probability, where the formula for calculating the second Gaussian probability is:

[0018] Where P(features_rest) is the Gaussian probability of the second feature tensor, features_rest is the second feature tensor, and μ rest Let σ be the mean of the second characteristic tensor. rest Let be the standard deviation of the second feature tensor, and ∈ be the noise level constant.

[0019] Optionally, the elastic net regularization loss is:

[0020] Where, λ 11 and λ l2The regularization parameters control the weights of the regularization. ||features_dc||1 is the result of L1 regularization of the first feature tensor, ||features_rest||1 is the result of L1 regularization of the second feature tensor, ||features_dc||2 is the result of L2 regularization of the first feature tensor, and ||features_rest||2 is the result of L2 regularization of the second feature tensor.

[0021] Optionally, the threshold set includes a gradient threshold, a scale threshold, and a pruning threshold.

[0022] Optionally, the Gaussian point set is analyzed one by one using the densification condition discriminant formula to obtain a densified Gaussian point set; the densification operation is performed by traversing the densified Gaussian point set; and it is determined whether the transparency of the Gaussian points after densification is less than a preset transparency threshold. If so, pruning is performed.

[0023] Optionally, the densification condition criterion formula is:

[0024] Where, densify mask[i] is the i-th point in the point cloud. Let be the gradient at the i-th point, grad threshold be the gradient threshold, and scaling be the gradient threshold. i Let be the scale of the i-th point, and let scaling threshold be the scale threshold.

[0025] The technical solution provided in this application has at least the following beneficial effects:

[0026] By extracting feature tensors from the preprocessed input image, a first feature tensor and a second feature tensor are obtained. Each feature tensor represents the feature information of a Gaussian point; the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the coefficients of the higher-order spherical harmonic function of the Gaussian point. Based on the mean and standard deviation of the first and second feature tensors, Gaussian probabilities are calculated to obtain the Gaussian probabilities of the first and second feature tensors. An elastic mesh regularization loss is introduced to constrain the first and second feature tensors, and the mean of their Gaussian probabilities is calculated. The calculated results are substituted into the loss function, and the loss function is minimized to obtain an optimized set of Gaussian points. A threshold group is then used to perform multi-scale densification and pruning on the Gaussian point set to generate a new view. In other words, by introducing Gaussian probability mapping and feature regularization, the problems of insufficient Gaussian points and low transparency in sparse scenes are overcome, optimizing the modeling and view synthesis process under sparse perspectives and effectively improving the accuracy of modeling, especially when processing sparse data. By performing probabilistic mapping on the input feature tensor, we can not only capture the potential patterns in the data but also quantify uncertainty, thereby improving the modeling effect and the accuracy of view synthesis under sparse perspectives. By applying multi-scale densification and pruning strategies to the Gaussian point set using threshold groups, we can accurately locate the Gaussian ellipsoid and complete density optimization, enabling the generation of new views in sparse environments, reducing model complexity, and improving computational efficiency.

[0027] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a novel view synthesis method based on Gaussian probability distribution and feature regularization proposed in this application.

[0030] Figure 2This is a flowchart illustrating the process of obtaining the first feature tensor Gaussian probability and the second feature tensor Gaussian probability in a novel view synthesis method based on Gaussian probability distribution and feature regularization according to this application. Detailed Implementation

[0031] This application provides a novel view synthesis method based on Gaussian probability distribution and feature regularization, addressing the technical problem in existing technologies where overfitting in sparse scenes leads to insufficient Gaussian points and low transparency, further affecting the accuracy of view synthesis. By introducing Gaussian probability mapping and feature regularization to overcome the issues of insufficient Gaussian points and low transparency in sparse scenes, the method optimizes the modeling and view synthesis process in sparse perspectives, effectively improving modeling accuracy, especially when handling sparse data. Probabilistic mapping of the input feature tensor not only helps capture the potential patterns in the data but also quantifies uncertainty, thereby improving the modeling effect and view synthesis accuracy in sparse perspectives. By applying over-scale densification and pruning strategies to the Gaussian point set using threshold groups, the position of the Gaussian ellipsoid is accurately located and density optimization is completed, enabling the generation of new views in sparse environments, reducing model complexity, and improving computational efficiency.

[0032] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0033] For examples, please refer to the appendix. Figure 1 This application provides a novel view synthesis method based on Gaussian probability distribution and feature regularization. The novel view synthesis method based on Gaussian probability distribution and feature regularization specifically includes the following steps:

[0034] S100: Extract feature tensors from the preprocessed input image to obtain a first feature tensor and a second feature tensor. The feature tensor represents the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the higher-order spherical harmonic function coefficients of the Gaussian point.

[0035] Specifically, feature tensors are extracted from the preprocessed input image, including a first feature tensor and a second feature tensor. These are feature tensors from the input image or scene data, used for model training and Gaussian probability calculation. As described, each feature tensor represents the feature information of each Gaussian point; the first feature tensor represents the color features of the Gaussian point, and the second feature tensor represents the higher-order spherical harmonic function coefficients of the Gaussian point. In image synthesis, color is an important visual attribute; therefore, the first feature tensor captures the color information (e.g., RGB values) of each point. Spherical harmonic functions are a set of mathematical tools used to describe functions on a sphere, representing directionality and rotational symmetry in 3D graphics and computer vision. Higher-order spherical harmonic function coefficients describe the lighting and geometric features of an object, especially how the object responds to lighting from different viewpoints.

[0036] S200: Based on the mean and standard deviation of the first and second feature tensors, calculate the Gaussian probability of the feature tensors to obtain the Gaussian probability of the first and second feature tensors.

[0037] Further details are attached. Figure 2 As shown, S200 of this application includes:

[0038] S210: Using the formulas for calculating the mean and standard deviation of the first and second characteristic tensors, respectively, calculate the mean and standard deviation of the first and second characteristic tensors to obtain the mean, standard deviation, and standard deviation of the first and second characteristic tensors; S220: Using the formula for calculating the first Gaussian probability, combine the mean and standard deviation of the first and second characteristic tensors to calculate the Gaussian probability to obtain the Gaussian probability of the first characteristic tensor; S230: Using the formula for calculating the second Gaussian probability, combine the mean and standard deviation of the second and second characteristic tensors to calculate the Gaussian probability to obtain the Gaussian probability of the second characteristic tensor.

[0039] Obtain the formula for calculating the mean of the first feature tensor, where the formula for calculating the mean of the first feature tensor is:

[0040] Where, μ dc The first feature tensor is the mean, N is the total number of points in the point cloud, and features_dc is the total number of points in the point cloud. i Let be the first feature tensor of the i-th point in the point cloud.

[0041] Obtain the formula for calculating the standard deviation of the first feature tensor, where the formula for calculating the standard deviation of the first feature tensor is:

[0042] Where, σ dcThe standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. i Let μ be the first feature tensor of the i-th point in the point cloud. dc The mean of the first characteristic tensor.

[0043] Obtain the formula for calculating the mean of the second feature tensor, where the formula for calculating the mean of the second feature tensor is:

[0044] Where, μ rest The second feature tensor mean, where N is the total number of points in the point cloud, and features_rest is the mean of the second feature tensor. i Let be the second feature tensor of the i-th point in the point cloud.

[0045] Obtain the formula for calculating the standard deviation of the second feature tensor, where the formula for calculating the standard deviation of the second feature tensor is:

[0046] Where, σ rest The standard deviation of the second feature tensor, N, is the total number of point clouds, and features_rest i Let μ be the second feature tensor of the i-th point in the point cloud. rest The mean of the second characteristic tensor.

[0047] Specifically, the mean of a feature tensor is the average value of each feature in the tensor, reflecting the global distribution of that feature. For image features, the mean is typically used to measure the central tendency of the feature, providing a sense of the overall image features. The standard deviation is the dispersion of each feature in the feature tensor, indicating how much the data points deviate from the mean. A larger standard deviation indicates that the feature varies more spatially, while a smaller standard deviation indicates that the feature distribution is more concentrated. Gaussian probability is used to describe the probability density function of feature data under a Gaussian distribution. Using the mean and standard deviation, the probability value of each data point under this distribution is calculated, and it is typically used to measure the distance between the data point and the mean.

[0048] Calculation formula using the first characteristic tensor mean The mean of the first feature tensor is calculated by applying the mean value to the first feature tensor. Where μ... dc The first feature tensor is the mean, N is the total number of points in the point cloud, and features_dc is the total number of points in the point cloud. i Let be the first feature tensor of the i-th point in the point cloud.

[0049] Calculation formula using the first characteristic tensor standard deviation The standard deviation of the first characteristic tensor is calculated. Where σ dc The standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. iLet μ be the first feature tensor of the i-th point in the point cloud. dc The mean of the first characteristic tensor.

[0050] Calculation formula using the second characteristic tensor mean The mean of the second feature tensor is calculated. Where μ rest The second feature tensor mean, where N is the total number of points in the point cloud, and features_rest is the mean of the second feature tensor. i Let be the second feature tensor of the i-th point in the point cloud.

[0051] Calculation formula using the second characteristic tensor standard deviation The standard deviation of the second characteristic tensor is calculated, yielding the standard deviation of the second characteristic tensor. Where σ... rest The standard deviation of the second feature tensor, N, is the total number of point clouds, and features_rest i Let μ be the second feature tensor of the i-th point in the point cloud. rest The mean of the second characteristic tensor.

[0052] Furthermore, this application also includes the following steps:

[0053] Obtain the formula for calculating the first Gaussian probability, where the formula for calculating the first Gaussian probability is:

[0054] Wherein, P(features) dc ) represents the Gaussian probability of the first feature tensor, features dc Let μ be the first characteristic tensor. dc Let σ be the mean of the first characteristic tensor. dc Let be the standard deviation of the first feature tensor, and ∈ be the noise level constant.

[0055] Obtain the formula for calculating the second Gaussian probability, where the formula for calculating the second Gaussian probability is:

[0056] Where P(features_rest) is the Gaussian probability of the second feature tensor, features_rest is the second feature tensor, and μ rest Let σ be the mean of the second characteristic tensor. rest Let be the standard deviation of the second feature tensor, and ∈ be the noise level constant.

[0057] Specifically, the Gaussian probability of the first feature tensor is calculated using the first Gaussian probability calculation formula, combined with the mean and standard deviation of the first feature tensor. The formula for calculating the first Gaussian probability is as follows: Wherein, P(features) dc ) represents the Gaussian probability of the first feature tensor, featuresdc Let μ be the first characteristic tensor. dc Let σ be the mean of the first characteristic tensor. dc Let be the standard deviation of the first feature tensor, and ∈ be the noise level constant.

[0058] Using the formula for calculating the second Gaussian probability, and combining the mean and standard deviation of the second feature tensor, the Gaussian probability of the second feature tensor is calculated to obtain the Gaussian probability of the second feature tensor. The formula for calculating the second Gaussian probability is: Where P(features_rest) is the Gaussian probability of the second feature tensor, features_rest is the second feature tensor, and μ rest Let σ be the mean of the second characteristic tensor. rest Let be the standard deviation of the second feature tensor, and ∈ be the noise level constant.

[0059] By calculating the probabilistic features of the Gaussian distribution (including matrix, standard deviation, and probability density) and incorporating them into the 3DGS loss function, the feature information of sparse points is increased, optimizing model training and improving modeling quality. Unlike classic 3DGS methods, which only use the Gaussian distribution for scene geometry representation and differentiable rendering, this approach uses the Gaussian distribution as a probabilistic statistical tool to establish a probabilistic mapping model for the feature space. This dual-application mechanism provides dual constraints under sparse observation conditions, maintaining the geometric expressive power of the original 3DGS while enhancing features through probabilistic statistics, capturing potential patterns in the data, and enhancing the modeling ability for sparse regions. Probabilistic statistical modeling effectively reduces uncertainty in the reconstruction process, thus supporting further inference and decision-making. By calculating the Gaussian probability of each data point, the position of the data point in its feature distribution is accurately measured, and the weights of different features are adjusted, resulting in more accurate and stable results in tasks such as view compositing and image reconstruction.

[0060] S300: An elastic mesh regularization loss is introduced to constrain the first and second feature tensors, and the mean of the Gaussian probabilities of the first and second feature tensors is calculated. The calculation result is substituted into the loss function to minimize the loss function and obtain the optimized Gaussian point set.

[0061] Furthermore, this application S300 includes:

[0062] The regularization loss of the elastic network is:

[0063] Where, λ 11 and λ l2The regularization parameters control the weights of the regularization. ||features_dc||1 is the result of L1 regularization of the first feature tensor, ||features_rest||1 is the result of L1 regularization of the second feature tensor, ||features_dc||2 is the result of L2 regularization of the first feature tensor, and ||features_rest||2 is the result of L2 regularization of the second feature tensor.

[0064] Specifically, when the 3D Gaussian function cannot adequately represent the scene, it initiates an adaptive density control mechanism to increase the density. However, due to a lack of constraints, the newly generated 3D Gaussian function is prone to overfitting the training view, causing it to continuously move towards overfitting. Therefore, elastic mesh regularization is used for feature constraints, combining L1 and L2 regularization terms.

[0065] L1 regularization is:

[0066]

[0067] Where d1 and d2 are the sizes of the feature vectors of features_dc and features_rest, respectively, and features_dc i Let features_rest be the first feature tensor of the i-th point in the point cloud. i Let be the second feature tensor of the i-th point in the point cloud.

[0068] L2 regularization becomes:

[0069]

[0070] Where d1 and d2 are the sizes of the feature vectors of features_dc and features_rest, respectively, and features_dc i Let features_rest be the first feature tensor of the i-th point in the point cloud. i Let be the second feature tensor of the i-th point in the point cloud.

[0071] Finally, the L1 and L2 regularizations are combined to obtain the elastic net regularization loss as follows:

[0072]

[0073] Where, λ 11 and λ l2The regularization parameters control the weights of the regularization. ||features_dc||1 is the result of L1 regularization of the first feature tensor, ||features_rest||1 is the result of L1 regularization of the second feature tensor, ||features_dc||2 is the result of L2 regularization of the first feature tensor, and ||features_rest||2 is the result of L2 regularization of the second feature tensor.

[0074] L1 regularization, through feature constraints, allows the model to automatically select important features and reset the weights of certain features to zero, helping to retain features that significantly contribute to the model and reducing model complexity. L2 regularization, on the other hand, suppresses overfitting by constraining weights, forcing the model to maintain smoothness and preventing some feature weights from having an excessively large impact on the results. Elastic net regularization innovatively combines the advantages of L1 and L2 regularization, achieving both feature selection and maintaining model stability, significantly enhancing generalization ability in high-dimensional feature spaces. This regularization strategy enables an adaptive balance between fitting ability and complexity when facing high-dimensional feature spaces. Combined with the proposed probabilistic mapping framework, it not only provides a robust optimization paradigm for 3D scene modeling and enhances the modeling accuracy of sparse regions through probabilistic statistical constraints, but also constructs a general probabilistic-regularized co-optimization framework; it also has good scalability and application potential in other fields requiring probabilistic inference and model refinement.

[0075] By introducing an elastic mesh regularization loss to constrain the first and second feature tensors, and calculating the mean of the Gaussian probabilities of the first and second feature tensors, the calculated mean is substituted into the loss function, and the model is optimized by minimizing this loss function. The Gaussian point information of the first and second feature tensors is optimized, including the mean, standard deviation, and transparency of each point. During the optimization process, the model generates new Gaussian points, improving the accuracy of the Gaussian points.

[0076] S400: Performs multi-scale densification and pruning on the Gaussian point set using threshold groups to generate a new view.

[0077] The threshold group includes gradient threshold, scale threshold, and pruning threshold.

[0078] Furthermore, this application S400 includes:

[0079] The Gaussian point set is analyzed one by one using the densification condition discriminant formula to obtain the densified Gaussian point set; the densification operation is performed by traversing the densified Gaussian point set; it is determined whether the transparency of the Gaussian points after densification is less than the preset transparency threshold. If so, pruning is performed.

[0080] Furthermore, this application also includes the following steps:

[0081] The formula for determining the densification condition is:

[0082] Where, densify mask[i] is the i-th point in the point cloud. Let be the gradient at the i-th point, grad threshold be the gradient threshold, and scaling be the gradient threshold. i Let be the scale of the i-th point, and let scaling threshold be the scale threshold.

[0083] Specifically, a threshold set is set, including gradient threshold, scale threshold, and pruning threshold. Gradient threshold is commonly used in image processing for edge detection, limiting the rate of change of pixel values ​​in an image. After setting the gradient threshold, it can be determined which points have larger gradients and require more points for densification. Scale threshold is used to control the scale (size) of points during point cloud densification or pruning. By setting a scale threshold, only points within a certain scale range are retained, thus avoiding unnecessary computation and improving efficiency. Pruning threshold is used in Gaussian point processing to determine whether to delete certain points. If a point's transparency, weight, or other attributes are below the pruning threshold, these points are considered unimportant and are removed.

[0084] When using dense input, both the number of 3D Gaussians and rendering performance steadily improve. One reason is that density control strategies increase the number of 3D Gaussians to meet the requirements of rendering the training view. However, with sparse input, limited information makes overfitting easy from a training perspective. Overfitting generates a large number of floating points and incorrect geometry, making it impossible to further optimize the training view information, ultimately leading to rendering failure of the new view. To address this issue, multi-scale densification and pruning strategies were improved to allow for effective operation across multiple scales. The core idea of ​​multi-scale densification is to improve the model density by operating at different resolutions or scales, while pruning determines which points should be removed at the appropriate scale.

[0085] For a Gaussian point set, the location of the Gaussian ellipsoid is accurately located and density optimization is performed through seed point selection and seed point supplementation strategies. Seed point selection aims to select the key points with the most modeling value, while the supplementation step specifically enhances the sampling density of local regions. If there are too few seed points, the transparency of the generated Gaussians will be insufficient, leading to their deletion during adaptive densification. Multiple thresholds are introduced for dynamic density control to prevent an insufficient number of Gaussian ellipsoids. Specifically, for each training iteration, the density and accuracy of the point cloud are dynamically adjusted through multi-scale densification and pruning operations to improve training efficiency and control model complexity.

[0086] By using the densification condition discriminant formula, and based on the Gaussian point set, the characteristics of each point (such as position, color, transparency, etc.) are analyzed one by one to determine whether it meets the densification condition. The densification condition discriminant formula is as follows: Here, densify mask[i] represents the i-th point in the point cloud, and densify_mask is a binary mask used to control the point cloud densification process. Each element corresponds to a point, with a value of 1 indicating that the point needs to be densified and a value of 0 indicating that it does not need to be densified. This determines which points need to be densified, i.e., adding more points to improve the detail representation of the model. Let be the gradient at the i-th point, representing the degree of change of that point during training. A larger gradient indicates that the point plays a more important role in the model. The grad threshold is used to determine which points have large gradients and require more points for densification. i Let be the scale of the i-th point, and let be the scaling threshold, which is used to determine which points are in a larger scale range and may require more points. For each point, check whether its gradient is greater than the threshold and whether its scale meets the requirements.

[0087] When the gradient magnitude of the point When the scale of a point exceeds the gradient threshold and its scaling value (scaling_i) exceeds the scaling threshold, the point will be densified, meaning more points will be added in its vicinity to improve the model's detail representation. If the condition is met, the point is densified; otherwise, it is not. This introduces an incremental densification threshold. Then, based on the specific scale and gradient conditions of the point cloud, when the densification threshold meets the current condition, the point cloud is densified. The selected points are sampled to increase the density of the point cloud, thus representing the model's details more finely. Finally, progressively increasing pruning thresholds are introduced, using the transparency of specific points as a criterion. Specifically, when the transparency of a point is below the threshold, the point is considered unimportant and can be pruned. Alternatively, pruning can be determined by checking the maximum scale of the point. If the point's scale is very small, pruning can be considered.

[0088] The multi-scale densification strategy dynamically selects Gaussian points to be densified based on the densification threshold and gradient threshold at the current scale; simultaneously, it applies different densification thresholds in each training step to achieve densification at multiple resolutions. The multi-scale pruning strategy determines which points should be removed based on the threshold and point transparency at each scale, ensuring that the model effectively prunes unimportant points at each scale, thereby reducing model complexity and improving computational efficiency. Furthermore, to prevent gradient explosion from causing training instability, a gradient pruning operation is introduced. The gradient range is constrained before each optimization step to ensure that the gradient does not become excessively large. Through the synergistic optimization of these methods, a balance is achieved between training stability, computational efficiency, and model accuracy, significantly improving the quality of point cloud modeling and training speed.

[0089] In summary, the novel view synthesis method based on Gaussian probability distribution and feature regularization provided in this application has the following beneficial effects:

[0090] By extracting feature tensors from the preprocessed input image, a first feature tensor and a second feature tensor are obtained. Each feature tensor represents the feature information of a Gaussian point; the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the coefficients of the higher-order spherical harmonic function of the Gaussian point. Based on the mean and standard deviation of the first and second feature tensors, Gaussian probabilities are calculated to obtain the Gaussian probabilities of the first and second feature tensors. An elastic mesh regularization loss is introduced to constrain the first and second feature tensors, and the mean of their Gaussian probabilities is calculated. The calculated results are substituted into the loss function, and the loss function is minimized to obtain an optimized set of Gaussian points. A threshold group is then used to perform multi-scale densification and pruning on the Gaussian point set to generate a new view. In other words, by introducing Gaussian probability mapping and feature regularization, the problems of insufficient Gaussian points and low transparency in sparse scenes are overcome, optimizing the modeling and view synthesis process under sparse perspectives and effectively improving the accuracy of modeling, especially when processing sparse data. By performing probabilistic mapping on the input feature tensor, we can not only capture the potential patterns in the data but also quantify uncertainty, thereby improving the modeling effect and the accuracy of view synthesis under sparse perspectives. By applying multi-scale densification and pruning strategies to the Gaussian point set using threshold groups, we can accurately locate the Gaussian ellipsoid and complete density optimization, enabling the generation of new views in sparse environments, reducing model complexity, and improving computational efficiency.

[0091] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0092] Obviously, those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A novel view synthesis method based on Gaussian probability distribution and feature regularization, characterized in that, include: Feature tensors are extracted from the preprocessed input image to obtain a first feature tensor and a second feature tensor. The feature tensors represent the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the higher-order spherical harmonic function coefficients of the Gaussian point. Based on the mean and standard deviation of the first and second feature tensors, Gaussian probabilities of the feature tensors are calculated to obtain the Gaussian probabilities of the first and second feature tensors. The elastic mesh regularization loss is introduced to constrain the first feature tensor and the second feature tensor. The mean of the Gaussian probability of the first feature tensor and the Gaussian probability of the second feature tensor is calculated. The calculation result is substituted into the loss function to minimize the loss function and obtain the optimized Gaussian point set. A new view is generated by multi-scale densification and pruning of the Gaussian point set using threshold groups.

2. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 1, characterized in that, include: Using the formulas for calculating the mean of the first feature tensor, the standard deviation of the first feature tensor, the mean of the second feature tensor, and the standard deviation of the second feature tensor, the mean and standard deviation of the first feature tensor and the second feature tensor are calculated to obtain the mean of the first feature tensor, the standard deviation of the first feature tensor, the mean of the second feature tensor, and the standard deviation of the second feature tensor. Using the first Gaussian probability calculation formula, combined with the mean and standard deviation of the first feature tensor, Gaussian probability is calculated to obtain the Gaussian probability of the first feature tensor. The Gaussian probability of the second feature tensor is obtained by using the formula for calculating the second Gaussian probability, combined with the mean of the second feature tensor and the standard deviation of the second feature tensor.

3. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 2, characterized in that, include: Obtain the formula for calculating the mean of the first feature tensor, where the formula for calculating the mean of the first feature tensor is: Where, μ dc The first feature tensor is the mean, N is the total number of points in the point cloud, and features_dc is the total number of points in the point cloud. i Let be the first feature tensor of the i-th point in the point cloud; Obtain the formula for calculating the standard deviation of the first feature tensor, where the formula for calculating the standard deviation of the first feature tensor is: Where, σ dc The standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. i Let μ be the first feature tensor of the i-th point in the point cloud. dc The mean of the first characteristic tensor; Obtain the formula for calculating the mean of the second feature tensor, where the formula for calculating the mean of the second feature tensor is: Where, μ rest The second feature tensor mean, where N is the total number of points in the point cloud, and features_rest is the mean of the second feature tensor. i Let be the second feature tensor of the i-th point in the point cloud; Obtain the formula for calculating the standard deviation of the second feature tensor, where the formula for calculating the standard deviation of the second feature tensor is: Where, σ rest The standard deviation of the second feature tensor, N, is the total number of point clouds, and features_rest i Let μ be the second feature tensor of the i-th point in the point cloud. rest The mean of the second characteristic tensor.

4. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 2, characterized in that, include: Obtain the formula for calculating the first Gaussian probability, where the formula for calculating the first Gaussian probability is: Wherein, P(features) dc ) represents the Gaussian probability of the first feature tensor, features dc Let μ be the first characteristic tensor. dc Let σ be the mean of the first characteristic tensor. dc Let be the standard deviation of the first feature tensor, and ∈ be the noise level constant; Obtain the formula for calculating the second Gaussian probability, where the formula for calculating the second Gaussian probability is: Where P(features_rest) is the Gaussian probability of the second feature tensor, features_rest is the second feature tensor, and μ rest Let σ be the mean of the second characteristic tensor. rest Let be the standard deviation of the second feature tensor, and ∈ be the noise level constant.

5. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 1, characterized in that, The regularization loss of the elastic network is: Where, λ 11 and λ l2 Here are the regularization parameters, which control the weights of the regularization. ||features_dc||1 is the result of L1 regularization of the first feature tensor, ||features_rest||1 is the result of L1 regularization of the second feature tensor, ||features_dc||2 is the result of L2 regularization of the first feature tensor, and ||features_rest||2 is the result of L2 regularization of the second feature tensor.

6. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 1, characterized in that, The threshold group includes gradient threshold, scale threshold, and pruning threshold.

7. A novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 6, characterized in that, Obtain threshold sets to densify and prune the Gaussian point set, including: By analyzing the Gaussian point set one by one using the densification condition discriminant formula, the densified Gaussian point set is obtained; Perform the densification operation by traversing the set of densified Gaussian points; Determine whether the transparency of the Gaussian points after densification is less than the preset transparency threshold. If so, prune the nodes.

8. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 7, characterized in that, The formula for determining the densification condition is: Where, densify mask[i] is the i-th point in the point cloud. Let be the gradient at the i-th point, grad threshold be the gradient threshold, and scaling be the gradient threshold. i Let be the scale of the i-th point, and let scaling threshold be the scale threshold.

Images