A new view synthesis method based on gaussian probability distribution and feature regularization

Abstract

The application provides a new view synthesis method based on Gaussian probability distribution and feature regularization, relates to the technical field of new view synthesis, and comprises the following steps: extracting a feature tensor from a preprocessed input image, performing Gaussian probability calculation based on the mean and standard deviation of the feature tensor; introducing an elastic net regularization loss to constrain the feature tensor, performing mean calculation on the Gaussian probability of the feature tensor, substituting the calculation result into a loss function, minimizing the loss function, obtaining an optimized Gaussian point set, performing multi-scale densification and pruning, and generating a new view. The application solves the technical problem that, due to overfitting in a sparse scene, the number of Gaussian points is too small and the transparency is low, which further affects the accuracy of view synthesis, and improves the modeling effect and the accuracy of view synthesis in a sparse view angle by introducing Gaussian probability mapping and feature regularization, thereby increasing efficiency and improving precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of novel view synthesis, and particularly relates to a novel view synthesis method based on Gaussian probability distribution and feature regularization. BACKGROUND

[0002] Novel view synthesis (NVS) is an important field in computer vision and graphics, aiming to capture and render realistic 3D representations of physical scenes, generating new 2D views from existing 3D models or images, thereby realizing realistic 3D scene rendering. In novel view synthesis, some advanced technologies such as 3D Gaussian splatting (3DGS) and neural radiance fields have been widely used. 3D Gaussian splatting represents the scene by iteratively optimizing a set of Gaussian distributions with parameters such as position, color, and transparency, forming a point cloud-based volumetric representation method that can provide color and density for any point in 3D space, thus producing a more realistic effect when rendering. Unlike this, neural radiance fields focus on implicit representation, synthesizing new views by learning the latent representation of the scene. Although these two methods have good rendering effects, they both face the problem of overfitting under sparse scene data, especially when the number of images is limited or the camera angle is small, resulting in incorrect geometric structure or poor rendering effect. Due to the small number of camera angles and the small radius of the scene, the number of generated Gaussian ellipsoids is insufficient, and their transparency is too low, causing Gaussian points to be filtered out during pruning, which in turn leads to a blank model, making it impossible to complete effective modeling. Not only will this result in a lack of detail in the generated view, but it will also cause errors in gradient calculation during the training process, further affecting the performance of the model, thereby affecting the accuracy of view synthesis.

[0003] In summary, the existing technology has the technical problem of view synthesis accuracy being further affected due to overfitting under sparse scenes, resulting in too few Gaussian points and low transparency. SUMMARY

[0004] The purpose of the present application is to provide a novel view synthesis method based on Gaussian probability distribution and feature regularization, to solve the technical problem of view synthesis accuracy being further affected due to overfitting under sparse scenes, resulting in too few Gaussian points and low transparency in the existing technology.

[0005] In view of the above problems, the application provides a new view synthesis method based on Gaussian probability distribution and feature regularization, wherein the new view synthesis method based on Gaussian probability distribution and feature regularization comprises the following steps: extracting feature tensors from preprocessed input images to obtain first feature tensors and second feature tensors, wherein the feature tensors represent feature information of each Gaussian point, the first feature tensors represent color features of the Gaussian points, and the second feature tensors represent high-order spherical harmonic function coefficients of the Gaussian points; performing Gaussian probability calculation on the feature tensors based on the mean and standard deviation of the first feature tensors and the second feature tensors to obtain first feature tensor Gaussian probability and second feature tensor Gaussian probability; introducing an elastic net regularization loss to constrain the first feature tensors and the second feature tensors, and performing mean calculation on the first feature tensor Gaussian probability and the second feature tensor Gaussian probability, and substituting the calculation result into a loss function to minimize the loss function and obtain an optimized Gaussian point set; and performing multi-scale densification and pruning on the Gaussian point set through a threshold group to generate a new view.

[0006] Optionally, the mean and standard deviation of the first feature tensors and the second feature tensors are calculated by using a first feature tensor mean calculation formula, a first feature tensor standard deviation calculation formula, a second feature tensor mean calculation formula and a second feature tensor standard deviation calculation formula to obtain a first feature tensor mean, a first feature tensor standard deviation, a second feature tensor mean and a second feature tensor standard deviation; Gaussian probability calculation is performed on the first feature tensors and the second feature tensors by using a first Gaussian probability calculation formula in combination with the first feature tensor mean and the first feature tensor standard deviation to obtain first feature tensor Gaussian probability; and Gaussian probability calculation is performed on the first feature tensors and the second feature tensors by using a second Gaussian probability calculation formula in combination with the second feature tensor mean and the second feature tensor standard deviation to obtain second feature tensor Gaussian probability.

[0007] Optionally, the first feature tensor mean calculation formula is obtained, wherein the first feature tensor mean calculation formula is as follows:

[0008] wherein μ dc is the first feature tensor mean, N is the total number of point clouds, features_dc i is the first feature tensor of the i th point in the point cloud.

[0009] The first feature tensor standard deviation calculation formula is obtained, wherein the first feature tensor standard deviation calculation formula is as follows:

[0010] wherein σ dc is the first feature tensor standard deviation, N is the total number of point clouds, features_dc i is the first feature tensor of the i th point in the point cloud, and μ dc is the first feature tensor mean.

[0011] A second feature tensor mean calculation formula is obtained, where the second feature tensor mean calculation formula is:

[0012] where μ rest is the second feature tensor mean, N is the total number of the point cloud, features_rest i is the second feature tensor of the i-th point in the point cloud.

[0013] A second feature tensor standard deviation calculation formula is obtained, where the second feature tensor standard deviation calculation formula is:

[0014] where σ rest is the second feature tensor standard deviation, N is the total number of the point cloud, features_rest i is the second feature tensor of the i-th point in the point cloud, and μ rest is the second feature tensor mean.

[0015] Optionally, a first Gaussian probability calculation formula is obtained, where the first Gaussian probability calculation formula is:

[0016] where P(features dc ) is the first feature tensor Gaussian probability, features dc is the first feature tensor, μ dc is the first feature tensor mean, σ dc is the first feature tensor standard deviation, and ∈ is a noise level constant.

[0017] A second Gaussian probability calculation formula is obtained, where the second Gaussian probability calculation formula is:

[0018] where P(features_rest) is the second feature tensor Gaussian probability, features_rest is the second feature tensor, μ rest is the second feature tensor mean, σ rest is the second feature tensor standard deviation, and ∈ is a noise level constant.

[0019] Optionally, the elastic net regularization loss is:

[0020] where λ 11 and λ l2is a regularization parameter that controls the weight of regularization, ||features_dc||1 is a result of L1 regularization of the first feature tensor, ||features_rest||1 is a result of L1 regularization of the second feature tensor, ||features_dc||2 is a result of L2 regularization of the first feature tensor, and ||features_rest||2 is a result of L2 regularization of the second feature tensor.

[0021] Optionally, the threshold set includes a gradient threshold, a scale threshold, and a pruning threshold.

[0022] Optionally, the set of Gaussian points is analyzed one by one using a densification condition discrimination formula to obtain a set of densified Gaussian points; a densification operation is performed on the set of densified Gaussian points; and it is determined whether the transparency of the Gaussian points after the densification processing is less than a preset transparency threshold, and if so, pruning is performed.

[0023] Optionally, the densification condition discrimination formula is as follows:

[0024] wherein densify mask[i] is the i th point in the point cloud, is the gradient of the i th point, grad threshold is the gradient threshold, scaling i is the scale of the i th point, and scaling threshold is the scale threshold.

[0025] The technical solutions provided in the present application have at least the following beneficial effects:

[0026] The first feature tensor and the second feature tensor are obtained by extracting feature tensors from a preprocessed input image, wherein the feature tensors represent feature information of each Gaussian point, the first feature tensor represents color features of the Gaussian points, and the second feature tensor represents high-order spherical harmonic function coefficients of the Gaussian points; Gaussian probability calculation of the feature tensors is performed based on the mean and standard deviation of the first feature tensor and the second feature tensor, to obtain first feature tensor Gaussian probability and second feature tensor Gaussian probability; the first feature tensor and the second feature tensor are constrained by introducing an elastic net regularization loss, and mean calculation is performed on the first feature tensor Gaussian probability and the second feature tensor Gaussian probability, and the calculation result is substituted into a loss function to minimize the loss function, to obtain an optimized Gaussian point set; and the Gaussian point set is subjected to multi-scale densification and pruning by a threshold group to generate a new view. That is, by introducing Gaussian probability mapping and feature regularization, the problem of too few Gaussian points and low transparency in a sparse scene is overcome, the modeling and view synthesis process in a sparse view angle are optimized, and the accuracy of modeling is effectively improved, especially when processing sparse data. By performing probability mapping on the input feature tensors, not only the potential laws of the data can be captured, but also the uncertainty can be quantified, thereby improving the modeling effect in a sparse view angle and the accuracy of view synthesis. By adopting the multi-scale densification and pruning strategy on the Gaussian point set by the threshold group, the positions of the Gaussian ellipsoids are accurately located and density optimization is completed, new view generation in a sparse environment is realized, the model complexity is reduced, and the calculation efficiency is improved.

[0027] The above description is only a summary of the technical solutions of the present application, in order to enable the technical means of the present application to be more clearly understood, and to be implemented in accordance with the content of the description, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only exemplary, and other drawings can be obtained by those skilled in the art without creating any inventive labor on the basis of the provided drawings.

[0029] Figure 1 The flowchart of a new view synthesis method based on Gaussian probability distribution and feature regularization of the present application.

[0030] Figure 2A flowchart for obtaining a first feature tensor Gaussian probability and a second feature tensor Gaussian probability in a new view synthesis method based on Gaussian probability distribution and feature regularization of the application. DETAILED DESCRIPTION

[0031] The application provides a new view synthesis method based on Gaussian probability distribution and feature regularization, which solves the technical problem in the prior art that overfitting easily occurs in a sparse scene, resulting in too few Gaussian points and low transparency, which further affects the accuracy of view synthesis. The introduction of Gaussian probability mapping and feature regularization overcomes the problem of too few Gaussian points and low transparency in a sparse scene, optimizes the modeling and view synthesis process in a sparse view angle, effectively improves the accuracy of modeling, especially when processing sparse data. By performing probability mapping on the input feature tensor, not only the potential law of the data can be captured, but also the uncertainty can be quantified, thereby improving the modeling effect in a sparse view angle and the accuracy of view synthesis. By adopting the excessive scale densification and pruning strategy on the Gaussian point set through threshold groups, the Gaussian ellipsoid position is accurately located and density optimization is completed, new view generation in a sparse environment is realized, model complexity is reduced, and calculation efficiency is improved.

[0032] Hereinafter, the technical solutions in the application will be described clearly and completely with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. It should be understood that the application is not limited by the example embodiments described herein. Based on the embodiments of the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the application. In addition, it should be noted that, for convenience of description, only parts related to the application are shown in the drawings, not all.

[0033] Embodiment, please refer to the accompanying Figure 1 The application provides a new view synthesis method based on Gaussian probability distribution and feature regularization, which specifically includes the following steps:

[0034] S100: Extracting a feature tensor from a preprocessed input image to obtain a first feature tensor and a second feature tensor, wherein the feature tensor represents the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the high-order spherical harmonic function coefficient of the Gaussian point.

[0035] Specifically, the feature tensors extracted from the pre-processed input image, including the first feature tensor and the second feature tensor, are feature tensors in the input image or scene data, which are used for model training and Gaussian probability calculation. According to the description, the feature tensors represent the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the high-order spherical harmonic function coefficient of the Gaussian point. In image synthesis, color is an important visual attribute, so the first feature tensor captures the color information (such as RGB value) of each point. Spherical harmonics is a set of mathematical tools used to describe functions on a sphere, which is used to represent directionality and rotational symmetry in 3D graphics and computer vision. High-order spherical harmonic function coefficients describe the lighting and geometric characteristics of an object, especially how the object responds to lighting at different viewing angles.

[0036] S200: Gaussian probability calculation of the feature tensor is performed based on the mean and standard deviation of the first feature tensor and the second feature tensor, and the first feature tensor Gaussian probability and the second feature tensor Gaussian probability are obtained.

[0037] Further, as shown in the accompanying Figure 2 S200 of the present application includes:

[0038] S210: The mean and standard deviation of the first feature tensor and the second feature tensor are calculated using the first feature tensor mean calculation formula, the first feature tensor standard deviation calculation formula, the second feature tensor mean calculation formula and the second feature tensor standard deviation calculation formula, and the first feature tensor mean, the first feature tensor standard deviation, the second feature tensor mean and the second feature tensor standard deviation are obtained; S220: The first feature tensor Gaussian probability is obtained by using the first Gaussian probability calculation formula and combining the first feature tensor mean and the first feature tensor standard deviation for Gaussian probability calculation; S230: The second feature tensor Gaussian probability is obtained by using the second Gaussian probability calculation formula and combining the second feature tensor mean and the second feature tensor standard deviation for Gaussian probability calculation.

[0039] The first feature tensor mean calculation formula is obtained, wherein the first feature tensor mean calculation formula is:

[0040] wherein μ dc is the first feature tensor mean, N is the total number of point clouds, features_dc i is the first feature tensor of the i-th point in the point cloud.

[0041] The first feature tensor standard deviation calculation formula is obtained, wherein the first feature tensor standard deviation calculation formula is:

[0042] wherein σ dcThe standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. i Let μ be the first feature tensor of the i-th point in the point cloud. dc The mean of the first characteristic tensor.

[0043] Obtain the formula for calculating the mean of the second feature tensor, where the formula for calculating the mean of the second feature tensor is:

[0044] Where, μ rest The second feature tensor mean, where N is the total number of points in the point cloud, and features_rest is the mean of the second feature tensor. i Let be the second feature tensor of the i-th point in the point cloud.

[0045] Obtain the formula for calculating the standard deviation of the second feature tensor, where the formula for calculating the standard deviation of the second feature tensor is:

[0046] Where, σ rest The standard deviation of the second feature tensor, N, is the total number of point clouds, and features_rest i Let μ be the second feature tensor of the i-th point in the point cloud. rest The mean of the second characteristic tensor.

[0047] Specifically, the mean of a feature tensor is the average value of each feature in the tensor, reflecting the global distribution of that feature. For image features, the mean is typically used to measure the central tendency of the feature, providing a sense of the overall image features. The standard deviation is the dispersion of each feature in the feature tensor, indicating how much the data points deviate from the mean. A larger standard deviation indicates that the feature varies more spatially, while a smaller standard deviation indicates that the feature distribution is more concentrated. Gaussian probability is used to describe the probability density function of feature data under a Gaussian distribution. Using the mean and standard deviation, the probability value of each data point under this distribution is calculated, and it is typically used to measure the distance between the data point and the mean.

[0048] Calculation formula using the first characteristic tensor mean The mean of the first feature tensor is calculated by applying the mean value to the first feature tensor. Where μ... dc The first feature tensor is the mean, N is the total number of points in the point cloud, and features_dc is the total number of points in the point cloud. i Let be the first feature tensor of the i-th point in the point cloud.

[0049] Calculation formula using the first characteristic tensor standard deviation The standard deviation of the first characteristic tensor is calculated. Where σ dc The standard deviation of the first feature tensor is denoted as , where N is the total number of points in the point cloud, and features_dc is the number of points in the point cloud. iis the first feature tensor of the i-th point in the point cloud, μ dc is the first feature tensor mean.

[0050] The second feature tensor mean is calculated using the second feature tensor mean calculation formula The second feature tensor mean is calculated using the second feature tensor mean calculation formula rest is the second feature tensor mean, N is the total number of point clouds, and features_rest i is the second feature tensor of the i-th point in the point cloud.

[0051] The second feature tensor standard deviation is calculated using the second feature tensor standard deviation calculation formula The second feature tensor standard deviation is calculated using the second feature tensor standard deviation calculation formula rest is the second feature tensor standard deviation, N is the total number of point clouds, and features_rest i is the second feature tensor of the i-th point in the point cloud, μ rest is the second feature tensor mean.

[0052] Further, the present application further comprises the following steps:

[0053] The first Gaussian probability calculation formula is obtained, wherein the first Gaussian probability calculation formula is:

[0054] wherein P(features dc ) is the first feature tensor Gaussian probability, features dc is the first feature tensor, μ dc is the first feature tensor mean, σ dc is the first feature tensor standard deviation, and ∈ is a noise level constant.

[0055] The second Gaussian probability calculation formula is obtained, wherein the second Gaussian probability calculation formula is:

[0056] wherein P(features_rest) is the second feature tensor Gaussian probability, features_rest is the second feature tensor, μ rest is the second feature tensor mean, σ rest is the second feature tensor standard deviation, and ∈ is a noise level constant.

[0057] Specifically, the first feature tensor Gaussian probability is obtained by using the first Gaussian probability calculation formula to calculate the Gaussian probability in combination with the first feature tensor mean and the first feature tensor standard deviation. The first Gaussian probability calculation formula is: wherein P(features dc ) is the first feature tensor Gaussian probability, featuresdc is the first feature tensor, μ dc is the first feature tensor mean, σ dc is the first feature tensor standard deviation, ∈ is the noise level constant.

[0058] The second Gaussian probability calculation formula is used to calculate the Gaussian probability by combining the second feature tensor mean and the second feature tensor standard deviation, and the second feature tensor Gaussian probability is obtained. The second Gaussian probability calculation formula is: wherein, P(features_rest) is the second feature tensor Gaussian probability, features_rest is the second feature tensor, μ rest is the second feature tensor mean, σ rest is the second feature tensor standard deviation, ∈ is the noise level constant.

[0059] By calculating the probability features (including matrix, standard deviation and probability density) of Gaussian distribution and integrating them into the 3DGS loss function, the feature information of sparse points is increased, the model training is optimized, and the modeling quality is improved. Unlike the classic 3DGS method, which only uses Gaussian distribution for scene geometry representation and differentiable rendering process, Gaussian distribution is used as a kind of probability statistical tool to establish a probability mapping model of feature space. This double application mechanism can provide double constraints under sparse observation conditions, not only maintaining the geometric expression ability of the original 3DGS, but also enhancing the features through probability statistics to capture the potential rules of data and enhance the modeling ability of sparse areas. The probability statistical modeling method can effectively reduce the uncertainty in the reconstruction process, and thus provide support for further inference and decision-making. By calculating the Gaussian probability of each data point, the position of the data point in its feature distribution is accurately measured, and the weights of different features are adjusted, so that more accurate and stable results are obtained in view synthesis, image reconstruction and other tasks.

[0060] S300: Introduce an elastic net regularization loss to constrain the first feature tensor and the second feature tensor, and calculate the mean of the first feature tensor Gaussian probability and the second feature tensor Gaussian probability, and substitute the calculation results into the loss function to minimize the loss function, and obtain the optimized Gaussian point set.

[0061] Further, the S300 of the present application comprises:

[0062] The elastic net regularization loss is:

[0063] wherein, λ 11 and λ l2is a regularization parameter that controls the weight of the regularization, ||features_dc||1is the result of L1 regularization of the first feature tensor, ||features_rest||1is the result of L1 regularization of the second feature tensor, ||features_dc||2is the result of L2 regularization of the first feature tensor, and ||features_rest||2is the result of L2 regularization of the second feature tensor.

[0064] Specifically, when the 3D Gaussian cannot sufficiently represent the scene, the 3D Gaussian starts an adaptive density control mechanism to increase the density. However, due to the lack of constraints, the newly generated 3D Gaussian function is prone to overfitting the training views, leading to its continuous development towards overfitting. Therefore, elastic net regularization is used for feature constraint, combining L1 and L2 regularization terms.

[0065] The L1 regularization is:

[0066]

[0067] where d1 and d2 are the sizes of the feature vectors of features_dc and features_rest, respectively, features_dc i is the first feature tensor of the i-th point in the point cloud, and features_rest i is the second feature tensor of the i-th point in the point cloud.

[0068] The L2 regularization is:

[0069]

[0070] where d1 and d2 are the sizes of the feature vectors of features_dc and features_rest, respectively, features_dc i is the first feature tensor of the i-th point in the point cloud, and features_rest i is the second feature tensor of the i-th point in the point cloud.

[0071] Finally, the L1 and L2 regularizations are combined to obtain the elastic net regularization loss as:

[0072]

[0073] where λ 11 and λ l2wherein is a regularization parameter that controls the weight of the regularization, ||features_dc||1 is the result of L1 regularization of the first feature tensor, ||features_rest||1 is the result of L1 regularization of the second feature tensor, ||features_dc||2 is the result of L2 regularization of the first feature tensor, and ||features_rest||2 is the result of L2 regularization of the second feature tensor.

[0074] L1 regularization constrains the features, and the model automatically selects important features, sets the weight of certain features to zero, helps to retain features that contribute significantly to the model, and reduces the complexity of the model. L2 regularization suppresses overfitting by constraining the weight, forces the model to maintain smoothness, and avoids the influence of certain feature weights on the result. Elastic net regularization innovatively combines the advantages of L1 and L2 regularization, can achieve feature selection and maintain model stability, and significantly enhances the generalization ability in high-dimensional feature space. This regularization strategy can adaptively balance the fitting ability and complexity when facing high-dimensional feature space. Combined with the proposed probability mapping framework, not only does it provide a robust optimization paradigm for three-dimensional scene modeling, but also enhances the modeling accuracy of sparse regions through probability statistical constraints, and constructs a general probability-regularization collaborative optimization framework. It also has good expansibility and application potential in other fields that require probability inference and model refinement.

[0075] The first feature tensor and the second feature tensor are constrained by introducing an elastic net regularization loss, and the mean values of the Gaussian probabilities of the first feature tensor and the second feature tensor are calculated. The calculated mean values are substituted into the loss function, and the model is optimized by minimizing the loss function. The Gaussian point information of the first feature tensor and the second feature tensor, including the mean value, standard deviation, and transparency of each point, is optimized. During the optimization process, the model generates new Gaussian points to improve the accuracy of the Gaussian points.

[0076] S400: Multi-scale densification and pruning of the Gaussian point set by the threshold set to generate a new view.

[0077] The threshold set includes a gradient threshold, a scale threshold, and a pruning threshold.

[0078] Further, the S400 of the present application comprises:

[0079] The densification condition discrimination formula is used to analyze the Gaussian point set one by one to obtain a densification Gaussian point set. The densification operation is performed on the densification Gaussian point set. It is judged whether the transparency of the Gaussian point after the densification processing is less than a preset transparency threshold. If yes, pruning is performed.

[0080] Further, the present application further comprises the following steps:

[0081] The densification condition judgment formula is:

[0082] wherein densify mask[i] is the i-th point in the point cloud, grad[i] is the gradient of the i-th point, grad threshold is the gradient threshold, scaling i scaling[i] is the scale of the i-th point, and scaling threshold is the scale threshold.

[0083] Specifically, a threshold group is set, including a gradient threshold, a scale threshold, and a pruning threshold. The gradient threshold is usually used for edge detection in image processing, limiting the rate of change of pixel values in the image. After setting the gradient threshold, it can be determined which points have a larger gradient and require more points for densification. The scale threshold is used to control the limitation of the scale (size) of the points in the process of point cloud densification or pruning. By setting the scale threshold, only points within a certain scale range are retained, thereby avoiding unnecessary calculations and improving efficiency. The pruning threshold is a criterion for deciding whether to delete certain points in the Gaussian point processing. If the transparency, weight, or other attributes of a point are lower than the pruning threshold, these points will be considered unimportant and will be removed.

[0084] When using dense input, the number of 3D Gaussians and rendering performance will steadily increase. One of the reasons is that the density control strategy increases the number of 3D Gaussians to meet the demand for rendering training views. However, when using sparse input, limited information leads to overfitting from a training perspective, and overfitting patterns produce a large number of floating points and incorrect geometries, resulting in the inability to further optimize training view information and ultimately leading to the failure of rendering new views. To solve this problem, a multi-scale densification and pruning strategy is improved to enable effective operation across multiple scales. The core idea of multi-scale densification is to improve the density of the model by operating at different resolutions or scales, and the pruning operation decides which points should be removed at the appropriate scale.

[0085] For the Gaussian point set, the seed point selection and seed point supplement strategy are used to accurately locate the Gaussian ellipsoid position and complete the density optimization. Seed point selection aims to select the most valuable key points, and the supplement step specifically enhances the sampling density of local areas. When there are too few seed points, the transparency of the generated Gaussians may not be sufficient, resulting in their deletion in adaptive densification. Multiple thresholds are introduced to dynamically control the density and prevent the number of Gaussian spheres from being too small. Specifically, for each training iteration, the density and accuracy of the point cloud are dynamically adjusted through multi-scale densification and pruning operations to improve training efficiency and control the complexity of the model.

[0086] The densification condition discrimination formula is used to analyze the characteristics of each point (such as position, color, transparency, etc.) according to the Gaussian point set, and to determine whether it meets the densification condition. The densification condition discrimination formula is Where densify mask[i] is the i-th point in the point cloud, and densify_mask is a binary mask used to control the densification process of the point cloud. Each element corresponds to a point, and a value of 1 indicates that the point needs to be densified, and a value of 0 indicates that it does not need to be densified, determining which points need to be densified, i.e., adding more points to improve the detailed representation of the model. The gradient of the i-th point, grad_i, represents the degree of change of the point during the training process, and a larger gradient means that the point plays a more important role in the model. The grad threshold is a gradient threshold used to determine which points have a large gradient and need more points for densification. The scaling i The scaling of the i-th point, scaling_i, is the scaling threshold used to determine which points are in a larger scale range and may need more points. For each point, check whether its gradient is greater than the threshold and whether its scale meets the requirements.

[0087] When the gradient of the point is greater than grad threshold and the scale of the point (scaling_i) is greater than scaling threshold, the point will be densified, i.e., more points will be added near the point to improve the detailed representation of the model. If the condition is met, the point is densified, otherwise it is not densified, i.e., an incremental densification threshold densify threshold is introduced, and then according to the different scale conditions and gradient conditions of the specific point cloud, when the densification threshold meets the current conditions, the point cloud is densified, and the selected points at this time are sampled to increase the density of the point cloud, so as to more finely represent the details of the model. Finally, a gradually increasing pruning threshold prune thresholds is introduced, and the transparency of the specific point is used as the judgment condition. Specifically, when the transparency of the point is lower than the threshold, the point is considered unimportant and can be pruned. Alternatively, by checking the maximum scale of the point, it is determined whether to prune. If the scale of the point is very small, pruning can be considered.

[0088] The multi-scale densification strategy is to dynamically select the Gaussian points that need to be densified according to the densification threshold and gradient threshold of the current scale; and different densification thresholds are applied in each training step to realize densification under multiple resolutions. The multi-scale pruning strategy is to determine which points should be removed based on the threshold of each scale and the transparency of the points, so that the model can effectively prune unimportant points at each scale, ensuring that unimportant points are effectively removed during training, thereby reducing the complexity of the model and improving computational efficiency. In addition, to prevent gradient explosion from causing unstable training, a gradient clipping operation is introduced. The gradient range is constrained before each optimization step to ensure that the gradient is not too large. Through the synergistic optimization of the above methods, a balance is achieved between training stability, computational efficiency and model accuracy, significantly improving the quality and training speed of point cloud modeling.

[0089] In summary, the new view synthesis method based on Gaussian probability distribution and feature regularization provided by the present application has the following beneficial effects:

[0090] By extracting feature tensors from the preprocessed input image, a first feature tensor and a second feature tensor are obtained, wherein the feature tensors represent the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the high-order spherical harmonic function coefficient of the Gaussian point; based on the mean and standard deviation of the first feature tensor and the second feature tensor, Gaussian probability calculation of the feature tensors is performed to obtain first feature tensor Gaussian probability and second feature tensor Gaussian probability; an elastic net regularization loss is introduced to constrain the first feature tensor and the second feature tensor, and the mean of the first feature tensor Gaussian probability and the second feature tensor Gaussian probability is calculated, and the calculation result is substituted into the loss function to minimize the loss function, thereby obtaining an optimized Gaussian point set; the Gaussian point set is subjected to multi-scale densification and pruning by a threshold group to generate a new view. That is, by introducing Gaussian probability mapping and feature regularization, the problem of too few Gaussian points and low transparency in a sparse scene is overcome, the modeling and view synthesis process under a sparse view angle is optimized, and the accuracy of modeling is effectively improved, especially when processing sparse data. By performing probability mapping on the input feature tensors, not only the potential laws of the data can be captured, but also the uncertainty can be quantified, thereby improving the modeling effect under a sparse view angle and the accuracy of view synthesis. By adopting the multi-scale densification and pruning strategy on the Gaussian point set by the threshold group, the position of the Gaussian ellipsoid is accurately located and density optimization is completed, thereby realizing new view generation in a sparse environment, reducing the complexity of the model, and improving computational efficiency.

[0091] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0092] Obviously, many modifications and changes can be made to the application as set forth above without departing from the spirit and scope of the application. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A novel view synthesis method based on Gaussian probability distribution and feature regularization, characterized in that, include: Feature tensors are extracted from the preprocessed input image to obtain a first feature tensor and a second feature tensor. The feature tensors represent the feature information of each Gaussian point, the first feature tensor represents the color feature of the Gaussian point, and the second feature tensor represents the higher-order spherical harmonic function coefficients of the Gaussian point. Based on the mean and standard deviation of the first and second feature tensors, Gaussian probabilities of the feature tensors are calculated to obtain the Gaussian probabilities of the first and second feature tensors. The elastic mesh regularization loss is introduced to constrain the first feature tensor and the second feature tensor. The mean of the Gaussian probability of the first feature tensor and the Gaussian probability of the second feature tensor is calculated. The calculation result is substituted into the loss function to minimize the loss function and obtain the optimized Gaussian point set. A new view is generated by multi-scale densification and pruning of the Gaussian point set using threshold groups. The regularization loss of the elastic network is: ； in, and These are regularization parameters, controlling the weights of the regularization. The result after L1 regularization of the first feature tensor. The result after L1 regularization of the second feature tensor The result after L2 regularization of the first feature tensor. The result after L2 regularization of the second feature tensor; The threshold set includes gradient threshold, scale threshold, and pruning threshold; The threshold set is used to densify and prune the point cloud of the input image, including: The Gaussian point set of the preprocessed input image is analyzed one by one using the densification condition discriminant formula to obtain the densified Gaussian point set; Perform the densification operation by traversing the set of densified Gaussian points; Determine whether the transparency of the Gaussian points after densification is less than the preset transparency threshold; if so, prune them. The formula for determining the densification condition is: ； in, Let i be the i-th point in the point cloud. Let be the gradient at the i-th point. For gradient threshold, Let i be the scale of the i-th point. This is the scale threshold.

2. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 1, characterized in that, include: Using the formulas for calculating the mean of the first feature tensor, the standard deviation of the first feature tensor, the mean of the second feature tensor, and the standard deviation of the second feature tensor, the mean and standard deviation of the first feature tensor and the second feature tensor are calculated to obtain the mean of the first feature tensor, the standard deviation of the first feature tensor, the mean of the second feature tensor, and the standard deviation of the second feature tensor. Using the first Gaussian probability calculation formula, combined with the mean and standard deviation of the first feature tensor, Gaussian probability is calculated to obtain the Gaussian probability of the first feature tensor. The Gaussian probability of the second feature tensor is obtained by using the formula for calculating the second Gaussian probability, combined with the mean of the second feature tensor and the standard deviation of the second feature tensor.

3. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 2, characterized in that, include: Obtain the formula for calculating the mean of the first feature tensor, where the formula for calculating the mean of the first feature tensor is: ； in, The mean of the first characteristic tensor. The total number of point clouds, Let be the first feature tensor of the i-th point in the point cloud; Obtain the formula for calculating the standard deviation of the first feature tensor, where the formula for calculating the standard deviation of the first feature tensor is: ； in, The standard deviation of the first characteristic tensor. The total number of point clouds, Let be the first feature tensor of the i-th point in the point cloud. The mean of the first characteristic tensor; Obtain the formula for calculating the mean of the second feature tensor, where the formula for calculating the mean of the second feature tensor is: ； in, The mean of the second characteristic tensor. The total number of point clouds, Let be the second feature tensor of the i-th point in the point cloud; Obtain the formula for calculating the standard deviation of the second feature tensor, where the formula for calculating the standard deviation of the second feature tensor is: ； in, The standard deviation of the second characteristic tensor. The total number of point clouds, Let i be the second feature tensor of the i-th point in the point cloud. The mean of the second characteristic tensor.

4. The novel view synthesis method based on Gaussian probability distribution and feature regularization as described in claim 2, characterized in that, include: Obtain the formula for calculating the first Gaussian probability, where the formula for calculating the first Gaussian probability is: ； in, Let Gaussian probability be the first feature tensor. Let be the first characteristic tensor. , The standard deviation of the first characteristic tensor. This is a noise level constant; Obtain the formula for calculating the second Gaussian probability, where the formula for calculating the second Gaussian probability is: ； in, The second feature tensor is the Gaussian probability. For the second characteristic tensor, The mean of the second characteristic tensor. The standard deviation of the second characteristic tensor. This is a noise level constant.

Images