Texture perception 3D Gaussian expansion method for sparse view reconstruction

Abstract

The invention discloses a texture perception 3D Gaussian expansion method for sparse view reconstruction. According to the method, a texture perception framework TA-GS for performing novel view synthesis by using a sparse input image is provided; a proposed texture-based Gaussian migration strategy utilizes texture strength to guide Gaussian primitives, so that fine-grained texture details are better represented under a sparse input view; the proposed depth alignment texture effectively captures local depth change and texture information, enhances geometric constraints and improves depth precision; in order to further enhance texture optimization, Phantom View regularization is introduced, a training view is enriched through interpolation, and texture regularization is applied, so that high-fidelity reconstruction of texture details and geometric structures is realized. The method is superior to the existing method in various indoor, outdoor and object-centered scenes, and has different image resolutions and input view counts.

Description

Technical Field

[0001] The present invention relates to the technical field of new view synthesis, and more particularly to a texture-aware 3D Gaussian expansion method for sparse view reconstruction. Background Art

[0002] Novel View Synthesis (NVS) is a prominent research topic in recent years and is of great significance in the fields of computer vision and graphics. Recently, 3D Gaussian expansion has emerged as a promising technique for NVS. In the case of dense input images, 3DGS represents the scene as a collection of explicit Gaussian basis points, enabling fast, high-quality and cost-effective novel view synthesis. However, in many real-world applications, such as robotics and autonomous driving, obtaining high-quality dense views is often very time-consuming, costly, or even impossible. The limited number of available views severely restricts the reconstruction capabilities of 3DGS. As the number of input training views decreases, 3DGS struggles to accurately reconstruct fine texture details, resulting in a degradation in rendering quality.

[0003] To gain deeper insight into this problem, we visualized Gaussian field point clouds for dense and sparse input views. Regions with rich texture details are more difficult to optimize, and generally require more Gaussian primitives to accurately represent and optimize these intricate details. However, when the input view becomes very sparse, the limited reference information prevents Gaussian primitives from effectively clustering in texture-rich regions, resulting in distorted or lost details.

[0004] Previous methods have attempted to alleviate these problems by enhancing geometric reconstruction or incorporating learned priors. DNGaussian utilizes hard and soft depth regularization as well as a global-local depth normalization strategy to provide geometric guidance for Gaussian primitives. Furthermore, FSGS initially employs a Gaussian unpooling strategy to densify Gaussian primitives, followed by a depth constraint to guide their geometric alignment. However, these methods fail to account for the importance of texture information in achieving high-fidelity rendering. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a texture-aware 3D Gaussian expansion method for sparse view reconstruction. This method proposes a texture-aware framework TA-GS for novel view synthesis using sparse input images; the proposed texture-based Gaussian migration strategy uses texture intensity to guide Gaussian primitives, thereby better representing fine-grained texture details under sparse input views; the proposed depth-aligned texture effectively captures local depth variations and texture information, strengthens geometric constraints and improves depth accuracy; to further enhance texture optimization, PhantomView regularization is introduced, which enriches the training views by interpolation and applies texture regularization, thereby achieving high-fidelity reconstruction of texture details and geometric structures. Extensive experiments show that the present invention outperforms existing methods in various indoor, outdoor, and object-centric scenarios with different image resolutions and input view counts.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A texture-aware 3D Gaussian expansion method for sparse view reconstruction includes the following steps:

[0008] Step S1, extracting texture intensity from the input image to identify texture-rich areas;

[0009] Step S2, during the densification process, the high-resolution primitives with low opacity are migrated to the area with high texture intensity to alleviate the texture distortion caused by the sparse view;

[0010] Step S3, associating depth gradients with texture information, and incorporating texture information into depth estimation using a gradient-based depth alignment strategy to strengthen constraints on geometric structures;

[0011] In step S4, texture regularization is applied to the phantom views, which enables the model to generate high-quality texture representations under sparse input conditions by optimizing texture details from more viewpoints.

[0012] Furthermore, step S1 includes the following steps:

[0013] Step S101: For the input image, calculate the difference between adjacent pixel values along the width W to obtain the horizontal gradient G x ; Calculate the difference between adjacent pixel values along the height H to obtain the vertical gradient G y ;

[0014]

[0015] In formula (1), C(i,j) is the pixel value of pixel (i,j); i = 1, 2, ..., H-1; j = 1, 2, ..., W-1;

[0016] Step S102: The horizontal gradient G x and vertical gradient G y The absolute values of are added and normalized to obtain the color gradient map G color ;

[0017] Step S103: Add a texture intensity attribute to each Gaussian basis element, and render the texture intensity attribute into a 2D image using a differentiable rendering technique. The calculation formula is as follows:

[0018]

[0019] In formula (2), (TI) i represents the texture intensity of the i-th Gaussian primitive; G TI represents the 2D TI map after rasterization; α i represents the opacity of the i-th Gaussian primitive; α $ The opacity of the j-th Gaussian basis element;

[0020] Step S104: Use the Pearson correlation coefficient loss function to optimize the texture intensity attribute of each Gaussian basis element so that it is consistent with the color gradient map G color Alignment, the calculation formula is as follows:

[0021]

[0022] Furthermore, step S2 includes the following steps:

[0023] Step S201 , identifying and retaining Gaussian primitives with opacity lower than a threshold, marking them as fuzzy Gaussian primitives, and adding a texture intensity attribute;

[0024] Step S202 , based on the texture intensity attribute probability distribution, migrating the fuzzy Gaussian primitive to a Gaussian primitive position with a higher texture intensity value;

[0025] Step S203 : adjusting the properties of the migrated Gaussian primitives to maintain visual continuity and prevent artifacts.

[0026] Furthermore, in step S203, the position, opacity, covariance and texture intensity of the migrated Gaussian primitives are adjusted to prevent artifacts. The process is as follows:

[0027]

[0028] In formula (4), target represents the Gaussian parameter with high texture intensity; updated represents the parameter after migration; N represents the total number of Gaussian primitives migrated to this position; μ represents the position of the Gaussian primitive; o represents the opacity of the Gaussian primitive; TI represents the texture intensity of the Gaussian primitive; Σ represents the covariance of the Gaussian primitive.

[0029] Furthermore, step S3 includes the following steps:

[0030] Step S301, obtaining rendering depth gradient and monocular depth gradient;

[0031] Step S302: Use the Pearson correlation coefficient loss function to calculate the loss between the rendered depth gradient and the monocular depth gradient. By minimizing the Pearson correlation coefficient loss, the rendered depth gradient and the monocular depth gradient are kept consistent in terms of local depth changes. The formula is as follows:

[0032]

[0033] In formula (5), GD render Gradient map representing rendered depth; GD mono Gradient map representing monocular depth.

[0034] Furthermore, the depth gradient of the rendering is generated: the horizontal gradient G of the rendering depth gradient is calculated x and vertical gradient G y , add the absolute values of the two and normalize them;

[0035] Generation of monocular depth gradient: Calculate the horizontal gradient G of the monocular depth gradient x and vertical gradient G y , which is obtained by adding the absolute values of the two and then normalizing them.

[0036] Furthermore, step S4 includes the following steps:

[0037] Step S401 , generating a phantom view by interpolation from the two closest training views;

[0038] Step S402: Combine the texture intensity loss and the depth gradient loss to calculate the phantom view loss. The formula is as follows:

[0039]

[0040] In formula (6), λ >1 and λ >J To balance the weight;

[0041] In step S403, the photometric loss, structural similarity loss, texture intensity loss, depth gradient loss, and phantom view loss are combined to form a final loss function L, and the loss function L is minimized. The formula is as follows:

[0042]

[0043] In formula (7), I represents the real image; Represents rendered image; L1 represents luminosity loss; LXXIY represents the structural similarity loss; λ1, λ J and λ Z As a parameter.

[0044] Furthermore, the interpolation method is based on the average of the camera position and rotation, and the formula is as follows:

[0045]

[0046] In formula (8), t∈P represents the camera position; q represents the quaternion; ε represents the random noise vector when sampling the phantom view, obeying The normal distribution of δ is the discrete degree of random noise ε.

[0047] The beneficial effects of the present invention are:

[0048] 1. This paper proposes a new framework, TA-GS, to address the reconstruction challenge of 3DGS under sparse views from a texture perspective. By introducing a texture-based Gaussian migration strategy to enhance the concentration of Gaussian primitives in texture areas, it ensures accurate representation of fine textures in the scene. At the same time, it further imposes constraints on geometry and introduces deep texture alignment, which introduces guidance from depth estimation. To alleviate the overfitting problem, Phantom View Regularization is adopted to utilize the additional texture information on the Phantom View.

[0049] 2. Through an innovative texture perception mechanism and geometric constraint strategy, this paper achieves a triple breakthrough in texture detail enhancement, geometric accuracy improvement, and training efficiency optimization in sparse view scenarios, providing a new theoretical framework and practical tools for 3D Gaussian rendering technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 4 is a flow chart of a texture-aware 3D Gaussian expansion method for sparse view reconstruction in this embodiment. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Example: A texture-aware 3D Gaussian expansion method for sparse view reconstruction, such as Figure 1 As shown, the method includes the following steps:

[0053] Step S1: extracting texture intensity from the input image to identify texture-rich areas; the amplitude of color changes (color gradients) in the input image reflects its high-frequency information, and larger gradients correspond to areas with richer textures;

[0054] In this embodiment, the texture intensity is extracted by a gradient extractor (GE) module; specifically:

[0055] Step S101: For the input image, calculate the difference between adjacent pixel values along the width W (horizontal direction) to obtain the horizontal gradient G x ; Calculate the difference between adjacent pixel values along the height H (vertical direction) to obtain the vertical gradient G y ;

[0056]

[0057] In formula (1), C(i,j) is the pixel value of pixel (i,j); i = 1, 2, ..., H-1; j = 1, 2, ..., W-1;

[0058] Step S102: The horizontal gradient G x and vertical gradient G y The absolute values of are added and normalized to obtain the color gradient map G color ;

[0059] In step S103, in order to embed these texture details into the 3D Gaussian primitives, in addition to retaining all the original Gaussian attributes, a texture intensity (TI) attribute is introduced to represent the texture information. The texture intensity attribute is then rendered into a 2D image using a differentiable rendering technique. The calculation formula is as follows:

[0060]

[0061] In formula (2), (TI) i represents the texture intensity of the i-th Gaussian primitive; G TI represents the 2D TI map after rasterization; α i represents the opacity of the i-th Gaussian primitive; α $ The opacity of the j-th Gaussian basis element;

[0062] In step S104, for continuous-valued image features, the traditional L1 loss focuses primarily on the absolute differences between pixels, while ignoring the relative trends of feature changes. In contrast, as a normalization metric, the Pearson correlation coefficient can better preserve the relative relationships between images. The specific process is outlined as follows:

[0063]

[0064] Where X and Y represent two different distributions, and E represents the mathematical expectation.

[0065] Therefore, in order to optimize the texture intensity attribute introduced by each Gaussian primitive, the texture intensity attribute and the color gradient map G are calculated. color The Pearson correlation coefficient loss between the texture intensity map and the color gradient map G color The linear correlation between them facilitates a more precise alignment of the texture intensity attributes, which is calculated as follows:

[0066]

[0067] In step S2, during the densification process, high-resolution primitives with low opacity are migrated to regions with high texture intensity, thereby alleviating texture distortion caused by sparse views.

[0068] Specifically, the following steps are included:

[0069] Step S201 , identifying and retaining Gaussian primitives with opacity below a certain threshold, marking them as fuzzy Gaussian primitives, and adding a texture intensity attribute;

[0070] Step S202 , based on the texture intensity attribute probability distribution, migrating the fuzzy Gaussian primitive to a Gaussian primitive position with a higher texture intensity value;

[0071] Step S203 : adjusting the properties of the migrated Gaussian primitives to maintain visual continuity and prevent artifacts.

[0072] In traditional 3D Gaussian algorithms, Gaussian primitives with opacity below a certain threshold are discarded during the densification phase, potentially leading to loss of detail in textured areas. Rather than removing these blurry Gaussian primitives, this embodiment utilizes migration to enhance the concentration of Gaussian primitives in textured areas, resulting in a more accurate and detailed representation of the scene.

[0073] To ensure that the rasterization results of the Gaussian primitives are consistent before and after migration, the properties of the Gaussian primitives, such as position, opacity, covariance, and texture intensity, are adjusted accordingly. These adjustments help maintain visual continuity and prevent artifacts caused by sudden changes in Gaussian properties during migration. The process is as follows:

[0074]

[0075] In formula (4), target represents the Gaussian parameter with high texture intensity; updated represents the parameter after migration; N represents the total number of Gaussian primitives migrated to this position; μ represents the position of the Gaussian primitive; o represents the opacity of the Gaussian primitive; TI represents the texture intensity of the Gaussian primitive; Σ represents the covariance of the Gaussian primitive.

[0076] This migration strategy is performed every 100 optimization iterations and follows the traditional 3D Gaussian densification strategy.

[0077] In step S3, the depth gradient is associated with the texture information, and the gradient-based depth alignment strategy is used to incorporate the texture information into the depth estimation to strengthen the constraints on the geometric structure.

[0078] Due to the insufficient constraints on the input view, Gaussian primitives struggle to converge to the correct position to accurately represent the scene geometry. To enforce geometric constraints, a traditional approach uses an image-level Pearson correlation loss between the rendered depth map and the monocular depth map estimated by the DPT model. However, directly applying this loss to the depth map channel fails to impose sufficient constraints on local regions, resulting in the misalignment of fine geometric details.

[0079] To solve this misalignment problem, this embodiment proposes a texture-aware depth alignment method. Specifically:

[0080] Step S301: Utilize a gradient extractor (GE) module to obtain rendered depth gradients and monocular depth gradients; rendered depth gradients and monocular depth gradients can effectively capture local depth variations and depth texture information, which are crucial for detailed geometric representation;

[0081] Wherein, based on the formula (1) in step S101, the horizontal gradient G of the rendered depth gradient is calculated x and vertical gradient G y , add the absolute values of the two and normalize them to get the rendering depth gradient; calculate the horizontal gradient G of the monocular depth gradient x and vertical gradient G y , the absolute values of the two are added and normalized to obtain the monocular depth gradient.

[0082] Step S302: Use the Pearson correlation coefficient loss function to calculate the loss between the rendered depth gradient and the monocular depth gradient. By minimizing the Pearson correlation coefficient loss, the rendered depth gradient and the monocular depth gradient are kept consistent in terms of local depth changes. The formula is as follows:

[0083]

[0084] In formula (5), GD render Gradient map representing rendered depth; GD mono Gradient map representing monocular depth.

[0085] In step S4, texture regularization is applied to phantom views to optimize texture details from more viewpoints, so that the model can generate high-quality texture representations under sparse input conditions. Specifically:

[0086] In step S401, in order to alleviate the inherent overfitting problem caused by sparse training views, a ghost view is generated by interpolation from the two closest training views in Euclidean space. The interpolation method is based on the average of the camera position and rotation, and the formula is as follows:

[0087]

[0088] In formula (8), t∈P represents the camera position; q represents the quaternion; ε represents the random noise vector when sampling the phantom view, obeying The normal distribution of , δ is the discrete degree of random noise ε;

[0089] Step S402: Combine the texture intensity loss and the depth gradient loss to calculate the phantom view loss. The formula is as follows:

[0090]

[0091] In formula (6), λ >1 and λ >J To balance the weights, they are set to 0.2 and 0.1 respectively;

[0092] By incorporating texture intensity loss and depth gradient loss into other viewpoints, richer texture information can be extracted and the geometric structure can be guided more effectively;

[0093] Step S403: In step S403, the photometric loss, structural similarity loss, texture intensity loss, depth gradient loss, and phantom view loss are combined to form a final loss function L, and the loss function L is minimized. The formula is as follows:

[0094]

[0095] In formula (7), I represents the real image; Represents rendered image; L1 represents luminosity loss; L XXIY represents the structural similarity loss; λ1, λ J and λ Z As a parameter.

[0096] Through the above optimizations, the distribution and properties of Gaussian primitives are gradually adjusted to minimize various losses, thereby achieving high-quality synthesis of new views.

[0097] The following experiments are conducted on four experimental sets:

[0098] LLFF dataset: This dataset includes 8 real-world scenes with detailed textures. Every 8 images are selected as the test set, and sparse views are uniformly sampled from the remaining images for training. Following FSGS, 3, 6, and 9 input views are evaluated at 1 / 8 resolution for comparison. In addition, 3 views at 1 / 4 and 1 / 8 resolution are used to train all methods.

[0099] Mip-NeRF360 dataset: This dataset contains nine scenes, each consisting of a complex central object or region surrounded by a complex background. Following FSGS, experiments were conducted using seven common scenes from this dataset, training all methods with 24 images at view factors of 4 and 8. Furthermore, 12 and 24 input views were tested at 1 / 4 resolution for comparison. Test images were selected according to the same convention as for the LLFF dataset described above.

[0100] DTU dataset: This dataset provides a comprehensive collection of object-level data. The same 15 scenes, each containing 3 views, are used for training in the experiments. Object masks are also utilized during evaluation to focus on the target object and remove background noise, as done in RegNeRF.

[0101] Blender Dataset: This dataset contains eight synthetic scenes generated using Blender. It uses 8 images for training and 25 images for testing, with a resolution of 400 × 400. This dataset demonstrates the accuracy of capturing details in object-centered scenes.

[0102] The mean and covariance of the generated images are used to calculate the values of the evaluation indicators FID and IS to quantify the performance of the model.

[0103] The quantitative results of the LLFF dataset are shown in Tables 1 and 2. The method proposed in this example is significantly better than other methods, with significant improvements in PSNR, SSIM, and LPIPS indicators.

[0104]

[0105] Table 1

[0106]

[0107] Table 2

[0108] The quantitative results of the Mip-NeRF360 dataset are shown in Tables 3 and 4. The method proposed in this example shows excellent performance in terms of PSNR, SSIM, and LPIPS indicators under different image resolutions and different numbers of input views.

[0109]

[0110]

[0111] Table 3

[0112]

[0113] Table 4

[0114] The quantitative results of the DTU dataset are shown in Table 5. The method proposed in this example achieved the best results in SSIM.

[0115]

[0116] Table 5

[0117] The quantitative results on the Blender dataset with an 8-view sparse input setting are shown in Table 6. The method proposed in this example also outperforms other methods in synthesizing scenes.

[0118] Methods PSNR SSIM LPIPS RegNeRF 23.86 0.852 0.105 SparseNeRF 24.04 0.876 0.113 3DGS 23.49 0.881 0.096 DNGaussian 24.16 0.887 0.089 FSGS 24.25 0.889 0.094 Ours 24.71 0.895 0.090

[0119] Table 6

[0120] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A texture-aware 3D Gaussian expansion method for sparse view reconstruction, characterized in that The steps include: Step S1, extracting texture intensity from the input image to identify texture-rich areas; Step S2, during the densification process, the high-resolution primitives with low opacity are migrated to the area with high texture intensity to alleviate the texture distortion caused by the sparse view; Step S3, associating depth gradients with texture information, and incorporating texture information into depth estimation using a gradient-based depth alignment strategy to strengthen constraints on geometric structures; In step S4, texture regularization is applied to the phantom views, which enables the model to generate high-quality texture representations under sparse input conditions by optimizing texture details from more viewpoints.

2. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 1, characterized in that: Step S1 includes the following steps: Step S101: For the input image, calculate the difference between adjacent pixel values along the width W to obtain the horizontal gradient G x ; Calculate the difference between adjacent pixel values along the height H to obtain the vertical gradient G y ; In formula (1), C(i,j) is the pixel value of pixel (i,j); i = 1, 2, ..., H-1; j = 1, 2, ..., W-1; Step S102: The horizontal gradient G x and vertical gradient G y The absolute values of are added and normalized to obtain the color gradient map G color ; Step S103: Add a texture intensity attribute to each Gaussian basis element, and render the texture intensity attribute into a 2D image using a differentiable rendering technique. The calculation formula is as follows: In formula (2), (TI) i represents the texture intensity of the i-th Gaussian primitive; G TI represents the 2DTI map after rasterization; α i represents the opacity of the i-th Gaussian primitive; α $ The opacity of the j-th Gaussian basis element; Step S104: Use the Pearson correlation coefficient loss function to optimize the texture intensity attribute of each Gaussian basis element so that it is consistent with the color gradient map G color Alignment, the calculation formula is as follows:

3. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 2, characterized in that: Step S2 includes the following steps: Step S201 , identifying and retaining Gaussian primitives with opacity lower than a threshold, marking them as fuzzy Gaussian primitives, and adding a texture intensity attribute; Step S202 , based on the texture intensity attribute probability distribution, migrating the fuzzy Gaussian primitive to a Gaussian primitive position with a higher texture intensity value; Step S203 : adjusting the properties of the migrated Gaussian primitives to maintain visual continuity and prevent artifacts.

4. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 3, characterized in that: In step S203, the position, opacity, covariance and texture intensity of the migrated Gaussian primitives are adjusted to prevent artifacts. The process is as follows: In formula (4), target represents the Gaussian parameter with high texture intensity; updated represents the parameter after migration; N represents the total number of Gaussian primitives migrated to this position; μ represents the position of the Gaussian primitive; o represents the opacity of the Gaussian primitive; TI represents the texture intensity of the Gaussian primitive; Σ represents the covariance of the Gaussian primitive.

5. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 3, characterized in that: Step S3 includes the following steps: Step S301, obtaining rendering depth gradient and monocular depth gradient; Step S302: Use the Pearson correlation coefficient loss function to calculate the loss between the rendered depth gradient and the monocular depth gradient. By minimizing the Pearson correlation coefficient loss, the rendered depth gradient and the monocular depth gradient are kept consistent in terms of local depth changes. The formula is as follows: In formula (5), GD render Gradient map representing rendered depth; GD mono A gradient map representing monocular depth.

6. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 5, characterized in that: Generation of rendering depth gradient: Calculate the horizontal gradient G of the rendering depth gradient x and vertical gradient G y , add the absolute values of the two and normalize them; Generation of monocular depth gradient: Calculate the horizontal gradient G of the monocular depth gradient x and vertical gradient G y , which is obtained by adding the absolute values of the two and then normalizing them.

7. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 5, characterized in that: Step S4 includes the following steps: Step S401 , generating a phantom view by interpolation from the two closest training views; Step S402: Combine the texture intensity loss and the depth gradient loss to calculate the phantom view loss. The formula is as follows: In formula (6), λ >1 and λ >J To balance the weight; In step S403, the photometric loss, structural similarity loss, texture intensity loss, depth gradient loss, and phantom view loss are combined to form a final loss function L, and the loss function L is minimized. The formula is as follows: In formula (7), I represents the real image; Represents rendered image; L1 represents luminosity loss; L XXIY represents the structural similarity loss; λ1, λ J and λ Z As a parameter.

8. The texture-aware 3D Gaussian expansion method for sparse view reconstruction according to claim 7, characterized in that: The interpolation method is based on the average of the camera position and rotation, and the formula is as follows: In formula (8), t∈P represents the camera position; q represents the quaternion; ε represents the random noise vector when sampling the phantom view, obeying The normal distribution of δ is the discrete degree of random noise ε.

Images