Attitude-limitation-free Gaussian sputtering method for single-view 3D photography

Abstract

The invention provides an attitude-limitation-free Gaussian sputtering method for single-view 3D photography, and the method comprises the steps: constructing an attention skipping attitude estimation network SPENet which comprises a large-kernel expansion convolution module LDC and an attention skipping feature interaction module SAFI, and achieving the unsupervised precise camera attitude estimation; a t distribution dual space Gaussian embedding model t-DSGE is constructed, Gaussian representation is expanded to a space and a visual attribute space so as to improve local detail presentation and scene coherence, and KL divergence loss is combined so as to maintain the coherence of the local detail presentation and the scene coherence. According to the method, unsupervised camera pose estimation is carried out by using a large-kernel expansion convolution module LDC and a skipping attention feature interaction module SAFI, the initialization process of a training scene is accelerated on the basis of ensuring the camera pose estimation precision, and the expression ability of 3DGS is enhanced by using a t-distribution dual space Gaussian embedding model t-DSGE, so that the accuracy of camera pose estimation is improved. And the precision of synthesizing the new view by the three-dimensional Gaussian sputtering method under the sparse pose-free view is improved.

Description

Technical Field

[0001] The present invention relates to the field of immersive media, namely new perspective synthesis, and in particular to a method for single-view Figure 3 A pose-free Gaussian sputtering method based on SPENet and t-DSGE for 3D photography. Background Art

[0002] Novel View Synthesis (NVS) is a computer vision method that generates unseen views of a scene based on a sequence of input views, allowing for 3D interaction with the image. It has long been a fundamental problem in computer vision research. 3D photography, a classic NVS task using a single view as input, provides a novel method for capturing and reproducing visual perception, significantly enhancing the immersive experience within a certain range of viewpoints. In recent years, the field of 3D photography has attracted the attention of many researchers.

[0003] FreeNeRF, proposed by Yang J et al., recognizes the importance of frequency in NeRF training, adjusts the range of input frequencies, and imposes a penalty on the density field near the camera, achieving performance comparable to other complex methods with few samples. Niemeyer M et al. introduced regularization for the geometry of unknown viewpoints and the appearance of rendered patches in RegNeRF, which can mitigate errors in estimating scene geometry in sparse input scenes and artifacts caused by divergent behavior in the early stages of training. Zhu Z et al. proposed a real-time few-shot framework, FSGS, based on 3DGS. It uses an efficient Gaussian inverse pooling technique to fill the gaps presented by the sparse initial point set, enhances adaptive density control, and can achieve real-time and realistic view synthesis under few-view conditions.

[0004] Patent CN202510138523.0 discloses a single-image panoramic 3D instance restoration method. This method utilizes multi-level foreground and background segmentation and occlusion culling techniques to achieve panoramic 3D reconstruction of indoor scenes in occluded environments. This method automatically removes occluded areas and produces a deoccluded image. Patent CN202411708139.1 proposes a new perspective synthesis method based on sparse views, embedding depth information into point clouds and leveraging the extreme point representation capabilities of 3DGS to achieve accurate geometric modeling without the need for complex manual regularization.

[0005] However, as mentioned in the above studies, most existing 3D photography methods heavily rely on accurate camera poses. In practical applications, obtaining accurate camera poses is very difficult, and users usually expect to generate 3D photography views from only a single image, in which case the camera pose is usually unknown.

[0006] Therefore, in order to obtain single-image 3D photography without pose restrictions, Khan N et al. proposed generating tiled multi-plane images (TMPI) with adaptive depth planes within the MPI block. This method can better simulate complex appearance effects, anti-aliased depth errors, and synthesize soft edges, while reducing the redundancy of the MPI height and reducing the computational complexity. Shih ML et al. created a complete layered depth image representation through context-aware color and depth restoration, which can efficiently render 3D images with motion parallax using standard graphics engines. However, the above studies still have problems such as unnatural filling of missing areas and discontinuities in the rendering results. At the same time, 3D photography also faces the challenge of extreme sparsity of input data, which brings many difficulties to accurately rendering fine local details and achieving scene consistency representation.

[0007] In recent years, 3D Guassian Splatting (3DGS) has gained recognition for its effectiveness in constructing 3D scenes. Its fundamental approach is to estimate an explicit sparse point cloud from a set of multi-view images using structure from motion (SfM). By using 3D Gaussian Splatting to represent the complex shape and visual features of a scene, it preserves the desirable properties of a continuous volumetric radiation field while avoiding unnecessary computation in empty space.

[0008] Invention patent CN 119006678 A discloses a 3D Gaussian sputtering optimization method for pose-free input. It deeply integrates the camera pose estimation method with the scene reconstruction process to construct an end-to-end optimization framework. By jointly optimizing the camera pose and scene structure, it achieves more accurate and efficient model training. While ensuring the accuracy of camera pose estimation, it accelerates the initialization process of the training scene and improves the accuracy of synthesizing new views using the 3D Gaussian sputtering method in sparse pose-free views. However, its use of a light prediction model to predict the distribution of light beams and calculate the camera pose based on the distribution characteristics of the light beam form has significant limitations, and the scene coherence of the synthesized new views needs to be improved. Summary of the Invention

[0009] The purpose of the present invention is to provide a Gaussian sputtering method that can achieve unsupervised and accurate camera pose estimation from a single image, enhance the representation capability of 3DGS, and improve the presentation of local details and scene coherence.

[0010] The technical solution to achieve the purpose of the present invention is: a method for single-view Figure 3 The method of Gaussian sputtering without attitude restriction for 3D photography includes the following steps:

[0011] Step 1: Construct a skip-attention pose estimation network SPENet, which includes a large-core dilated convolution module LDC and a skip-attention feature interaction module SAFI;

[0012] Step 2: Downsample the input RGB and depth images through the convolutional backbone network and send them to the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI to generate a multi-scale feature map;

[0013] Step 3: Bilinearly upsample the spatial dimensions and integrate the features from the three stages of the encoder through convolutional layers. A prediction head is set after each upsampling block to generate multi-scale inverse feature maps.

[0014] Step 4: Upsample and integrate the multi-scale inverse feature maps scale by scale to obtain a global feature map;

[0015] Step 5: The global feature map passes through three convolution layers consisting of three adaptively sized convolution kernels to output a rough camera pose matrix.

[0016] Step 6: Perform Gram-Schmidt orthogonalization and determinant adjustment on the rough camera pose matrix to obtain an accurate camera pose estimate;

[0017] Step 7: Construct the t-distribution dual space Gaussian embedding model t-DSGE;

[0018] Step 8: Calculate two different Gaussian G in the scene in the spatial position space and visual attribute space respectively. i and G j similarities between;

[0019] Step 9: Use KL divergence loss to strengthen the consistency between the spatial and visual spaces;

[0020] Step 10: Optimize the framework parameters using the pose-free input image.

[0021] Step 11: Use the trained t-distribution dual space Gaussian embedding model t-DSGE to enhance the representation of local details, rendering smoothness, and global consistency of the 3D scene.

[0022] Furthermore, the skip attention pose estimation network SPENet described in step 1 is constructed, including the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI, as follows:

[0023] Based on the 3D Gaussian sputtering framework, we construct a skip attention pose estimation network SPENet, using the horizontal field of view FoV x , vertical field of view FoV y, the rotation matrix R and the translation matrix t represent all the camera pose parameters used in the 3D Gaussian sputtering; since there may be multiple sets of camera parameters corresponding to a single image, a pre-trained dense prediction transformer DPT is used to estimate the depth map of the input single image. Under the depth map constraint, the unique corresponding camera parameters are obtained, where DPT is trained on the MIX6 dataset containing 1.4 million pairs of image-depth maps;

[0024] The large kernel dilated convolution module (LDC) uses a large kernel dilated convolution layer to extract features. By stacking LDC models with different dilation rates r, the model aggregates sufficient multi-scale context.

[0025] Given an input feature X of dimension H×W×C, the output produced by the LDC module is for:

[0026]

[0027] Among them DDWConv r (·) is an 11×11 dilated convolution with dilation rate r, BN is a batch normalization layer, Ln is a point-wise convolution layer, and Ln G is a Ln layer with a GELU activation function; in each stage, the LDC module is repeated three times, with r set to 1, 2, and 3 respectively;

[0028] The skip attention feature interaction module SAFI uses the self-similarity between pixel queries and corresponding encoder feature maps to effectively fuse global-local features. The SAFI module is assumed to have an input feature map X with a dimension of H×W×C. First, X is projected onto a query Q=XW of the same dimension. q , key K = XW k Sum V = XW v , where W q 、W k and W v is the weight matrix; for a pixel query q at i i , processing all the keys in the matrix K is computationally infeasible, so we limit the attention range to the local window and divide the Q, K and V matrices into W×W windows, where W is set to 7; let Q w , K w and V w represent the query, key, and value corresponding to the pixels in window w, respectively, then the output is for:

[0029]

[0030] Where B is a number of size w 2 ×w 2The learnable bias represents the relative position embedding of each query and key pair; the attention is calculated for each window w, and then the window is rearranged to restore the corresponding spatial position in Q, enhancing the nonlinearity of the feature:

[0031]

[0032] Where LN represents the layer normalization operation.

[0033] Furthermore, as described in step 2, the input RGB and depth images are downsampled through the convolutional backbone network and sent to the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI to generate a multi-scale feature map, as follows:

[0034] Step 2.1: Pass the input RGB and depth images of size H×W×3 through the convolutional backbone network, downsample the image using 3×3 convolution, and then apply two additional 3×3 convolutions with a stride of 1 for local feature extraction, generating a network of size Feature map of

[0035] Step 2.2: Concatenate the feature map with the average pooled input image, and then further downsample the feature map using a 3×3 convolution with a stride of 2 to generate a feature map of size The feature map is sent to the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI;

[0036] Step 2.3, set the size to The feature map is connected to the average pooled input image and downsampled in the third stage to generate a size of The feature map of is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI, and then the size of The feature map is connected to the average pooled input image and downsampled in the fourth stage to generate a size of The multi-scale feature map is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI.

[0037] Furthermore, the global feature map described in step 5 is passed through three convolution layers consisting of three adaptively sized convolution kernels to output a rough camera pose matrix, as follows:

[0038] The global feature map passes through three convolutional layers consisting of three adaptively sized convolutional kernels, ensuring that the SPENet output is a coarse camera pose matrix P with a single channel and 3×5 dimensions.

[0039] Furthermore, the rough camera pose matrix described in step 6 is subjected to Gram-Schmidt orthogonalization and determinant adjustment to obtain an accurate camera pose estimate, as follows:

[0040] Perform Gram-Schmidt orthogonalization and determinant adjustment on the first three columns of matrix P to obtain the rotation matrix R. Ensure that R satisfies the properties of the rotation matrix. The third column of P is the t matrix, and the last column contains FoV x , FoV y and a matrix of useless values.

[0041] Furthermore, the construction of the t-distribution dual space Gaussian embedding model t-DSGE described in step 7 is as follows:

[0042] In 3D Gaussian sputtering, the t-SNE principle is extended to represent Gaussians in two different spaces: the spatial position space and the visual attribute space. The spatial position space is defined by the 3D coordinates of each Gaussian, represented by the position vector and the covariance matrix Represents, defining the position and spatial distribution of each Gaussian in the scene; the visual attribute space represents the visual attributes of each Gaussian, including the opacity logarithm and the appearance features encoded by the spherical harmonic SH function of degree D; through the loss function based on KL divergence, it further ensures that neighboring Gaussians have highly similar appearance and opacity, enhances the representation ability of 3DGS, and improves the rendering ability of local details.

[0043] Furthermore, in step 8, in the spatial position space, the Gaussian G is calculated. i and G j The similarities between them are:

[0044] set up Represents a set of N Gaussians in the scene, where each Gaussian G i By position vector Covariance matrix Opacity Logarithm and color features To define, where the color feature is encoded by the spherical harmonic function of degree D, then in the spatial position space, the Gaussian G i and G j The calculation formula for the similarity between them is:

[0045]

[0046] in, is Gaussian G i Inverse of the covariance matrix.

[0047] Furthermore, in step 8, in the visual attribute space, the Gaussian G is calculated. i and G j The similarity between them is:

[0048]

[0049] where v i =[o i ,c i ] represents Gaussian G i visual properties.

[0050] Furthermore, the KL divergence loss described in step 9 is used to strengthen the consistency between the spatial and visual spaces. The formula is:

[0051] .

[0052] Furthermore, in step 10, the loss function for optimizing the framework parameters using the pose-free input image includes the following parts:

[0053]

[0054] in, For the photometric reconstruction loss, the l1-norm difference between the input image and the rendered image from the same viewpoint is evaluated:

[0055]

[0056] Where C represents the predicted image, Corresponding to the real image;

[0057] For the depth-related loss, the relationship between the rendered depth and the predicted depth is evaluated, and a relaxed relative depth loss based on Pearson correlation is adopted as follows:

[0058]

[0059] Where D is the predicted depth;

[0060] is the logarithmic probability gradient of the denoising diffusion model DDM:

[0061]

[0062] Where p is provided by DDM, given a training view In the case of , the posterior probability of the density and color fields.

[0063] Compared with the existing technology, the present invention has the following significant advantages: (1) It adopts the skip attention pose estimation network SPENet, and performs unsupervised camera pose estimation by multi-scale feature aggregation under the 3DGS framework through the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI. On the basis of ensuring the accuracy of camera pose estimation, it speeds up the initialization process of the training scene; (2) It adopts the dual space representation method t-DSGE, and extends the t-SNE principle to 3D Gaussian sputtering. By embedding Gaussians in space and visual attribute space and connecting them through KL divergence constraints, it enhances the representation ability of 3DGS, improves the rendering ability of local details, and improves the accuracy of synthesizing new views by the three-dimensional Gaussian sputtering method under sparse pose-free views. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 The present invention is a method for single vision Figure 3 Schematic diagram of the process of the attitude-free Gaussian sputtering method for 3D photography.

[0065] Figure 2 The following is the workflow and schematic diagram of SPENet in the present invention.

[0066] Figure 3 Schematic diagram of the structure of the LDC module and SAFI module in the present invention.

[0067] Figure 4 This is a comparison chart of the 3D photographic rendering results of an embodiment of the present invention on the Tanks and Temples dataset with similar methods.

[0068] Figure 5 This is a comparison chart of the 3D photography rendering results of an embodiment of the present invention on the Spaces dataset with similar methods. DETAILED DESCRIPTION

[0069] The following combination Figures 1 to 5 The present invention is further described in detail with reference to the accompanying drawings and specific embodiments.

[0070] like Figure 1 As shown, the present invention is a method for single vision Figure 3 The method of Gaussian sputtering without attitude restriction for 3D photography includes the following steps:

[0071] Step 1: Construct the skip attention pose estimation network SPENet, which includes the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI, as follows:

[0072] Based on the 3D Gaussian sputtering framework, we construct a skip attention pose estimation network SPENet, using the horizontal field of view FoV x , vertical field of view FoV y, the rotation matrix R and the translation matrix t represent all the camera pose parameters used in 3D Gaussian sputtering; since there may be multiple sets of camera parameters corresponding to a single image, the pre-trained dense prediction transformer DPT is used to estimate the depth map of the input single image. Under the constraint of the depth map, the unique corresponding camera parameters can be obtained. The DPT is trained on the MIX6 dataset containing 1.4 million pairs of image-depth maps; the skip attention pose estimation network SPENet can perform unsupervised camera pose estimation through multi-scale feature aggregation under the 3DGS framework to obtain the accurate camera pose of a single image input;

[0073] like Figure 3 As shown in the figure, the large kernel dilated convolution module LDC uses a large kernel dilated convolution layer to extract features, which can understand the overall scene from a broader perspective. The use of dilated convolution can enable the model to improve its receptive field while maintaining a fixed output feature map size. By stacking LDC models with different dilation rates r, the model can aggregate sufficient multi-scale context;

[0074] Given an input feature X of dimension H×W×C, the output produced by the LDC module is for:

[0075]

[0076] Among them DDWConv r (·) is an 11×11 dilated convolution with dilation rate r, BN is a batch normalization layer, Ln is a point-wise convolution layer, and Ln G is a Ln layer with a GELU activation function; in each stage, the LDC module is repeated three times, with r set to 1, 2, and 3 respectively;

[0077] like Figure 3 As shown in the figure, the skip attention feature interaction module SAFI uses the self-similarity between the pixel query and the corresponding encoder feature map to effectively fuse global-local features. The SAFI module is assumed to have an input feature map X with a dimension of H×W×C, which is first projected onto a query Q=XW of the same dimension. q , key K = XW k Sum V = XW v , where W q 、W k and W v is the weight matrix; for a pixel query q at i i , processing all the keys in the matrix K is computationally infeasible, so we limit the attention range to the local window and divide the Q, K and V matrices into W×W windows, where W is set to 7; let Q w , K w and V wrepresent the query, key, and value corresponding to the pixel in window w, respectively, then its output is for:

[0078]

[0079] Where B is a number of size w 2 ×w 2 The learnable bias represents the relative position embedding of each query and key pair; the attention is calculated for each window w, and then the window is rearranged to restore the corresponding spatial position in Q, enhancing the nonlinearity of the feature:

[0080]

[0081] Where LN represents the layer normalization operation.

[0082] Step 2: Downsample the input RGB and depth images through the convolution backbone network and send them to the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI to generate a multi-scale feature map, such as Figure 2 As shown, the details are as follows:

[0083] Step 2.1: Pass the input RGB and depth images of size H×W×3 through the convolutional backbone network, downsample the image using 3×3 convolution, and then apply two additional 3×3 convolutions with a stride of 1 for local feature extraction, generating a network of size Feature map of

[0084] Step 2.2: Concatenate the feature map with the average pooled input image to mitigate the loss of spatial information due to the reduction in feature size. Then, use a 3×3 convolution with a stride of 2 to further downsample the feature map to generate a size of The feature map is sent to the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI;

[0085] Step 2.3, set the size to The feature map is connected to the average pooled input image and downsampled in the third stage to generate a size of The feature map of is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI, and then the size of The feature map is connected to the average pooled input image and downsampled in the fourth stage to generate a size of The multi-scale feature map is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI.

[0086] Step 3: Bilinearly upsample the spatial dimensions and integrate the features from the three stages of the encoder through convolutional layers. A prediction head is set after each upsampling block to generate multi-scale inverse feature maps.

[0087] Step 4: Upsample and integrate the multi-scale inverse feature maps scale by scale to obtain a global feature map;

[0088] Step 5: The global feature map passes through three convolution layers consisting of three adaptively sized convolution kernels to output a rough camera pose matrix, as follows:

[0089] The global feature map passes through three convolutional layers consisting of three adaptively sized convolutional kernels, ensuring that the SPENet output is a coarse camera pose matrix P with a single channel and 3×5 dimensions.

[0090] Step 6: Perform Gram-Schmidt orthogonalization and determinant adjustment on the rough camera pose matrix to obtain an accurate camera pose estimate, as follows:

[0091] Perform Gram-Schmidt orthogonalization and determinant adjustment on the first three columns of matrix P to obtain the rotation matrix R. Ensure that R satisfies the properties of the rotation matrix. The third column of P is the t matrix, and the last column contains FoV x , FoV y and a matrix of useless values.

[0092] Step 7: Construct the t-distribution dual space Gaussian embedding model t-DSGE, as follows:

[0093] In 3D Gaussian sputtering, the t-SNE principle is extended to represent Gaussians in two different spaces: the spatial position space and the visual attribute space. The spatial position space is defined by the 3D coordinates of each Gaussian, represented by the position vector and the covariance matrix These parameters define the position and spatial distribution of each Gaussian in the scene; the visual attribute space represents the visual attributes of each Gaussian, including the opacity logarithm and the appearance features encoded by the spherical harmonic SH function of degree D; through the loss function based on KL divergence, it further ensures that neighboring Gaussians have highly similar appearance and opacity, enhances the representation ability of 3DGS, and improves the rendering ability of local details;

[0094] Step 8: Calculate two different Gaussian G in the scene in the spatial position space and visual attribute space respectively. i and G j The similarities between them are as follows:

[0095] In the spatial position space, calculate the Gaussian G i and Gj The similarities between them are as follows:

[0096] set up Represents a set of N Gaussians in the scene, where each Gaussian G i By position vector Covariance matrix Opacity Logarithm and color features To define, where the color feature is encoded by the spherical harmonic function of degree D, then in the spatial position space, the Gaussian G i and G j The calculation formula for the similarity between them is:

[0097]

[0098] in, is Gaussian G i The inverse of the covariance matrix. The main purposes of taking the inverse are: (1) to compute the Mahalanobis distance, which takes into account the shape and spread of the distribution to normalize the correlation between dimensions; and (2) to adapt the similarity measure to local anisotropy, ensuring consistent distance measurements across different regions.

[0099] In the visual attribute space, calculate the Gaussian G i and G j The similarity between them is:

[0100]

[0101] where v i =[o i ,c i ] represents Gaussian G i Visual similarity metrics emphasize that spatially neighboring Gaussians should have similar visual properties, thereby enhancing visual coherence in rendered scenes.

[0102] Step 9: Use KL divergence loss to strengthen the consistency between the spatial and visual spaces. The formula is:

[0103]

[0104] Using KL divergence loss ensures that spatially close Gaussians will have similar visual properties, thereby promoting smooth transitions and reducing visual artifacts in rendered scenes. Using KL divergence to align visual representations with spatial relationships is crucial for maintaining local detail accuracy and smooth rendering.

[0105] Step 10: Use the pose-free input image to optimize the framework parameters. The loss function includes the following parts:

[0106]

[0107] in, For the photometric reconstruction loss, the l1-norm difference between the input image and the rendered image from the same viewpoint is evaluated:

[0108]

[0109] Where C represents the predicted image, Corresponding to the real image;

[0110] The depth-dependent loss evaluates the relationship between rendered depth and predicted depth. The present invention uses a pre-trained dense prediction transformer (DPT) to generate a monocular depth map. For supervision, DPT is trained on the MIX6 dataset, which contains 1.4 million image-depth map pairs.

[0111] For the depth-related loss, the relationship between the rendered depth and the predicted depth is evaluated, and a relaxed relative depth loss based on Pearson correlation is adopted as follows:

[0112]

[0113] Where D is the predicted depth;

[0114] The present invention utilizes the denoising diffusion model (DDM) pre-trained on the Hypersim dataset as a learning prior for color and geometry. DDM provides a log-probability gradient that can be back-propagated to enhance the present method. as follows:

[0115]

[0116] Where p represents the value provided by DDM in a given training view In the case of , the posterior probability of the density and color fields.

[0117] Step 11: Use the trained t-distribution dual space Gaussian embedding model t-DSGE to enhance the representation of local details, rendering smoothness, and global consistency of the 3D scene.

[0118] Example

[0119] This embodiment uses the Tanks and Temples dataset and the Spaces dataset to comprehensively evaluate the results. For methods that require camera pose, COLMAP is used to obtain its camera pose. The Tanks and Temples dataset includes indoor and outdoor scenes, and feature objects include tanks, temples, castles, rooms, and large outdoor landscapes. Each scene provides multiple high-resolution images taken from different perspectives. An intermediate split is used that contains frames evenly sampled from eight more challenging outdoor environment high-resolution videos. The Spaces dataset consists of 100 indoor and outdoor scenes captured by a device using 16 cameras. The image set of each scene was taken at 5-10 device positions with a spacing of approximately 10 cm between each other.

[0120] The present invention uses peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learning-based perceptual image block similarity (LPIPS), and mean absolute error (l1↓) to evaluate the quality of synthesized views. Peak signal-to-noise ratio (PSNR↑) is used to measure the accuracy of RGB reconstruction (in dB), with higher values indicating better performance. SSIM↑ is used to assess image quality degradation (in %, with higher scores indicating better performance). Learning-based perceptual image block similarity (LPIPS↓) is used to assess the perceptual similarity between image blocks, with lower scores indicating higher similarity.

[0121] We compare our methods to several state-of-the-art sparse input models, including SVMPI, VMPI, MINE, AdaMPI, TMPI, and DiffusioneRF. All methods accept the same input: a single, unconstrained RGB image, allowing them to be evaluated as end-to-end 3D photography methods. Due to the limitation of a single-view input, the generated 3D photography is limited in scope. Experimental results for all compared methods are obtained using publicly available code or papers.

[0122] This example was implemented in PyTorch. Complexity experiments were performed using an NVIDIA GeForce RTX 3080 GPU, while others were performed using an NVIDIA A100 graphics card with 80GB of memory. This example was optimized over 10,000 iterations. For the optimization process, the Adam optimizer was used with an initial learning rate of 0.01. The learning rate was gradually decayed at each training step until the final reduction factor was 0.001. In the experiment, λ D Set to 0.005, ω DDM Set to 0.2, λ t Set to 0.001.

[0123] As shown in Table 1, the proposed method outperforms other methods on both datasets. On the Tanks and Temples dataset, the proposed method has a PSNR value of 19.31, which is 0.62 higher than the second-best method TMPI; the SSIM value is 0.597, which is 0.028 higher than TMPI, indicating higher accuracy and detail preservation; the LPIPS value is 0.241 and the l1 error is 0.052, which are also better than the second-best method, showing a reduction in perceptual error and pixel-level error. On the Spaces dataset, the proposed method has a PSNR value of 26.97, which is 0.80 higher than AdaMPI; the SSIM is 0.806, which is 0.056 higher than TMPI, indicating a stronger structure preservation ability; the proposed method has the lowest LPIPS of 0.124 and l1 of 0.024 in the same group, achieving the most accurate and visually similar results, highlighting the superiority of the proposed method in accuracy. These quantitative improvements in multiple indicators demonstrate that the proposed method is more robust and effective than existing methods in achieving higher image quality, structural fidelity, and perceptual similarity.

[0124] Table 1 Quantitative comparison on the TanksandTemples dataset and the Spaces dataset

[0125]

[0126] like Figure 4 As shown in the figure, in the Tank and Temples dataset, the method of the present invention outperforms other methods in rendering new viewpoints from a single input image. Specifically, VMPI exhibits obvious artifacts near edges, where occlusion is not handled properly, resulting in blurred or distorted boundaries; MINE is unable to capture fine details in textured areas, especially in complex textured areas, resulting in loss of clarity; AdaMPI encounters challenges in lighting consistency, with rendered shadows appearing unnatural and inconsistent with the original lighting direction; and TMPI has problems in complex structure areas, resulting in stretching or compression of details. In contrast, the method of the present invention can effectively address these challenges, generating more realistic and consistent rendering effects, with clearer details, better occlusion handling, and improved scene lighting coherence.

[0127] like Figure 5As shown in the figure, the proposed method shows significant advantages on the Spaces dataset in rendering new perspectives from a single input image. Specifically, SVMPI has difficulties in handling occlusions, resulting in blurred or distorted edges, especially near object boundaries; VMPI has limitations in maintaining texture details in complex areas, resulting in loss of clarity, especially in fine-textured areas; AdaMPI and TMPI have problems with the clarity of tree trunks, resulting in blurred textures, while VMPI and MINE suffer from detail distortion on wall clocks. In contrast, the proposed method excels in capturing fine scene details, accurately rendering edges, effectively handling occlusions, and maintaining consistent lighting in new perspectives, providing more realistic and visually coherent rendering effects.

[0128] Table 2 shows a complexity analysis of all methods using a single 350×630 RGB image. This resolution was chosen because the baseline methods could not meet the available memory requirements for higher resolutions on the specified hardware. GMAC represents giga-multiply-add operations per second, and Peak represents the maximum memory usage of the process in GB. The proposed method outperforms the other methods in terms of computational efficiency, speed, and memory usage. Specifically, it achieved the lowest GMAC value of 46.53, the fastest runtime of 20ms, and the lowest peak memory usage of 2.68GB. These results indicate that the proposed method requires fewer operations, runs significantly faster, and consumes less memory than competing methods, demonstrating its superior suitability for efficient and scalable processing.

[0129] Table 2 Complexity evaluation using a single 350×630 RGB image on NVIDIA GeForce RTX 3080 GPU

[0130]

[0131] Table 3 quantitatively demonstrates the effectiveness of SPENet and t-DSGE in our work. For the Tanks and Temples dataset, using SPENet alone improves PSNR from 8.42 to 18.64, SSIM from 0.043 to 0.547, and reduces LPIPS from 0.993 to 0.262. Adding t-DSGE alone achieves a PSNR of 9.64, an SSIM of 0.108, and a LPIPS of 0.976. Combining the two approaches provides the best results, achieving a PSNR of 19.31, an SSIM of 0.597, an LPIPS of 0.241, and an l1 error of 0.052. Similarly, on the Spaces dataset, using only SPENet can improve the PSNR value from 10.06 to 25.50, SSIM from 0.419 to 0.724, and reduce LPIPS from 0.950 to 0.193, while the combined method can further improve PSNR to 26.97, SSIM to 0.806, LPIPS to 0.124, and l1 error to 0.024.

[0132] Table 3 Ablation study on TanksandTemples dataset and Spaces dataset

[0133]

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media, including but not limited to magnetic disk storage, CD-ROMs, optical storage, and the like, containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices, systems, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0138] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A Gaussian sputtering method with no attitude restriction for single-view 3D photography, characterized in that: The following steps are involved: Step 1: Construct a skip-attention pose estimation network SPENet, which includes a large-core dilated convolution module LDC and a skip-attention feature interaction module SAFI; Step 2: Downsample the input RGB and depth images through the convolutional backbone network and send them to the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI to generate a multi-scale feature map; Step 3: Bilinearly upsample the spatial dimensions and integrate the features from the three stages of the encoder through convolutional layers. A prediction head is set after each upsampling block to generate multi-scale inverse feature maps. Step 4: Upsample and integrate the multi-scale inverse feature maps scale by scale to obtain a global feature map; Step 5: The global feature map passes through three convolution layers consisting of three adaptively sized convolution kernels to output a rough camera pose matrix. Step 6: Perform Gram-Schmidt orthogonalization and determinant adjustment on the rough camera pose matrix to obtain an accurate camera pose estimate; Step 7: Construct the t-distribution dual space Gaussian embedding model t-DSGE; Step 8: Calculate two different Gaussian G in the scene in the spatial position space and visual attribute space respectively. i and G j similarities between; Step 9: Use KL divergence loss to strengthen the consistency between the spatial and visual spaces; Step 10: Optimize the framework parameters using the pose-free input image. Step 11: Use the trained t-distribution dual space Gaussian embedding model t-DSGE to enhance the representation of local details, rendering smoothness, and global consistency of the 3D scene.

2. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 1, characterized in that: The skip attention pose estimation network SPENet constructed in step 1 includes the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI, as follows: Based on the 3D Gaussian sputtering framework, we construct a skip attention pose estimation network SPENet, using the horizontal field of view FoV x , vertical field of view FoV y , the rotation matrix R and the translation matrix t represent all the camera pose parameters used in the 3D Gaussian sputtering; since there may be multiple sets of camera parameters corresponding to a single image, a pre-trained dense prediction transformer DPT is used to estimate the depth map of the input single image. Under the depth map constraint, the unique corresponding camera parameters are obtained, where DPT is trained on the MIX6 dataset containing 1.4 million pairs of image-depth maps; The large kernel dilated convolution module (LDC) uses a large kernel dilated convolution layer to extract features. By stacking LDC models with different dilation rates r, the model aggregates sufficient multi-scale context. Given an input feature X of dimension H′W′C, the output produced by the LDC module is for: Among them DDWConv r (·) is an 11′11 dilated convolution with dilation rate r, BN is a batch normalization layer, Ln is a point-wise convolution layer, and Ln G is a Ln layer with a GELU activation function; in each stage, the LDC module is repeated three times, with r set to 1, 2, and 3 respectively; The skip attention feature interaction module SAFI uses the self-similarity between pixel queries and corresponding encoder feature maps to effectively fuse global-local features. The SAFI module is assumed to have an input feature map X with a dimension of H×W×C. First, X is projected onto a query Q=XW of the same dimension. q , key K = XW k Sum V = XW v , where W q 、W k and W v is the weight matrix; for a pixel query q at i i , processing all the keys in the matrix K is computationally infeasible, so we limit the attention range to the local window and divide the Q, K and V matrices into W×W windows, where W is set to 7; let Q w , K w and V w represent the query, key, and value corresponding to the pixels in window w, respectively, then the output is for: Where B is a number of size w 2 ×w 2 The learnable bias represents the relative position embedding of each query and key pair; the attention is calculated for each window w, and then the window is rearranged to restore the corresponding spatial position in Q, enhancing the nonlinearity of the feature: Where LN represents the layer normalization operation.

3. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 2, characterized in that: As described in step 2, the input RGB and depth images are downsampled through the convolution backbone network and sent to the large kernel dilated convolution module LDC and the skip attention feature interaction module SAFI to generate a multi-scale feature map, as follows: Step 2.1: Pass the input RGB and depth images of size H×W×3 through the convolutional backbone network, downsample the image using 3×3 convolution, and then apply two additional 3×3 convolutions with a stride of 1 for local feature extraction, generating a network of size Feature map of Step 2.2: Concatenate the feature map with the average pooled input image, and then further downsample the feature map using a 3×3 convolution with a stride of 2 to generate a feature map of size The feature map is sent to the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI; Step 2.3, set the size to The feature map is connected to the average pooled input image and downsampled in the third stage to generate a size of The feature map of is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI, and then the size of The feature map is connected to the average pooled input image and downsampled in the fourth stage to generate a size of The multi-scale feature map is fed into the large kernel expansion convolution module LDC and the skip attention feature interaction module SAFI.

4. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 3, wherein: The global feature map described in step 5 is passed through three convolutional layers consisting of three adaptively sized convolution kernels to output a rough camera pose matrix, as follows: The global feature map passes through three convolutional layers consisting of three adaptively sized convolutional kernels, ensuring that the SPENet output is a coarse camera pose matrix P with a single channel and 3×5 dimensions.

5. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 4, wherein: The rough camera pose matrix described in step 6 is subjected to Gram-Schmidt orthogonalization and determinant adjustment to obtain an accurate camera pose estimate, as follows: Perform Gram-Schmidt orthogonalization and determinant adjustment on the first three columns of matrix P to obtain the rotation matrix R. Ensure that R satisfies the properties of the rotation matrix. The third column of P is the t matrix, and the last column contains FoV x , FoV y and a matrix of useless values.

6. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 5, characterized in that: The construction of the t-distribution dual space Gaussian embedding model t-DSGE described in step 7 is as follows: In 3D Gaussian sputtering, the t-SNE principle is extended to represent Gaussians in two different spaces: the spatial position space and the visual attribute space. The spatial position space is defined by the 3D coordinates of each Gaussian, represented by the position vector and the covariance matrix Represents, defining the position and spatial distribution of each Gaussian in the scene; the visual attribute space represents the visual attributes of each Gaussian, including the opacity logarithm and the appearance features encoded by the spherical harmonic SH function of degree D; through the loss function based on KL divergence, it further ensures that neighboring Gaussians have highly similar appearance and opacity, enhances the representation ability of 3DGS, and improves the rendering ability of local details.

7. The method for unposed-attitude-limited Gaussian sputtering for single-view 3D photography according to claim 6, wherein: In step 8, in the spatial position space, calculate the Gaussian G i and G j The similarities between them are as follows: set up Represents a set of N Gaussians in the scene, where each Gaussian G i By position vector Covariance matrix Opacity Logarithm and color features To define, where the color feature is encoded by the spherical harmonic function of degree D, then in the spatial position space, the Gaussian G i and G j The calculation formula for the similarity between them is: in, is Gaussian G i Inverse of the covariance matrix.

8. The method for the attitude-free Gaussian sputtering for single-view 3D photography according to claim 7, wherein: In step 8, in the visual attribute space, calculate the Gaussian G i and G j The similarity between them is: where v i =[o i ,c i ] represents Gaussian G i visual properties.

9. The method for unposed-attitude-limited Gaussian sputtering for single-view 3D photography according to claim 8, wherein: The KL divergence loss described in step 9 is used to strengthen the consistency between the spatial and visual spaces. The formula is:

10. The attitude-free Gaussian sputtering method for single-view 3D photography according to claim 9, characterized in that: In step 10, the loss function for optimizing the framework parameters using the pose-free input image consists of the following parts: in, For the photometric reconstruction loss, the l1-norm difference between the input image and the rendered image from the same viewpoint is evaluated: Where C represents the predicted image, Corresponding to the real image; For the depth-related loss, the relationship between the rendered depth and the predicted depth is evaluated, and a relaxed relative depth loss based on Pearson correlation is adopted as follows: Where D is the predicted depth; is the logarithmic probability gradient of the denoising diffusion model DDM: Where p represents the value provided by DDM in a given training view In the case of , the posterior probability of the density and color fields.

Images