Credibility-driven scalable multi-view 3D gaussian splatting method

Abstract

The application discloses a kind of based on credibility drive's extensible multi-view 3D Gaussian splashing method.This application contains three modules: new view initialization module based on multi-view stereo vision index, perception constraint depth inference module and perspective credibility Gaussian rendering module;New view initialization module based on multi-view stereo vision index obtains initial low-resolution depth map and rendering map;Perception constraint depth inference module obtains reliable depth estimation value by evaluating the consistency of perception, so as to effectively deal with the challenge brought by source image quality fluctuation and scene geometric complexity;Perspective credibility Gaussian rendering module introduces perspective-related credibility information in the Gaussian inference process to improve the fidelity of rendered image and the generalization ability of the method.The application performs excellently in the above-mentioned module under the joint action, and can be widely applied in virtual reality, augmented reality, film special effect production and other fields.

Description

Technical Field

[0001] This invention relates to the field of computer vision and graphics, specifically to a credibility-driven scalable multiview architecture. Figure 3 CredSplatting, a Gaussian splashing method, is used for novel view synthesis (NVS) and complex scene reconstruction. CredSplatting employs a Perception-Constrained Depth Inference (PCDI) module to selectively integrate multi-view features and evaluate perceptual consistency to obtain reliable depth estimates, effectively addressing challenges posed by source image quality and scene geometric variations. Furthermore, this invention introduces a View-Credence Gaussian Rendering (VCGR) module, which incorporates view-specific credibility information during Gaussian inference, significantly improving the fidelity of the rendered image and the method's generalization ability. This invention is applicable to high-quality novel view generation and 3D reconstruction tasks in various complex scenes and can be widely applied in virtual reality, augmented reality, and film and television special effects production, providing an efficient and robust solution for the development of related technologies. Background Technology

[0002] The core of novel view synthesis is to generate highly realistic images from entirely new viewpoints using source images and corresponding camera pose information. Currently, mainstream solutions are mainly divided into methods based on Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS). NeRF-based methods synthesize high-quality novel viewpoint images through implicit neural representations combined with differentiable volume rendering, but their high computational cost for training and rendering limits their widespread adoption in real-time applications. In contrast, 3DGS-based methods employ explicit scene representations and achieve high-quality real-time rendering through differentiable rasterization. Similar to NeRF, 3DGS methods still require extensive iterative optimization for each new scene to obtain ideal rendering results. How to quickly generalize well-trained models to unseen scenes while maintaining the quality of synthesized images, and reducing the additional optimization time for new scenes, has become a hot topic and trend in current NVS research.

[0003] With increasing attention focused on generalizable NVS tasks, several generalizable methods based on NeRF or 3DGS have emerged. For example, the generalizable NeRF paradigm MVSNeRF constructs a cost volume based on multi-view stereo vision and uses a 3DCNN to jointly infer scene geometry and appearance information to train a generalizable neural network for the NVS task. The generalizable Gaussian paradigm pixelSplat directly utilizes paired images to reconstruct the radiation field parameterized by 3D Gaussian primitives, achieving generalizable NVS. While these generalizable methods have made significant progress in reducing scene-by-scene optimization, rendering quality still requires further improvement. Issues to consider include rendering errors due to insufficient field of view coverage when there are few input views, and image boundary artifacts due to occlusion, scale differences, and multi-view information conflicts when there are many input views. Most existing general-purpose 3DGS methods typically rely on regressing Gaussian parameters from a single cost volume generated from a fixed source view, which amplifies the effects of insufficient view overlap and noise from individual views. Furthermore, the source images differ in visual fidelity, visible area, and noise levels, resulting in an uneven contribution to new perspective rendering. Fine-tuning for specific scenes is often insufficient to alleviate these issues. Against this backdrop, more rational utilization of multi-view information is a key factor in improving generalizable NVS performance.

[0004] To address the aforementioned challenges, a method needs to be designed that comprehensively considers the relationship between imaging quality and visibility from various viewpoints. By reasonably filtering reliable information from the input viewpoints, more accurate and robust Gaussian meta-parameters of the geometric structure can be inferred, thereby enabling more robust synthesis of generalizable new viewpoints.

[0005] Therefore, this invention proposes a trustworthiness-driven scalable multi-view approach. Figure 3 CredSplatting, a Gaussian splashing method, is used to fully exploit reliable information from different viewpoints to achieve fast and high-fidelity rendering and 3D reconstruction of unknown scenes in situations where there is uncertainty in any input viewpoint. Unlike previous works that relied solely on a single multi-view cost volume for parameter regression, CredSplatting learns the visibility score of each pixel for its corresponding viewpoint, indicating the contribution ratio of each input view, and comprehensively considers multiple cost volumes during the inference of rendering parameters. This method adaptively selects the viewpoint information that contributes most to the synthesis of the new viewpoint within the effective field of view coverage of the new view, inferring more robust Gaussian meta-parameters to reduce artifacts and geometric errors, thereby improving rendering quality. CredSplatting employs an end-to-end training paradigm, using only RGB images as supervision signals, which effectively fine-tunes unseen scenes and enhances generalization ability across datasets. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a trust-driven, scalable multi-view architecture. Figure 3 The core task of novel view synthesis is to generate highly realistic images from entirely new viewpoints using source images and their corresponding camera pose information. Currently, mainstream solutions are mainly divided into NeRF-based and 3DGS-based methods. Although these methods have made significant progress in high-quality rendering, they still face many challenges. For example, when the input views are sparse, insufficient field of view coverage leads to incorrect rendering; when the input views are dense, boundary artifacts are caused by occlusion, scale differences, or conflicts in multi-view information. Furthermore, existing general-purpose 3DGS methods typically rely on regressing Gaussian parameters from a single cost volume generated from a fixed source view, which amplifies the effects of insufficient view overlap and noise in individual views, leading to a decrease in rendering quality.

[0007] Based on this, the present invention proposes a trustworthiness-driven scalable multi-view architecture. Figure 3 The CredSplatting method introduces a Perception-Constrained Depth Inference (PCDI) module and a View-Credence Gaussian Rendering (VCGR) module. The PCDI module selectively integrates reliable information to obtain accurate depth estimates by evaluating the perceptual consistency of multi-view features. This module effectively addresses the challenges posed by source image quality and scene geometric variations, reducing errors caused by insufficient view overlap or noise. The VCGR module learns the visibility score of each pixel for its corresponding viewpoint, quantifying the contribution of each input view to the rendering of the new viewpoint. This module, combined with view-specific confidence information, adaptively selects the viewpoint information that contributes most to the synthesis of the new viewpoint during the Gaussian inference process, thereby reducing artifacts and geometric errors and improving the fidelity of the rendered image. This invention can adaptively select the viewpoint information that contributes most to the synthesis of the new viewpoint within the effective field of view coverage of the new view, inferring more robust Gaussian meta-parameters, thereby reducing artifacts and geometric inaccuracies and improving rendering quality.

[0008] The technical solution of this invention to solve its technical problem includes the following steps:

[0009] Step 1: Collect the DTU, Real Forward-Facing, and Tanks and Temples datasets;

[0010] Step 2: Divide the DTU dataset into training and validation sets. Select a portion of data from the Real Forward-Facing dataset and the Tanks and Temples dataset as an additional validation set. The model will be trained using only the DTU dataset.

[0011] The DTU dataset, collected and released by the Technical University of Denmark, is a public dataset for multi-view stereo reconstruction and 3D modeling, containing 128 indoor scenes. The Real Forward-Facing dataset, released at SIGGRAPH in 2019, is a dataset for generating novel perspectives in sparse viewpoints, containing 8 scenes. The Tanks and Temples dataset, released by Intel Labs in 2017, provides 21 real-world sampled scenes, including large vehicles and buildings.

[0012] Step 3: Data preprocessing. Scale the images in the training and validation sets to a uniform size and normalize their pixel values ​​to the range [0, 1]. Calculate the distance between the dataset images and the camera pose of the image to be predicted based on the camera pose, sort them, and use the L nearest dataset images as input.

[0013] Step 4: Build a neural network model for the new perspective synthesis task. The neural network model includes a new view initialization module based on multi-view stereo vision index, a perceptual constraint depth inference module, and a view confidence Gaussian rendering module.

[0014] Step 5: Train and infer the model, and save the model weights that produce the best results.

[0015] Step 6: Model performance evaluation and result analysis.

[0016] Furthermore, the new view initialization module based on multi-view stereo vision index comprises two parts: the MVS depth inference part and the Gaussian meta-inference part.

[0017] Furthermore, the specific implementation of the MVS deep inference part is as follows:

[0018] The MVS deep inference part employs a learning-based traditional multi-view stereo vision network to infer the depth map and image of the target view to be predicted. First, a feature pyramid network is used to infer the depth map and image of the target view from the source view. Extracting multi-scale features The three multi-scale features have dimensions of the source view size. , Size is the input source view dimension Multiscale features It was used to estimate the initial depth map; subsequently, the depth hypothesis plane was plotted in the view frustum of the target view. To be adopted, among which Next, the multi-scale features of all source views. Mapping to the target view using differentiable homography distortion:

[0019]

[0020]

[0021] in, This represents the homography matrix from source view i to target view at sampling depth d. and These represent the camera intrinsics, rotation parameters, and translation parameters of the source view and the target view, respectively. It is the camera's main axis in the target view. It is the identity matrix. It is the sampling depth. Multi-scale features of the i-th source view Source feature I obtained after distortion; using source feature I after distortion And construct a multi-view cost body based on variance metric. The cost volume is then fed into a 3D U-Net for regularization to infer the initial depth map of the target view.

[0022] The specific implementation of the Gaussian meta-inference part is as follows:

[0023] 3D Gaussian point plotting parameterizes a 3D scene into a set of Gaussian primitives, each Gaussian primitive consisting of attributes. It indicates. Among them, It is the center of Gorsky's unit, and These represent the scaling factor and rotation factor, respectively. and Represents opacity and color factor; establishes a point-to-point mapping between Gaussian units and pixels; before performing Gaussian point rendering, a volumetric rendering based on a single depth-aware sampling point is performed, where the sampling point position along the ray is determined by the initial depth map. The radiance and volume density of the sampling points were determined from the multi-view aggregated features. It is obtained by decoding.

[0024] Multi-view aggregation feature Solving for: Multi-view aggregation features By combining multi-view cost body Multi-scale features of source views And obtained after passing through a convolutional neural network.

[0025] Using the initial depth map After volumetric rendering, the obtained sampling points generate a blended color map under the target view. and depth map In addition, multi-view aggregation features Spatial augmentation is performed using 2D U-Net and combined with the fused color map to generate context-enhanced features. Subsequently, separate multilayer perceptrons were used to enhance features from the context. Decoding color factors Opacity Scaling factor and rotation factor Meanwhile, the center of Gorsky's unit By using camera parameters to fuse depth maps The back projection is determined in world space; finally, based on rasterization rendering, the decoded Gaussian primitives are used to generate an initial target view by averaging the color map of the Gaussian rendering and the color map of the volumetric rendering.

[0026] Furthermore, the perceptual constraint depth reasoning module selectively integrates multi-view cost volumes to suppress interference from features misaligned with the target view, while enhancing the contribution of highly consistent features to generate a reliable depth map consistent with the scene geometry.

[0027] Furthermore, the perceptual constraint deep reasoning module is specifically implemented as follows:

[0028] First, a two-dimensional convolutional neural network is used to extract high-dimensional features from the initial target view synthesized in the initial rendering stage. Context robustness is enhanced by aggregating multi-view aggregated features with the initial target view, resulting in target view features. :

[0029]

[0030] in, Indicates a convolutional layer. Indicates channel cascading operation; and These represent the color map for Gaussian rendering and the color map for volumetric rendering, respectively. This represents the weighted sum of two color maps. This indicates a multi-view aggregation feature.

[0031] Then, by constructing pairwise correlation cost volumes between each source view and the initial target view, the relationships between features are modeled individually; in the initial depth map Build a local depth search buffer around it Sampling was performed, among which Within this buffer, a set of depth hypothesis planes are sampled to map source view features to the initial target view via homography warping. The warped source features are paired with target view features, and their correlation is calculated using the following formula to obtain the correlation cost volume. :

[0032]

[0033]

[0034]

[0035] in, Represent the target view features and the multi-scale features of the i-th source view, respectively. Source feature II obtained after distortion. This represents the homography matrix from source view i to target view at sampling depth d. and These represent the camera intrinsics, rotation parameters, and translation parameters of the source view and the target view, respectively. It is the camera's main axis in the target view. It is the identity matrix. It is the sampling depth.

[0036] Perceptual consistency score for each original image It is based on the corresponding correlation cost body The perceptual consistency score was obtained through a perceptual learning unit consisting of two-dimensional convolutional neural network layers. This unit includes a convolutional module composed of convolutions, GroupNorm, and ReLU, a ResNet block, a convolution that reduces the number of channels to 1, and a Sigmoid function. Under the guidance of relevant cost body and the distorted source features Adaptive fusion is performed, and the fused multi-view cost volume is obtained. Represented as:

[0037]

[0038] In addition to the correlation cost encoding the similarity between the target view and the source views, a variance-based cost is added to encode the consistency among all source views. This enriches the final cost volume by integrating multi-view information. Therefore, the final multi-view fusion cost volume... Represented as:

[0039]

[0040] in, This represents the variance of all distorted source feature II.

[0041] A multi-scale 3D CNN network is used to process the cost of perceptually constrained multi-view fusion. Regularization is performed, and then probability weights are applied to the regularized features along the depth hypothesis plane to obtain a detailed depth map of the initial target view. .

[0042] Furthermore, the view confidence Gaussian rendering module utilizes full-resolution Gaussian meta-inference and combines the feature confidence of each view to eliminate over-reliance on finite cost volumes.

[0043] Furthermore, the Gaussian rendering module for viewpoint credibility is implemented as follows:

[0044] First, the feature credibility of each source view relative to the target view is quantified by scoring perceptual consistency along the depth dimension. Perform pixel-wise max pooling to generate a confidence probability map. Its calculation formula is ,in This indicates the number of selected cost volumes used for Gaussian rendering. The confidence probability graph is normalized to ensure comparability among all views.

[0045]

[0046]

[0047] in, This represents cumulative credibility, while Indicates the first Normalized confidence level of each view.

[0048] Correlation cost body Extended to the differential cost body The cascading results thus form the enhanced cost body. The details are as follows:

[0049]

[0050]

[0051] For each cost body After regularization using 3D U-Net, it is combined with the multi-scale features of the source view. Cascaded features and fused them using a 3D CNN to generate an enhanced feature body. Based on the set of enhanced feature volumes, Gaussian elements that are robust to view-related noise are inferred.

[0052] Similar to the operation of the new view initialization module based on multi-view stereo vision indexing, the view confidence Gaussian rendering module uses depth-aware volumetric rendering to assist in learning the color factors of Gaussian primitives. The difference lies in that, guided by view confidence, multiple cost volumes are adaptively integrated to infer color factors, thereby mitigating the impact of view-related artifacts. Radiance is derived independently from each cost volume, corresponding to a single source view.

[0053] Due to the reliability of normalization Reflecting the relative reliability of each source view with respect to the target view, the weighted and reliability-based combined volume rendering equation is expressed as:

[0054]

[0055] in, and These represent the density and color at the sampling points along the ray, respectively. and These represent the number of selected cost bodies and the number of sampling points, respectively. Indicates the first The cost body The density of each sampling point Indicates the first The cost body's first Normalized confidence level of each sampling point Indicates the first The cost body's first The density of each sampling point. Indicates the first The cost body's first Color of each sampling point, Indicates the first The cost body's first The confidence level of each sampling point is normalized. After volume rendering as expressed by formula (13), a more refined color map is generated under the target view. and depth map .

[0056] By introducing view confidence, multiple cost volumes are jointly utilized, rather than relying solely on a single cost volume to infer pixel-wise color attributes. Therefore, this approach helps mitigate boundary artifacts caused by multi-view noise or occlusion. This adaptive rendering ultimately ensures stable rendering quality across different input views.

[0057] To infer Gaussian units, all enhanced feature volumes Using the normalized confidence graph Adaptive fusion to obtain feature maps :

[0058]

[0059] in, Subsequently, 2D U-Net enhancement was used to improve the consistency of spatial context. The output feature map of 2D U-Net and the volumetric rendering output color map were then compared. The components are combined and then decoded into color factors by a multilayer perceptron. Opacity Scaling and rotation factor The Gaussian center is obtained from the output feature map of the 2D U-Net through independent multilayer perceptron decoding. Then, by analyzing the depth map of the fusion... The back-projection into 3D world space determines the target. Finally, using these inferred Gaussian elements, a high-quality target image is rendered.

[0060] Furthermore, the present invention also provides a trust-driven scalable multi-view architecture. Figure 3 The D-Gauss splash system includes the aforementioned trust-driven scalable multi-view system. Figure 3 D Gaussian splashing method.

[0061] Beneficial effects of this invention:

[0062] This invention addresses the inefficiencies of existing Gaussian models in scene-by-scene optimization and their insufficient adaptability to different numbers of image inputs from varying viewpoints, proposing a novel solution. Existing methods often suffer from rendering errors due to insufficient field of view coverage when processing sparse views; while when processing dense views, they may exhibit boundary artifacts due to occlusion, scale differences, or conflicts in multi-view information. To resolve these challenges, this invention designs the CredSplatting architecture, which achieves more robust new viewpoint synthesis and 3D reconstruction by comprehensively analyzing the relationship between imaging quality and visibility of each viewpoint. Specifically, CredSplatting introduces a perceptual consistency evaluation mechanism to filter reliable input information from multi-view features to generate accurate depth estimates. Simultaneously, this architecture calculates the visibility score of each pixel for its corresponding viewpoint, quantifying the contribution ratio of each input view to the rendering of the new viewpoint, thereby obtaining a high-quality Gaussian primitive representation. This method effectively reduces errors caused by insufficient viewpoint overlap or noise interference, significantly reduces the incidence of artifacts and geometric errors, and further improves rendering quality and reconstruction accuracy. Experimental results show that CredSplatting exhibits excellent performance in new perspective synthesis, geometry reconstruction, and cross-scene generalization, demonstrating broad application potential and providing strong technical support for fields such as virtual reality, augmented reality, and film and television special effects production. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a new view initialization module based on Multi-View Stereo Vision (MVS) indexing.

[0064] Figure 2 This is a schematic diagram of the Perception-Constrained Depth Inference (PCDI) module.

[0065] Figure 3 This is a schematic diagram of the View-Credence Gaussian Rendering (VCGR) module. Detailed Implementation

[0066] The present invention will be further explained and described below with reference to the accompanying drawings and embodiments.

[0067] like Figure 1 , Figure 2 and Figure 3 As shown, a trust-driven scalable multi-view architecture Figure 3 The D-Gaussian splashing method aims to achieve high-quality new perspective generation and 3D reconstruction in complex scenes, and specifically includes the following steps:

[0068] Step 1: Collect the DTU, Real Forward-Facing, and Tanks and Temples datasets;

[0069] Step 2: Divide the DTU dataset into training and validation sets. Select a portion of data from the Real Forward-Facing dataset and the Tanks and Temples dataset as an additional validation set. The model will be trained using only the DTU dataset.

[0070] Step 3: Data preprocessing. Scale the images in the training and validation sets to a uniform size and normalize their pixel values ​​to the range [0, 1]. Calculate the distance between the dataset images and the camera pose of the image to be predicted based on the camera pose, sort them, and use the L nearest dataset images as input.

[0071] Step 4: To achieve high-quality new perspective generation, this invention proposes a new perspective generation network, which includes three modules: a new view initialization module based on multi-view stereo vision (MVS) indexing, a perception-constrained depth inference (PCDI) module, and a view-creence Gaussian rendering (VCGR) module. The structure and function of CredSplatting and its modules are described in detail below, along with relevant mathematical formulas.

[0072] The overall structure of CredSplatting is as follows: Figure 1 As shown, the algorithm includes a new view initialization module based on multi-view stereo vision (MVS) indexing, a perception-constrained depth inference (PCDI) module, and a view-credence Gaussian rendering (VCGR) module. First, an MVS network is used to infer the depth map of the unseen target view. Guided by the depth inference, multi-view features are warped and aggregated to generate pixel-wise Gaussian units for rendering the initial target view. Next, we construct pairwise cost volumes between each source view and the initial target view and learn the perceptual consistency score for each view. These scores are used to guide the fusion of multi-view costs, thereby inferring a more refined depth map. Finally, based on the view credibility of each source view, features from each view are selectively fused during Gaussian unit inference to render a high-quality target image.

[0073] Furthermore, the new view initialization module based on multi-view stereo vision index comprises two parts: the MVS depth inference part and the Gaussian meta-inference part.

[0074] Furthermore, the specific implementation of the MVS deep inference part is as follows:

[0075] The MVS deep inference part employs a learning-based traditional multi-view stereo vision network to infer the depth map and image of the target view to be predicted. First, a feature pyramid network is used to infer the depth map and image of the target view from the source view. Extracting multi-scale features The three multi-scale features have dimensions of the source view size. , Size is the input source view dimension Multiscale features It was used to estimate the initial depth map; subsequently, the depth hypothesis plane was plotted in the view frustum of the target view. Adopted, among which Next, the multi-scale features of all source views. Mapping to the target view using differentiable homography distortion:

[0076]

[0077]

[0078] in, This represents the homography matrix from source view i to target view at sampling depth d. and These represent the camera intrinsics, rotation parameters, and translation parameters of the source view and the target view, respectively. It is the camera's main axis in the target view. It is the identity matrix. It is the sampling depth. Multi-scale features of the i-th source view Source feature I obtained after distortion; using source feature I after distortion And construct a multi-view cost body based on variance metric. The cost volume is then fed into a 3D U-Net for regularization to infer the initial depth map of the target view.

[0079] The specific implementation of the Gaussian meta-inference part is as follows:

[0080] 3D Gaussian point plotting parameterizes a 3D scene into a set of Gaussian primitives, each Gaussian primitive consisting of attributes. It indicates. Among them, It is the center of Gorsky's unit, and These represent the scaling factor and rotation factor, respectively. and Represents opacity and color factor; establishes a point-to-point mapping between Gaussian units and pixels; before performing Gaussian point rendering, a volumetric rendering based on a single depth-aware sampling point is performed, where the sampling point position along the ray is determined by the initial depth map. The radiance and volume density of the sampling points were determined from the multi-view aggregated features. It is obtained by decoding.

[0081] Multi-view aggregation feature Solving for: Multi-view aggregation features By combining multi-view cost body Multi-scale features of source views And obtained after passing through a convolutional neural network.

[0082] Using the initial depth map After volumetric rendering, the obtained sampling points generate a blended color map under the target view. and depth map In addition, multi-view aggregation features Spatial augmentation is performed using 2D U-Net and combined with the fused color map to generate context-enhanced features. Subsequently, separate multilayer perceptrons were used to enhance features from the context. Decoding color factors Opacity Scaling factor and rotation factor Meanwhile, the center of Gorsky's unit By using camera parameters to fuse depth maps The back projection is determined in world space; finally, based on rasterization rendering, the decoded Gaussian primitives are used to generate an initial target view by averaging the color map of the Gaussian rendering and the color map of the volumetric rendering.

[0083] Furthermore, the perceptual constraint depth reasoning module selectively integrates multi-view cost volumes to suppress interference from features misaligned with the target view, while enhancing the contribution of highly consistent features to generate a reliable depth map consistent with the scene geometry.

[0084] Furthermore, the perceptual constraint deep reasoning module is specifically implemented as follows:

[0085] First, a two-dimensional convolutional neural network is used to extract high-dimensional features from the initial target view synthesized in the initial rendering stage. Context robustness is enhanced by aggregating multi-view aggregated features with the initial target view, resulting in target view features. :

[0086]

[0087] in, Indicates a convolutional layer. Indicates channel cascading operation; and These represent the color map for Gaussian rendering and the color map for volumetric rendering, respectively. This represents the weighted sum of two color maps. This indicates a multi-view aggregation feature.

[0088] Then, by constructing pairwise correlation cost volumes between each source view and the initial target view, the relationships between features are modeled individually; in the initial depth map Build a local depth search buffer around it Sampling was performed, among which Within this buffer, a set of depth hypothesis planes are sampled to map source view features to the initial target view via homography warping. The warped source features are paired with target view features, and their correlation is calculated using the following formula to obtain the correlation cost volume. :

[0089]

[0090]

[0091]

[0092] in, Represent the target view features and the multi-scale features of the i-th source view, respectively. Source feature II obtained after distortion. This represents the homography matrix from source view i to target view at sampling depth d. and These represent the camera intrinsics, rotation parameters, and translation parameters of the source view and the target view, respectively. It is the camera's main axis in the target view. It is the identity matrix. It is the sampling depth.

[0093] Perceptual consistency score for each original image It is based on the corresponding correlation cost body The perceptual consistency score was obtained through a perceptual learning unit consisting of two-dimensional convolutional neural network layers. This unit includes a convolutional module composed of convolutions, GroupNorm, and ReLU, a ResNet block, a convolution that reduces the number of channels to 1, and a Sigmoid function. Under the guidance of relevant cost body and the distorted source features Adaptive fusion is performed, and the fused multi-view cost volume is obtained. Represented as:

[0094]

[0095] In addition to the correlation cost encoding the similarity between the target view and the source views, a variance-based cost is added to encode the consistency among all source views. This enriches the final cost volume by integrating multi-view information. Therefore, the final multi-view fusion cost volume... Represented as:

[0096]

[0097] in, This represents the variance of all distorted source feature II.

[0098] A multi-scale 3D CNN network is used to process the cost of perceptually constrained multi-view fusion. Regularization is performed, and then probability weights are applied to the regularized features along the depth hypothesis plane to obtain a detailed depth map of the initial target view. .

[0099] Furthermore, the view confidence Gaussian rendering module utilizes full-resolution Gaussian meta-inference and combines the feature confidence of each view to eliminate over-reliance on finite cost volumes.

[0100] Furthermore, the Gaussian rendering module for viewpoint credibility is implemented as follows:

[0101] First, the feature credibility of each source view relative to the target view is quantified by scoring perceptual consistency along the depth dimension. Perform pixel-wise max pooling to generate a confidence probability map. Its calculation formula is ,in This indicates the number of selected cost volumes used for Gaussian rendering. The confidence probability graph is normalized to ensure comparability among all views.

[0102]

[0103]

[0104] in, This represents cumulative credibility, while Indicates the first Normalized confidence level of each view.

[0105] Correlation cost body Extended to the differential cost body The cascading results thus form the enhanced cost body. The details are as follows:

[0106]

[0107]

[0108] For each cost body After regularization using 3D U-Net, it is combined with the multi-scale features of the source view. Cascaded features and fused them using a 3D CNN to generate an enhanced feature body. Based on the set of enhanced feature volumes, Gaussian elements that are robust to view-related noise are inferred.

[0109] Similar to the operation of the new view initialization module based on multi-view stereo vision indexing, the view confidence Gaussian rendering module uses depth-aware volumetric rendering to assist in learning the color factors of Gaussian primitives. The difference lies in that, guided by view confidence, multiple cost volumes are adaptively integrated to infer color factors, thereby mitigating the impact of view-related artifacts. Radiance is derived independently from each cost volume, corresponding to a single source view.

[0110] Due to the reliability of normalization Reflecting the relative reliability of each source view with respect to the target view, the weighted and reliability-based combined volume rendering equation is expressed as:

[0111]

[0112] in, and These represent the density and color at the sampling points along the ray, respectively. and These represent the number of selected cost bodies and the number of sampling points, respectively. Indicates the first The cost body The density of each sampling point Indicates the first The cost body's first Normalized confidence level of each sampling point Indicates the first The cost body's first The density of each sampling point. Indicates the first The cost body's first Color of each sampling point, Indicates the first The cost body's first The confidence level of each sampling point is normalized. After volume rendering as expressed by formula (13), a more refined color map is generated under the target view. and depth map .

[0113] By introducing view confidence, multiple cost volumes are jointly utilized, rather than relying solely on a single cost volume to infer pixel-wise color attributes. Therefore, this approach helps mitigate boundary artifacts caused by multi-view noise or occlusion. This adaptive rendering ultimately ensures stable rendering quality across different input views.

[0114] To infer Gaussian units, all enhanced feature volumes Using the normalized confidence graph Adaptive fusion to obtain feature maps :

[0115]

[0116] in, Subsequently, 2D U-Net enhancement was used to improve the consistency of spatial context. The output feature map of 2D U-Net and the volumetric rendering output color map were then compared. The components are combined and then decoded into color factors by a multilayer perceptron. Opacity Scaling and rotation factor The Gaussian center is obtained from the output feature map of the 2D U-Net through independent multilayer perceptron decoding. Then, by analyzing the depth map of the fusion... The back-projection into 3D world space determines the target. Finally, using these inferred Gaussian elements, a high-quality target image is rendered.

[0117] Furthermore, the present invention also provides a trust-driven scalable multi-view architecture. Figure 3 The D-Gauss splash system includes the aforementioned trust-driven scalable multi-view system. Figure 3 D Gaussian splashing method.

[0118] Step 5: Train the framework using the training set partitioned in Step 2. This framework is implemented using CUDA 12.1, Python 3.9, and PyTorch 2.0.0, and trained on dual NVIDIA 4090 GPUs. The learning rate is set to 5e-4, and the training is validated every 1000 samples, with the best weights saved. During training, the number of input viewpoints is a choice between 2, 3, and 4, with probabilities of 0.1, 0.8, and 0.1 respectively. During validation, the number of input viewpoints is 3.

[0119] For the loss function used in training, this invention performs end-to-end training using only the supervision signal of the target RGB image. The composite loss function used in the training process is represented by a weighted combination of Mean Squared Error (MSE) loss, Structural Similarity (SSIM) loss, and Perceptual Loss, and its mathematical expression is as follows:

[0120]

[0121] in, This represents the total loss in stage k. In this invention, k is chosen to be 1 or 2. and Structural similarity loss and perceived loss The weighting coefficients.

[0122] The components of the composite training loss function are defined as follows:

[0123]

[0124]

[0125]

[0126] in, Represents the total number of pixels in the image. and These represent the inferred color value and the actual color value, respectively. and These represent the rendered image and the actual image, respectively. This represents the structural similarity function, while Indicates passage A perceptual feature extractor instantiated from a network.

[0127] Step 6: Model Performance Evaluation and Result Analysis. Using the weights obtained in Step 5 that best perform on the validation set, calculate and analyze the results on the validation set. Rendering accuracy evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). Geometric accuracy evaluation metrics include Absolute Error (Abs err), Accuracy under Threshold 2 (Acc 2), and Accuracy under Threshold 10 (Acc 10).

[0128] Experimental results show that the proposed method significantly outperforms existing scene-by-scene and generalized methods in terms of rendering quality and geometric accuracy. It exhibits stable rendering and reconstruction quality under different source image data conditions and demonstrates good cross-scene generalization ability on multiple datasets. The proposed method shows significant application potential in handling the adaptive needs of unknown scenes.

[0129] Table 1 Comparison of Rendering Accuracy Experimental Results for DTU Dataset

[0130]

[0131] The results of inputting 2, 3, 4, and 5 source images into the DTU dataset are shown in Table 1. GredSplatting outperforms other methods in all metrics. Our model achieves the best performance, with a PSNR 0.81 dB higher than the second-best model when using source image input. Notably, some previous methods show a significant performance drop when inferring new perspective images using 5 source images, while our method effectively alleviates this problem.

[0132] Table 2 Comparison of Experimental Results on Geometric Accuracy of DTU Dataset

[0133]

[0134] This invention not only excels in rendering effects but also in geometric reconstruction. We conducted experiments on the geometric accuracy of the depth map generated by the method of this invention, and the results are shown in Table 2. Here, GredSplatting (MVS) represents the depth result reconstructed by MVS, and GredSplatting (fusion) represents the depth result after multi-view fusion. It can be seen that after multi-view fusion, the geometric accuracy is improved under different view input conditions.

[0135] Table 3 Comparison of Rendering Accuracy Experimental Results for Real Forward-Facing and Tanks and Temples Datasets

[0136]

[0137] Table 3 presents the quantitative results for the Real Forward-Facing and Tanks and Temples datasets. All methods were pre-trained on the DTU training set and tested on the target dataset without further fine-tuning. Competitive performance is achieved by effectively fusing multi-view source features through the proposed PCDI and VCGR modules, driven by confidence scores to enhance reliable features while filtering out noise or inconsistent contributions. Experimental results demonstrate that the proposed method exhibits excellent rendering and generalization performance on all datasets, regardless of whether it involves two or three source views.

[0138] In summary, the GredSplatting method proposed in this invention introduces perceptual consistency constraints by combining a perceptual constraint-based deep inference module. This allows for the filtering and fusion of reliable information from multiple perspectives to generate accurate target viewpoint depth. Finally, the viewpoint feasibility Gaussian rendering module adaptively combines multi-view cost information based on the reliability of different perspectives to infer robust Gaussian primitive parameters, thereby gradually achieving accurate inference and robust rendering of Gaussian primitives for new perspectives. It exhibits stable rendering and reconstruction quality under varying source image counts and demonstrates good cross-scene generalization ability across multiple datasets. The proposed method shows significant application potential in handling the adaptive needs of unknown scenes.

[0139] The above description is merely a preferred embodiment of the present invention and does not limit the scope of protection of the present invention in any way. It should be emphasized that any minor modifications or optimizations made by those skilled in the art without departing from the basic idea of ​​the present invention should be considered within the scope of protection of the present invention.

Claims

1. A credibility-driven, scalable multi-view 3D Gaussian splashing method, characterized in that, Includes the following steps: Step 1: Collect the DTU, Real Forward-Facing, and Tanks and Temples datasets; Step 2: Divide the DTU dataset into training and validation sets. Select a portion of data from the Real Forward-Facing dataset and the Tanks and Temples dataset as an additional validation set. The model will be trained using only the DTU dataset. Step 3: Data preprocessing. Scale the images in the training and validation sets to a uniform size and normalize their pixel values ​​to the range [0, 1]. Calculate the distance between the dataset images and the camera pose of the image to be predicted based on the camera pose, sort them, and use the L nearest dataset images as input. Step 4: Build a neural network model for the new perspective synthesis task. The neural network model includes a new view initialization module based on multi-view stereo vision index, a perceptual constraint depth inference module, and a view confidence Gaussian rendering module. Step 5: Model training and inference, and save the model weights with the best performance; Step 6: Model performance evaluation and result analysis; The new view initialization module based on multi-view stereo vision index consists of two parts: the MVS depth inference part and the Gaussian meta-inference part. The specific implementation of the MVS deep inference part is as follows: 1-1. Using a feature pyramid network from the source view Extracting multi-scale features The three multi-scale features have dimensions of the source view size. , , Size is the input source view dimension. Multiscale features It was used to estimate the initial depth map; 1-2. Assume the depth plane in the view frustum space of the target view. Sampling was performed, among which ; 1-3. Multi-scale features of all source views By mapping differentiable homography distortion to the target view, the homography matrix from source view i to target view at sampling depth d is obtained. Simultaneously, obtain the multi-scale features of the i-th source view. Source feature I obtained after distortion; A multi-view cost volume is constructed using the distorted source features I and based on the variance metric. The cost volume is then fed into a 3D U-Net for regularization to infer the initial depth map of the target view. .

2. The credibility-driven scalable multi-view 3D Gaussian splashing method according to claim 1, characterized in that, The specific implementation of the Gaussian meta-inference part is as follows: 2-1.3D Gaussian point plotting method parameterizes a 3D scene into a set of Gaussian primitives, each Gaussian primitive consisting of attributes. Indicates; among which, It is the center of Gorsky's unit, and These represent the scaling factor and rotation factor, respectively. and Represents opacity and color factor; establishes a point-to-point mapping between Gaussian units and pixels; before performing Gaussian point rendering, a volumetric rendering based on a single depth-aware sampling point is performed, where the sampling point position along the ray is determined by the initial depth map. The radiance and volume density of the sampling points were determined from the multi-view aggregated features. Decoded from the middle; 2-2. Using the initial depth map After volumetric rendering, the obtained sampling points generate a blended color map under the target view. and depth map In addition, multi-view aggregation features Spatial augmentation is performed using 2D U-Net and combined with the fused color map to generate context-enhanced features. ; 2-3. Use independent multilayer perceptrons to enhance features from context. Decoding colors Opacity Scaling factor and rotation factor Meanwhile, the center of Gorsky's unit By using camera parameters to fuse depth maps Determined by back projection onto world space; 2-4. Rasterization-based rendering utilizes the decoded Gaussian primitives to generate an initial target view by averaging the color maps of Gaussian rendering and volumetric rendering.

3. The credibility-driven scalable multi-view 3D Gaussian splashing method according to claim 1, characterized in that, The perceptual constraint deep reasoning module selectively integrates multi-view cost volumes to suppress interference from features misaligned with the target view, while enhancing the contribution of highly consistent features to generate a reliable depth map consistent with the scene geometry.

4. The credibility-driven scalable multi-view 3D Gaussian splashing method according to claim 3, characterized in that, The perceptual constraint deep reasoning module is specifically implemented as follows: 3-1. A two-dimensional convolutional neural network is used to extract high-dimensional features from the initial target view synthesized in the initial rendering stage; context robustness is enhanced by aggregating multi-view aggregated features with high-dimensional features to obtain target view features. : 3-2. By constructing pairwise correlation cost volumes between each source view and the initial target view, the relationships between features are modeled individually; in the initial depth map Build a local depth search buffer around it Sampling is performed; Within this buffer, a set of depth hypothesis planes are sampled to map source view features to the initial target view via homography distortion; Multi-scale features of the i-th source view The source feature II obtained after distortion is paired with the target view feature to obtain the correlation cost volume. ; 3-3. Learning perceptual consistency score Under the guidance of relevant cost body and the distorted source features Adaptive fusion is performed to obtain the fused multi-view cost volume. ; In addition to the correlation cost encoding the similarity between the target view and the source views, a variance-based cost is added to encode the consistency among all source views. This enriches the final cost volume by integrating multi-view information, resulting in the final multi-view fusion cost volume. ; 3-4. Using a multi-scale 3D CNN network to process the perceptually constrained multi-view fusion cost volume Regularization is performed, and then probability weights are applied to the regularized features along the depth hypothesis plane to obtain a detailed depth map of the initial target view. .

5. The credibility-driven scalable multi-view 3D Gaussian splashing method according to claim 4, characterized in that, The aforementioned view confidence Gaussian rendering module utilizes full-resolution Gaussian meta-inference and combines the feature confidence of each view to eliminate over-reliance on finite cost volumes.

6. The credibility-driven scalable multi-view 3D Gaussian splashing method according to claim 5, characterized in that, The Gaussian rendering module for viewpoint credibility is implemented as follows: 4-1. Quantify the feature credibility of each source view relative to the target view by scoring perceptual consistency along the depth dimension. Perform pixel-wise max pooling to generate a confidence probability map. Subsequently, the confidence probability graph was normalized to ensure comparability among all views; 4-2. Correlation Cost Body Extended to the differential cost body The cascading results thus form the enhanced cost body. ; For each cost body After regularization using 3D U-Net, it is combined with the multi-scale features of the source view. Cascaded features and fused them using a 3D CNN to generate an enhanced feature body. Based on enhanced feature volume The set of Gaussian elements is used to infer the robustness of view-related noise; 4-3. Depth-aware volumetric rendering is used to assist in learning the color factors of Gaussian primitives. Radiance is independently derived from each cost volume, corresponding to a single source view; due to normalization reliability... This reflects the relative reliability of each source view relative to the target view, thus weighting and utilizing the combined volumetric rendering equations of reliability to generate a more refined color map blended under the target view. and depth map ; 4-4. In order to infer Gaussian units, all enhanced feature volumes Using the normalized confidence graph Adaptive fusion to obtain feature maps ; 4-5. Feature Map Enhancement with 2D U-Net improves spatial context consistency; the output feature map of 2D U-Net is compared with the color map output of volumetric rendering. The components are combined and then decoded into color factors by a multilayer perceptron. Opacity Scaling and rotation factor The Gaussian center is obtained from the output feature map of the 2D U-Net through independent multilayer perceptron decoding. Then, by analyzing the depth map of the fusion... The back projection onto the 3D world space is determined.

7. A trust-driven, scalable multi-view 3D Gaussian splashing system, characterized in that, The system includes the credibility-driven scalable multi-view 3D Gaussian splashing method as described in claim 1.

Images