A hierarchical mask-based gaussian depth regularization novel view synthesis method

Abstract

This invention discloses a novel viewpoint synthesis method based on Gaussian depth regularization using layered masks. It first constructs a layered depth optimization mechanism with strong foreground supervision and weak background constraints, and then introduces collaborative constraints on multi-view depth continuity and texture consistency to achieve stable optimization of 3D Gaussian parameters. This method effectively suppresses monocular depth noise propagation, improves 3D reconstruction accuracy and visual consistency, and balances training efficiency with engineering feasibility, making it suitable for applications such as virtual reality, digital twins, and 3D content generation. The innovation of this invention lies in the mask-guided layered supervision and multi-view collaborative optimization mechanism, which solves the problem of lack of structural constraints in existing 3DGS, representing a significant technological improvement. This invention is based on mature depth estimation and segmentation models, has a high degree of technological maturity, and possesses a solid foundation for implementation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a novel viewpoint synthesis method based on Gaussian depth regularization using hierarchical masks. Background Technology

[0002] Novel viewpoint synthesis techniques are an important research direction in computer vision and 3D reconstruction, with widespread demand in applications such as virtual reality, digital twins, and 3D content generation. In recent years, 3D Gaussian Splatting (3DGS) has attracted attention due to its efficient rendering capabilities and high visual quality. However, existing 3DGS methods mainly rely on multi-viewpoint color consistency for optimization. In realistic scenes with sparse viewpoints, complex occlusion, or mixed foreground and background, problems such as geometric blurring, depth drift, and texture discontinuity easily occur, limiting their practical application effectiveness.

[0003] To enhance geometric constraint capabilities, some existing techniques attempt to incorporate monocular depth estimation results as auxiliary supervision. However, due to the lack of an effective foreground region differentiation mechanism, monocular depth noise is indiscriminately introduced into the background region, thereby weakening model stability. Simultaneously, most methods fail to adequately model the depth continuity and texture consistency of the target region under multiple viewpoints, making it difficult to achieve high-precision new viewpoint synthesis while ensuring training efficiency. Furthermore, existing mask segmentation methods are independent of the 3DGS training process, and a collaborative constraint mechanism for 3D Gaussian optimization has not yet been formed. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a novel viewpoint synthesis method based on Gaussian depth regularization using layered masks. By applying strong supervision to the foreground target and weak constraints to the background region, and combining multi-view depth and texture consistency constraints, this method improves the geometric accuracy and visual consistency of the reconstructed 3D Gaussian model scene while maintaining good engineering feasibility and practical application value.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A novel viewpoint synthesis method based on Gaussian depth regularization using layered masks includes the following steps:

[0007] Step 1: Preprocessing and segmentation of multi-view image data

[0008] A multi-view image dataset of the target scene is acquired, and all acquired images undergo distortion correction and image size normalization preprocessing. Subsequently, the processed dataset is divided into a training set viewpoint for model training and a test set (or validation set) viewpoint for evaluating the model's synthesis performance.

[0009] Step 2: Acquiring Sparse Point Clouds and Camera Parameters

[0010] The Structure-from-Motion (SfM) algorithm is used to extract and match features from the input image in step 1, calculate and output the sparse SfM point cloud of the scene, and simultaneously obtain the camera parameters corresponding to the image. And export the parameters for the training set and the test set respectively, including:

[0011] Camera intrinsic parameters: These are parameters that describe the internal optical characteristics of the camera, including focal length and principal point, forming the camera intrinsic parameter matrix. It is used to establish the mapping relationship between the camera coordinate system and the image pixel coordinate system.

[0012] Extrinsic parameters: Parameters describing the camera's pose in the world coordinate system, including the rotation matrix. (Rotation) and transverse vector (Translation) is used to determine the spatial position and shooting direction of cameras from different viewpoints.

[0013] Step 3: Monocular Depth Prior Prediction

[0014] The training set images obtained in step 1 are input frame by frame into a pre-trained monocular depth estimation large model (e.g., DepthAnything v2) to predict the dense monocular depth prior map corresponding to each training viewpoint (denoted as ). This prior graph provides information about the relative depth of the scene.

[0015] Step 4: Foreground-aware layered mask extraction

[0016] For the multi-view images in the training set, perform the following operations to extract a layered weight mask that distinguishes the foreground from the background:

[0017] 1. Use semantically guided object detection models (such as GroundingDINO) to identify target objects, and use non-maximum suppression (NMS) and scale filtering strategies to remove redundant and erroneous bounding boxes.

[0018] 2. Input the filtered bounding boxes as prompts into the segmentation model (such as SAM) to generate the initial binary instance mask.

[0019] 3. Combine the monocular depth prior map from step 3 to set a depth threshold and remove erroneous instances belonging to the background; at the same time, calculate the Pearson correlation coefficient between masks under different viewpoints to remove mis-segmented regions that are inconsistent across viewpoints.

[0020] 4. The mask is then smoothed: Gaussian smoothing is applied to the preserved accurate foreground mask. To achieve hierarchical supervision, the weight of the foreground region is set to 1.0, and a smaller base weight (e.g., 0.05) is given to the background region. Finally, the hierarchical weight mask matrix corresponding to each viewpoint is output, denoted as M(i).

[0021] Step 5: Global scale alignment for monocular depth

[0022] Due to the monocular depth prior map obtained in step 3 Lacking an absolute physical scale, it needs to be aligned with the SfM point cloud:

[0023] 1. Project the 3D SfM point cloud obtained in step 2 onto the image plane of each training viewpoint using the camera's intrinsic and extrinsic parameters to obtain the sparse true depth map of the corresponding viewpoint, denoted as . .

[0024] 2. Extraction and For the effective overlapping pixels, the Random Sampling Consensus (RANSAC) algorithm combined with linear regression is used to estimate the global scale factor (denoted as s) and offset (denoted as t) that map from monocular depth to SfM depth.

[0025] 3. Using the calculated s and t, perform a preliminary analysis on the entire monocular depth map. A linear transformation is performed to calculate an aligned dense depth map with a uniform global scale, denoted as . The calculation formula is: .

[0026] Step 6: 3D Gaussian (3DGS) Scene Initialization

[0027] Using the spatial 3D coordinates of the sparse SfM point cloud obtained in step 2 as the initial value for the center position of the 3D Gaussian sphere, we initialize various learnable parameters of the 3D Gaussian Splatting (3DGS), including: the center position of the Gaussian sphere, covariance (representing 3D scale and rotation), spherical harmonic coefficients (representing viewpoint-dependent color), and opacity. Simultaneously, we set the total number of training iterations for network optimization and the viewpoint sampling strategy.

[0028] Step 7: Rendering process based on random mask (forward propagation)

[0029] In each training iteration, a training viewpoint is randomly selected as the current target viewpoint. Using the 3DGS model parameters from the current iteration, rendering is performed using differentiable rasterization technology.

[0030] 1. RGB Color Rendering: Combines the color and opacity of the Gaussian sphere along the rays, outputting a rendered RGB image of the current viewpoint, denoted as . .

[0031] 2. Random Depth Rendering: To prevent overfitting of the model to unaligned Gaussian spheres, a random dropout mechanism is introduced during depth rendering. With a certain probability (e.g., using Dropout), the contribution of some Gaussian spheres to the depth is randomly ignored, and the rendered depth map of the current viewpoint is output, denoted as . .

[0032] Step 8: Calculation of mask-guided hierarchical depth-supervised loss

[0033] Using the hierarchical weight mask M(i) extracted in step 4 and the aligned dense depth map calculated in step 5 The rendering depth generated in step 7 Perform supervision. Calculate the hierarchical depth supervision loss (denoted as...). ): When calculating the absolute error (L1 Loss), the weight mask M(i) is multiplied pixel by pixel. This operation allows the network to apply strong constraints (high weights) to foreground object regions and weak constraints (low weights) to blurred background regions during optimization, thereby maintaining the overall stability of the geometric structure.

[0034] Step 9: Calculation of Multi-view Depth Continuity Consistency Loss

[0035] To constrain the geometric consistency of the scene in 3D space:

[0036] 1. Select the current rendering viewpoint (source viewpoint) and a viewpoint that is spatially adjacent to it. (Target viewpoint). Obtained through depth rendering. and ;

[0037] 2. Obtain the depth map from the source viewpoint. By using the camera extrinsics of both cameras, the image is reprojected onto the target viewpoint's image coordinate system to obtain a reprojected depth map. ;

[0038] 3. Calculate the reprojected depth map Depth map of the target viewpoint itself The differences between them. In this process, Gaussian weights are introduced based on the distribution of reprojection errors, giving high weights to well-matched regions and suppressing abnormal regions caused by object occlusion or viewpoint blind spots. Finally, the multi-view depth consistency loss (denoted as ) is obtained. ).

[0039] Step 10: Calculation of Multi-view Texture Consistency Loss

[0040] To prevent texture overfitting due to sparse viewpoints, cross-viewpoint structural similarity constraints are applied to the rendered image:

[0041] 1. Rendering depth map using the source viewpoint And the camera's intrinsic and extrinsic parameters, rendering the source viewpoint image. Each pixel in the image is back-projected into three-dimensional space, and then projected and reprojected onto the image plane of the target viewpoint to obtain the transformed image.

[0042] 2. Perform structural similarity (SSIM) calculations between the transformed image and the actual image captured from the target viewpoint.

[0043] 3. Similar to step 9, an adaptive weighting mechanism is used to reduce interference from abrupt changes in illumination and occlusion areas, ultimately calculating the multi-view texture consistency loss (denoted as ). ).

[0044] Step 11: Calculation of RGB photometric reconstruction loss

[0045] Directly calculate the rendered RGB image output from step 7. The pixel-level color reconstruction error (typically including L1 distance and D-SSIM loss) between the actual captured images corresponding to this viewpoint, as the basic RGB supervised loss, is denoted as... .

[0046] Step 12: Total Loss Optimization and Parameter Update

[0047] The calculated loss functions are weighted and summed according to the set hyperparameter weights to construct the total loss function. ;

[0048] in This represents the overall loss value in the current iteration; This represents the RGB photometric reconstruction loss calculated in step 11, used to constrain the basic color accuracy; This represents the hierarchical depth supervision loss calculated in step 8, used to constrain the accuracy of the overall scene depth; This represents the multi-view depth consistency loss calculated in step 9, used to correct geometric inconsistencies under multiple views; This represents the multi-view texture consistency loss calculated in step 10, which is used to improve the smoothness and continuity of texture features across different viewpoints. , , These are pre-defined hyperparameter weights that control the relative importance of each loss function.

[0049] Using the calculated total loss Calculate the gradient and update all 3DGS model parameters initialized in step 6 using the backpropagation algorithm. Repeat steps 7 to 12 for iterative training until the loss value converges, and finally output the trained 3D Gaussian scene model.

[0050] Furthermore, step 4 is detailed below.

[0051] First, the Grounding DINO model is used to perform object detection on multi-view images under semantic guidance. Non-maximum suppression (NMS) is introduced to eliminate highly overlapping redundant bounding boxes, and a spatial scale operator is used to filter out small outlier noise bounding boxes. The optimized bounding boxes are then used as cues to input into the SAM model, thereby generating a binary foreground mask for the target.

[0052] Subsequently, to obtain a high-precision foreground object mask, the initially generated mask was pruned and smoothed. Depth values ​​were sampled for each mask instance using the depth map predicted by Depth-Anything-V2, and background object masks were removed by setting a depth threshold. Simultaneously, the consistency of the mask across multiple views was verified using the Pearson correlation coefficient, effectively filtering out redundant low-frequency noise caused by missegmentation and multiple segmentations. Finally, Gaussian smoothing was applied to the mask, and the background region weight was set to 0.05 to obtain the final mask M(i), which was then superimposed on the dense depth map for depth-supervised training.

[0053] Furthermore, step 7 is detailed below:

[0054] In each training iteration, a training viewpoint (and its spatially neighboring viewpoints) is randomly sampled. Using the current 3DGS model parameters, standard RGB color rendering and depth rendering based on random masks are performed respectively using differentiable rasterization techniques to obtain the rendered RGB image. With rendering depth map The specific dual-track rendering logic is as follows:

[0055] (1) RGB color rendering path (using the standard Alpha rendering mode):

[0056] To synthesize realistic new perspective images and calculate photometric errors, the system preserves the contribution of all activated Gaussian units within the view frustum. For any pixel p on the image plane, its rendered color value is obtained by performing traditional alpha blending on the set N of Gaussian units ordered by spatial depth on intersecting rays. :

[0057] (1);

[0058] in, This represents the color feature vector of the i-th Gaussian sphere (usually calculated from the spherical harmonic function based on the viewing angle); This represents the basic opacity parameter of the i-th Gaussian sphere; The transmittance of the i-th Gaussian sphere represents the proportion of light energy that remains unblocked after passing through all the Gaussian spheres in front of it before reaching the i-th Gaussian sphere.

[0059] (2) Random Depth Rendering Mode

[0060] Because traditional alpha depth rendering, when constrained by depth priors, can easily distort gradient loss and produce unstable geometric distributions due to misaligned Gaussian spheres, this invention changes the traditional alpha calculation mode in the depth rendering process by introducing a randomized discard mask.

[0061] For any camera view that needs to render a depth map Generate a binary mask with the same number of Gaussian ellipsoids. Each Gaussian ball is discarded with probability p, and then only the masked random subset i of Gaussian balls is preserved. Perform depth rendering, and render the result using random Gaussian depth rendering. Used for monocular depth prior supervision and multi-view depth consistency constraints for scenes.

[0062] (2);

[0063] in, Represents a specific pixel point on a two-dimensional image plane; This represents the generated binary mask set, containing the mask states of all Gaussian ellipsoids; N represents the set of Gaussian primitives that overlap at pixel p and participate in rendering. This represents the mask state value of the i-th Gaussian ball, which can be 0 or 1 (1 means keep, 0 means discard). Let represent the transmittance of the i-th Gaussian sphere, which represents the energy remaining after light passes through all the Gaussian spheres in front of it. This represents the opacity of the i-th Gaussian sphere. This represents the depth value of the center of the i-th Gaussian sphere along the optical axis in the current camera coordinate system.

[0064] Random depth supervision is achieved by adding a random mask M to the transparency of the Gaussian set G, thus obtaining... By using the rendering depth of a random Gaussian graph, geometric constraints are imposed on the scene to ensure that the correct Gaussian primitives maintain consistent positions across different random subsets.

[0065] Furthermore, step 8 is detailed below:

[0066] Sparse point clouds are obtained using Structure from Motion (SfM), and these 3D points are reprojected onto the corresponding 2D image planes by combining the camera intrinsic and extrinsic parameters for each viewpoint. This process generates depth maps projected from the sparse point clouds for each viewpoint. For each perspective, only consider The set of pixels with valid depth values As shown in formula (1), in The RANSAC linear regression method is used on the ensemble to calculate the monocular depth estimate. Depth to sparse SFM projection points Optimal scale factor s and translation bias t:

[0067] (3);

[0068] in Let s and t represent the variables that minimize the summation expression that follows (i.e., the sum of squares of alignment errors); Represents sparse SfM projection depth map The set of pixels with valid values ​​(note: it is a set of pixel locations, not the depth map itself); Represents a set A specific pixel in the image; This indicates the depth prior map of a single camera at the pixel level. Predicted depth at that location; This indicates the sparse SfM projection depth map at the pixel level. The true reference depth value at the location; s represents the global scale factor to be determined, used to correct the scaling error of monocular depth prediction; t represents the translation bias to be determined, used to correct the reference offset error of monocular depth prediction.

[0069] Then, based on the s and t obtained from formula (2), the dense depth prior map after global scale calibration is obtained in formula (3). :

[0070] (4);

[0071] in This is a depth map estimated from a monocular depth prior model. This calibration step preserves the density of the monocular predicted view depth while ensuring that all monocular predicted depth maps... Both can maintain scale consistency in 3DGS scenes, correcting the inherent scale ambiguity and baseline bias in monocular depth estimation.

[0072] Next, the calibrated dense depth map Depth supervision for 3DGS, calculation and The depth loss between , as in equation (5)

[0073] (5);

[0074] in, The mask representing the current viewpoint guides the hierarchical depth supervision loss value; It represents the set of all valid pixels on the current image plane (valid pixel field); express A single pixel within the set; The validity weight mask value at pixel p (i=p in M(i) in step four) is mainly used to assign high weight to foreground objects and low weight to background objects, while filtering out unreliable depth estimation regions. This represents the depth value at pixel p obtained from 3DGS rendering. This represents the true supervisory value of the scale-aligned calibrated depth map at pixel p; The denominator term for loss normalization represents the sum of all effective weights or effective pixels in the entire image. It is used to average the accumulated loss value across all pixels to maintain the magnitude of the loss during gradient optimization.

[0075] Furthermore, step 9 is detailed below:

[0076] First, use camera parameters to calculate the relationship with the camera. The camera with the closest Euclidean distance and the smallest angle. .

[0077] (6);

[0078] in express Camera position vector, Indicates distance to camera The set of the k nearest cameras is given by equation (7):

[0079] (7);

[0080] in, Let S represent the camera orientation vector; S represents the set of all cameras.

[0081] Then for the view Generate two different random masks And M(j), random Gaussian depth rendering is performed on the two viewpoints using formula (2) to obtain the rendering depth map. and .

[0082] Will View depth map reprojection to Calculate the reprojection depth from the viewpoint Robust reprojection loss. Considering the nonlinearity of camera motion and the problems of occlusion and projection distortion in the scene, directly optimizing the reprojection error is easily affected by outliers. Therefore, as shown in Equation (8), a Gaussian weighting mechanism based on the error magnitude is designed to evaluate the geometric consistency between views. This mechanism assigns higher weights to regions with high matching degree and smooth depth changes. This automatically suppresses the influence of occluded areas during the optimization process, and its calculation method is as shown in equation (8).

[0083] (8);

[0084] in These are Gaussian weights that reflect the confidence level of geometric consistency. Hyperparameters for controlling weight sensitivity. This represents the depth reprojection error at pixel p, which is the difference between the original rendered depth value from the target viewpoint and the depth value reprojected from the source viewpoint.

[0085] Finally, as shown in equation (9), the multi-view geometric consistency loss is calculated:

[0086] (9);

[0087] in, and These represent the depth observation value from the target's perspective and the depth value after reprojection from the source's perspective, respectively. To prevent numerical stability constants with a denominator of zero; For valid pixel regions; To calculate the robust loss function.

[0088] Furthermore, step 10 is detailed below:

[0089] Render the source view Ci image Projected onto target view On the image plane, a distorted image is obtained using bilinear sampling. Then, calculate. From the perspective of the target Rendered image Differences in structural similarity between them.

[0090] Similar to the weight design of the depth consistency constraint (as described in Equation (8)), an adaptive Gaussian weighting mechanism is used to allocate loss weights to enhance the robustness of the optimization. This mechanism assigns greater weights to regions with high texture matching, while automatically reducing the constraint strength for inconsistent regions caused by highlights, shadows, or occlusions. The texture consistency loss is defined as follows:

[0091] (10);

[0092] (11);

[0093] in, This represents the result of reprojecting the source view image to the target view image; This represents the structural difference loss at pixel p; This is a sensitivity hyperparameter for texture consistency. Using this method, the true texture details of the scene were successfully restored from multiple perspectives, significantly improving the reconstruction quality and rendering fidelity while ensuring accurate geometric structure.

[0094] The beneficial effects of this invention are as follows:

[0095] 1. First, addressing the issues of insufficient geometric constraints and easily blurred foreground structures in existing 3DGS methods, this invention introduces a mask-based hierarchical supervision mechanism. This mechanism applies strong depth constraints to the foreground region and weak constraints to the background region, allowing the model to focus on optimizing the geometric structure of the target region during training, effectively improving the accuracy and stability of foreground 3D reconstruction. Experimental results show that, under the same training conditions, the depth error in the foreground region can be reduced by more than 20%.

[0096] 2. Secondly, to address the problem that monocular depth prior noise can easily interfere with model training, this invention combines multi-view depth continuity and consistency constraints to jointly regularize the depth results under different views, suppressing the propagation of single-view depth outliers, which theoretically improves the robustness of depth supervision and reduces the occurrence of depth drift.

[0097] 3. Furthermore, to address the issue of inconsistent textures across multiple viewpoints, this invention introduces a multi-view texture consistency loss, which ensures that the color distribution of the 3D Gaussian remains continuous and stable under different viewpoints, thereby effectively reducing texture breakage and ghosting phenomena in new viewpoint synthesis and improving rendering quality.

[0098] 4. Furthermore, the method of this invention can be seamlessly integrated into existing SfM and 3DGS training processes without the need for additional hardware or manual annotation, and has good engineering feasibility and promotional application value. Attached Figure Description

[0099] Figure 1 This is a diagram of the algorithm framework of the present invention.

[0100] Figure 2 This is a framework diagram of the mask recognition and segmentation module described in this invention.

[0101] Figure 3 This is a framework diagram of the multi-view depth texture consistency constraint module. Detailed Implementation

[0102] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0103] like Figure 1 As shown, the present invention provides a novel viewpoint synthesis method based on Gaussian depth regularization using layered masks, comprising the following steps:

[0104] Step 1: Preprocessing and segmentation of multi-view image data

[0105] A multi-view image dataset of the target scene is acquired, and all acquired images undergo distortion correction and image size normalization preprocessing. Subsequently, the processed dataset is divided into a training set viewpoint for model training and a test set (or validation set) viewpoint for evaluating the model's synthesis performance.

[0106] Step 2: Acquiring Sparse Point Clouds and Camera Parameters

[0107] The Structure-from-Motion (SfM) algorithm is used to extract and match features from the input image in step 1, calculate and output the sparse SfM point cloud of the scene, and simultaneously obtain the camera parameters corresponding to the image. And export the parameters for the training set and the test set respectively, including:

[0108] Camera intrinsic parameters: Parameters describing the internal optical characteristics of the camera, including focal length and principal point, forming the camera intrinsic parameter matrix. It is used to establish the mapping relationship between the camera coordinate system and the image pixel coordinate system.

[0109] Extrinsic parameters: Parameters describing the camera's pose in the world coordinate system, including the rotation matrix. (Rotation) and transverse vector (Translation) is used to determine the spatial position and shooting direction of cameras from different viewpoints.

[0110] Step 3: Monocular Depth Prior Prediction

[0111] The training set images obtained in step 1 are input frame by frame into a pre-trained monocular depth estimation large model (e.g., DepthAnything v2) to predict the dense monocular depth prior map corresponding to each training viewpoint (denoted as ). This prior graph provides information about the relative depth of the scene.

[0112] Step 4: Foreground-aware layered mask extraction

[0113] like Figure 2 As shown, for multi-view images in the training set, the following operations are performed to extract a layered weight mask that distinguishes the foreground from the background:

[0114] 1. Use semantically guided object detection models (such as Grounding DINO) to identify target objects, and use non-maximum suppression (NMS) and scale filtering strategies to remove redundant and erroneous bounding boxes.

[0115] 2. Input the filtered bounding boxes as prompts into the segmentation model (such as SAM) to generate the initial binary instance mask.

[0116] 3. Combine the monocular depth prior map from step 3 to set a depth threshold and remove erroneous instances belonging to the background; at the same time, calculate the Pearson correlation coefficient between masks under different viewpoints to remove mis-segmented regions that are inconsistent across viewpoints.

[0117] 4. The mask is then smoothed: Gaussian smoothing is applied to the preserved accurate foreground mask. To achieve hierarchical supervision, the weight of the foreground region is set to 1.0, and a smaller base weight (e.g., 0.05) is given to the background region. Finally, the hierarchical weight mask matrix corresponding to each viewpoint is output, denoted as M(i).

[0118] Step 5: Global scale alignment for monocular depth

[0119] Due to the monocular depth prior map obtained in step 3 Lacking an absolute physical scale, it needs to be aligned with the SfM point cloud:

[0120] 1. Project the 3D SfM point cloud obtained in step 2 onto the image plane of each training viewpoint using the camera's intrinsic and extrinsic parameters to obtain the sparse true depth map of the corresponding viewpoint, denoted as . .

[0121] 2. Extraction and For the effective overlapping pixels, the Random Sampling Consensus (RANSAC) algorithm combined with linear regression is used to estimate the global scale factor (denoted as s) and offset (denoted as t) that map from monocular depth to SfM depth.

[0122] 3. Using the calculated s and t, perform a preliminary analysis on the entire monocular depth map. A linear transformation is performed to calculate an aligned dense depth map with a uniform global scale, denoted as . The calculation formula is: .

[0123] Step 6: 3D Gaussian (3DGS) Scene Initialization

[0124] Using the spatial 3D coordinates of the sparse SfM point cloud obtained in step 2 as the initial value for the center position of the 3D Gaussian sphere, we initialize various learnable parameters of the 3D Gaussian Splatting (3DGS), including: the center position of the Gaussian sphere, covariance (representing 3D scale and rotation), spherical harmonic coefficients (representing viewpoint-dependent color), and opacity. Simultaneously, we set the total number of training iterations for network optimization and the viewpoint sampling strategy.

[0125] Step 7: Rendering process based on random mask (forward propagation)

[0126] In each training iteration, a training viewpoint is randomly selected as the current target viewpoint. Using the 3DGS model parameters from the current iteration, rendering is performed using differentiable rasterization technology.

[0127] 1. RGB Color Rendering: Combines the color and opacity of the Gaussian sphere along the rays, outputting a rendered RGB image of the current viewpoint, denoted as . .

[0128] 2. Random Depth Rendering: To prevent overfitting of the model to unaligned Gaussian spheres, a random dropout mechanism is introduced during depth rendering. With a certain probability (e.g., using Dropout), the contribution of some Gaussian spheres to the depth is randomly ignored, and the rendered depth map of the current viewpoint is output, denoted as . .

[0129] Specifically as follows:

[0130] In each training iteration, a training viewpoint (and its spatially neighboring viewpoints) is randomly sampled. Using the current 3DGS model parameters, standard RGB color rendering and depth rendering based on random masks are performed respectively using differentiable rasterization techniques to obtain the rendered RGB image. With rendering depth map The specific dual-track rendering logic is as follows:

[0131] (1) RGB color rendering path (using the standard Alpha rendering mode):

[0132] To synthesize realistic new perspective images and calculate photometric errors, the system preserves the contribution of all activated Gaussian units within the view frustum. For any pixel p on the image plane, its rendered color value is obtained by performing traditional alpha blending on the set N of Gaussian units ordered by spatial depth on intersecting rays. :

[0133] (1);

[0134] in, This represents the color feature vector of the i-th Gaussian sphere (usually calculated from the spherical harmonic function based on the viewing angle); This represents the basic opacity parameter of the i-th Gaussian sphere; The transmittance of the i-th Gaussian sphere represents the proportion of light energy that remains unblocked after passing through all the Gaussian spheres in front of it before reaching the i-th Gaussian sphere.

[0135] (2) Random Depth Rendering Mode

[0136] Because traditional alpha depth rendering, when constrained by depth priors, can easily distort gradient loss and produce unstable geometric distributions due to misaligned Gaussian spheres, this invention changes the traditional alpha calculation mode in the depth rendering process by introducing a randomized discard mask.

[0137] For any camera view that needs to render a depth map Generate a binary mask with the same number of Gaussian ellipsoids. Each Gaussian ball is discarded with probability p, and then only the masked random subset i of Gaussian balls is preserved. Perform depth rendering, and render the result using random Gaussian depth rendering. Used for monocular depth prior supervision and multi-view depth consistency constraints for scenes.

[0138] (2);

[0139] in, Represents a specific pixel point on a two-dimensional image plane; This represents the generated binary mask set, containing the mask states of all Gaussian ellipsoids; N represents the set of Gaussian primitives that overlap at pixel p and participate in rendering. This represents the mask state value of the i-th Gaussian ball, which can be 0 or 1 (1 means keep, 0 means discard). Let represent the transmittance of the i-th Gaussian sphere, which represents the energy remaining after light passes through all the Gaussian spheres in front of it. This represents the opacity of the i-th Gaussian sphere. This represents the depth value of the center of the i-th Gaussian sphere along the optical axis in the current camera coordinate system.

[0140] Random depth supervision is achieved by adding a random mask M to the transparency of the Gaussian set G, thus obtaining... By using the rendering depth of a random Gaussian graph, geometric constraints are imposed on the scene to ensure that the correct Gaussian primitives maintain consistent positions across different random subsets.

[0141] Step 8: Calculation of mask-guided hierarchical depth-supervised loss

[0142] Using the hierarchical weight mask M(i) extracted in step 4 and the aligned dense depth map calculated in step 5 The rendering depth generated in step 7 Perform supervision. Calculate the hierarchical depth supervision loss (denoted as...). ): When calculating the absolute error (L1 Loss), the weight mask M(i) is multiplied pixel by pixel. This operation allows the network to apply strong constraints (high weights) to foreground object regions and weak constraints (low weights) to blurred background regions during optimization, thereby maintaining the overall stability of the geometric structure.

[0143] Specifically as follows:

[0144] Sparse point clouds are obtained using Structure from Motion (SfM), and these 3D points are reprojected onto the corresponding 2D image planes by combining the camera intrinsic and extrinsic parameters for each viewpoint. This process generates depth maps projected from the sparse point clouds for each viewpoint. For each perspective, only consider The set of pixels with valid depth values As shown in formula (2), in The RANSAC linear regression method is used on the ensemble to calculate the monocular depth estimate. Depth to sparse SFM projection points Optimal scale factor s and translation bias t:

[0145] (3);

[0146] in Let s and t represent the variables that minimize the summation expression that follows (i.e., the sum of squares of alignment errors); Represents sparse SfM projection depth map The set of pixels with valid values ​​(note: it is a set of pixel locations, not the depth map itself); Represents a set A specific pixel in the image; This indicates the depth prior map of a single camera at the pixel level. Predicted depth at that location; This indicates the sparse SfM projection depth map at the pixel level. The true reference depth value at the location; s represents the global scale factor to be determined, used to correct the scaling error of monocular depth prediction; t represents the translation bias to be determined, used to correct the reference offset error of monocular depth prediction.

[0147] Then, based on the s and t obtained from formula (3), the dense depth prior map after global scale calibration is obtained in formula (4). :

[0148] (4);

[0149] in This is a depth map estimated from a monocular depth prior model. This calibration step preserves the density of the monocular predicted view depth while ensuring that all monocular predicted depth maps... Both can maintain scale consistency in 3DGS scenes, correcting the inherent scale ambiguity and baseline bias in monocular depth estimation.

[0150] Next, the calibrated dense depth map Depth supervision for 3DGS, calculation and The depth loss between , as in equation (5)

[0151] (5);

[0152] in, The mask representing the current viewpoint guides the hierarchical depth supervision loss value; It represents the set of all valid pixels on the current image plane (valid pixel field); express A single pixel within the set; The validity weight mask value at pixel p (i=p in M(i) in step four) is mainly used to assign high weight to foreground objects and low weight to background objects, while filtering out unreliable depth estimation regions. This represents the depth value at pixel p obtained from 3DGS rendering. This represents the true supervisory value of the scale-aligned calibrated depth map at pixel p; The denominator term for loss normalization represents the sum of all effective weights or effective pixels in the entire image. It is used to average the accumulated loss value across all pixels to maintain the magnitude of the loss during gradient optimization.

[0153] Step 9: Calculation of Multi-view Depth Continuity Consistency Loss

[0154] Specifically as follows:

[0155] like Figure 3 As shown, first, using camera parameters, calculate the relationship between the camera and the camera. The camera with the closest Euclidean distance and the smallest angle. .

[0156] (6);

[0157] in express Camera position vector, Indicates distance to camera The set of the k nearest cameras is given by equation (7):

[0158] (7);

[0159] in, Let S represent the camera orientation vector; S represents the set of all cameras.

[0160] Then for the view Generate two different random masks And M(j), random Gaussian depth rendering is performed on the two viewpoints using formula (1) to obtain the rendering depth map. and .

[0161] Will View depth map reprojection to Calculate the reprojection depth from the viewpoint Robust reprojection loss. Considering the nonlinearity of camera motion and the problems of occlusion and projection distortion in the scene, directly optimizing the reprojection error is easily affected by outliers. Therefore, as shown in Equation (8), a Gaussian weighting mechanism based on the error magnitude is designed to evaluate the geometric consistency between views. This mechanism assigns higher weights to regions with high matching degree and smooth depth changes. This automatically suppresses the influence of occluded areas during the optimization process, and its calculation method is as shown in equation (8).

[0162] (8);

[0163] in These are Gaussian weights that reflect the confidence level of geometric consistency. Hyperparameters for controlling weight sensitivity; This represents the depth reprojection error at pixel p, i.e., the difference between the original rendered depth value from the target viewpoint and the depth value reprojected from the source viewpoint. Finally, as shown in equation (9), the multi-view geometric consistency loss is calculated:

[0164] (9);

[0165] in, and These represent the depth observation value from the target's perspective and the depth value after reprojection from the source's perspective, respectively. To prevent numerical stability constants with a denominator of zero; For valid pixel regions; To calculate the robust loss function.

[0166] Step 10: Calculation of Multi-view Texture Consistency Loss

[0167] To prevent texture overfitting due to sparse viewpoints, cross-viewpoint structural similarity constraints are applied to the rendered image:

[0168] Render the source view Ci image Projected onto target view On the image plane, a distorted image is obtained using bilinear sampling. Then, calculate. From the perspective of the target Rendered image Differences in structural similarity between them.

[0169] Similar to the weight design of the depth consistency constraint (as described in Equation (8)), an adaptive Gaussian weighting mechanism is used to allocate loss weights to enhance the robustness of the optimization. This mechanism assigns greater weights to regions with high texture matching, while automatically reducing the constraint strength for inconsistent regions caused by highlights, shadows, or occlusions. The texture consistency loss is defined as follows:

[0170] (10);

[0171] (11);

[0172] in, This represents the result of reprojecting the source view image to the target view image; This represents the structural difference loss at pixel p; This is a sensitivity hyperparameter for texture consistency. Using this method, the true texture details of the scene were successfully restored from multiple perspectives, significantly improving the reconstruction quality and rendering fidelity while ensuring accurate geometric structure.

[0173] Step 11: Calculation of RGB photometric reconstruction loss

[0174] Directly calculate the rendered RGB image output from step 7. The pixel-level color reconstruction error (typically including L1 distance and D-SSIM loss) between the actual captured images corresponding to this viewpoint, as the basic RGB supervised loss, is denoted as... .

[0175] Step 12: Total Loss Optimization and Parameter Update

[0176] The calculated loss functions are weighted and summed according to the set hyperparameter weights to construct the total loss function. (12);

[0177] in This represents the overall loss value in the current iteration; This represents the RGB photometric reconstruction loss calculated in step 11, used to constrain the basic color accuracy; This represents the hierarchical depth supervision loss calculated in step 8, used to constrain the accuracy of the overall scene depth; This represents the multi-view depth consistency loss calculated in step 9, used to correct geometric inconsistencies under multiple views; This represents the multi-view texture consistency loss calculated in step 10, which is used to improve the smoothness and continuity of texture features across different viewpoints. , , These are pre-defined hyperparameter weights that control the relative importance of each loss function.

[0178] Using the calculated total loss Calculate the gradient and update all 3DGS model parameters initialized in step 6 using the backpropagation algorithm. Repeat steps 7 to 12 for iterative training until the loss value converges, and finally output the trained 3D Gaussian scene model.

[0179] The core advantages of this patent are as follows:

[0180] 1. Improves rendering quality under extremely sparse perspectives, overcoming the over-smoothing problem of existing technologies. Experiments on standard feedforward scene datasets (such as NeRF-LLFF) using extremely sparse perspectives (e.g., 2-view, 5-view, and 12-view) demonstrate that the method of this invention significantly outperforms existing depth-based regularization methods (such as DepthRegGS) and other mainstream sparse perspective synthesis techniques (such as SparseGS and FSGS) in objective evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Perceptual Patch Similarity (LPIPS). Existing sparse perspective synthesis methods tend to perform aggressive smoothing in unknown regions (e.g., SparseGS) or lead to over-smoothing of geometry (e.g., FSGS), resulting in the loss of high-frequency background details. In contrast, this invention preserves more accurate geometry and appearance.

[0181] 2. Enhanced geometric stability and effective elimination of blur artifacts caused by traditional depth priors. This invention constructs a hierarchical depth supervision module, efficiently establishing a stable and accurate scene geometry under sparse viewpoint conditions. In qualitative experiments, this invention effectively mitigates the blurring effects caused by directly introducing traditional monocular depth priors. Through hierarchical constraints, this invention significantly reduces model reconstruction errors while maintaining the stability of the global geometric framework.

[0182] 3. Improved fidelity and visual realism of high-frequency textures. This invention demonstrates superior performance in scenes with rich textures (e.g., Fern) and complex occlusion (e.g., Orchids). Through a multi-view texture consistency constraint module, this invention effectively balances geometric accuracy and texture fidelity, successfully recovering subtle scene details. This results in excellent performance on the LPIPS (Perceptual Patch Similarity) metric, significantly improving the perceptual quality and realism of the generated images.

[0183] 4. Significantly improves the model's generalization ability and noise robustness under new perspectives, effectively mitigating overfitting. The biggest challenge in sparse perspective synthesis is that the model is prone to local overfitting to a limited number of input perspectives. This invention innovatively introduces a Gaussian depth rendering strategy based on random masks. Ablation experiments demonstrate that this mechanism effectively penalizes isolated Gaussian primitives lacking stable geometric support, thereby significantly reducing the risk of overfitting in sparse scenes. Furthermore, combined with multi-view geometric consistency constraints with a Dropout mechanism, this invention not only corrects the local geometric inconsistencies introduced by monocular priors but also effectively avoids overfitting of the model to local noise, greatly improving the model's geometric generalization ability and robustness.

[0184] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A novel viewpoint synthesis method based on Gaussian depth regularization using layered masks, characterized by the following steps: Step 1: Preprocessing and segmentation of multi-view image data; A multi-view image dataset of the target scene is acquired. All acquired images are preprocessed by distortion correction and image size unification. Then, the processed dataset is divided into a training set viewpoint for model training and a test set viewpoint for evaluating the model synthesis effect. Step 2: Acquisition of sparse point cloud and camera parameters; The structure-of-motion (SfM) algorithm is used to extract and match features from the input image in step 1, calculate and output the sparse SfM point cloud of the scene, and simultaneously obtain the camera parameters corresponding to the image. And export the parameters for the training set and the test set respectively, including: Camera intrinsic parameters: Parameters that describe the internal optical characteristics of a camera, including focal length and optical center, forming the camera intrinsic parameter matrix. It is used to establish the mapping relationship between the camera coordinate system and the image pixel coordinate system; Camera extrinsic parameters: Parameters describing the camera's pose in the world coordinate system, including the rotation matrix. and transverse vector It is used to determine the spatial position and shooting direction of cameras from different viewpoints; Step 3: Monocular depth prior prediction; The training set images obtained in step 1 are input frame by frame into the pre-trained monocular depth estimation model to predict the dense monocular depth prior map corresponding to each training viewpoint, denoted as . The prior graph provides information about the relative depth of the scene. Step 4: Foreground-aware layered mask extraction; For multi-view images in the training set, extract layered weight masks that distinguish between foreground and background; Step 5: Global scale alignment for monocular depth; Due to the monocular depth prior map obtained in step 3 Lacking an absolute physical scale, it needs to be aligned with the SfM point cloud:

1. Project the 3D SfM point cloud obtained in step 2 onto the image plane of each training viewpoint using the camera's intrinsic and extrinsic parameters to obtain the sparse true depth map of the corresponding viewpoint, denoted as . ; 2. Extraction and For the effective overlapping pixels, the global scale factor s and offset t of the monocular depth to SfM depth are estimated by combining the random sampling consensus algorithm with linear regression.

3. Using the calculated s and t, perform a preliminary analysis on the entire monocular depth map. A linear transformation is performed to calculate an aligned dense depth map with a uniform global scale, denoted as . The calculation formula is: ; Step 6: Initialize the 3D Gaussian scene; Using the spatial 3D coordinates of the sparse SfM point cloud obtained in step 2 as the initial value of the center position of the 3D Gaussian sphere, we initialize the learnable parameters of the 3D Gaussian representation, including: the center position of the Gaussian sphere, covariance, spherical harmonic coefficients, and opacity; at the same time, we set the total number of training iterations and the viewpoint sampling strategy for network optimization. Step 7: Rendering process based on random mask; In each training iteration, a training viewpoint is randomly selected as the current target viewpoint; the 3DGS model parameters under the current iteration are used for rendering through differentiable rasterization technology. Step 8: Calculation of mask-guided hierarchical depth-supervised loss; Using the hierarchical weight mask M(i) extracted in step 4 and the aligned dense depth map calculated in step 5 The rendering depth generated in step 7 Perform supervision; calculate the hierarchical depth supervision loss, denoted as... When calculating the absolute error, the weight mask M(i) is multiplied pixel by pixel; Step 9: Calculate the multi-view depth continuity consistency loss; Step 10: Calculate the multi-view texture consistency loss; To prevent texture overfitting due to sparse viewpoints, cross-viewpoint structural similarity constraints are applied to the rendered image. Step 11: Calculation of RGB photometric reconstruction loss; Directly calculate the rendered RGB image output from step 7. The pixel-level color reconstruction error between the actual captured image and the viewpoint is used as the basic RGB supervised loss, denoted as . ; Step 12: Total loss optimization and parameter update; The calculated loss functions are weighted and summed according to the set hyperparameter weights to construct the total loss function. ; in This represents the overall loss value in the current iteration; This represents the RGB photometric reconstruction loss calculated in step 11, used to constrain the basic color accuracy; This represents the hierarchical depth supervision loss calculated in step 8, used to constrain the accuracy of the overall scene depth; This represents the multi-view depth consistency loss calculated in step 9, used to correct geometric inconsistencies under multiple views; This represents the multi-view texture consistency loss calculated in step 10, which is used to improve the smoothness and continuity of texture features across different viewpoints. , , These are pre-defined hyperparameter weights that control the relative importance of each loss function; Using the calculated total loss Calculate the gradient and update all 3DGS model parameters initialized in step 6 using the backpropagation algorithm; repeat steps 7 to 12 for iterative training until the loss value converges, and finally output the trained 3D Gaussian scene model.

2. The novel viewpoint synthesis method based on Gaussian depth regularization using layered masks according to claim 1, characterized in that: Step 4 is as follows: First, the Grounding DINO model is used to perform target detection on multi-view images under semantic guidance. Non-maximum suppression is introduced to eliminate highly overlapping redundant bounding boxes, and spatial scale operators are used to filter out small outlier noise bounding boxes. The optimized bounding boxes are used as cues to input into the SAM model to generate the binary foreground mask of the target. Subsequently, the initially generated mask is pruned and smoothed; depth values ​​of each mask instance are sampled using the depth map predicted by Depth-Anything-V2, and background object masks are removed by setting a depth threshold; at the same time, the consistency of the mask under multiple views is verified by the Pearson correlation coefficient, effectively filtering out redundant low-frequency noise caused by missegmentation and multiple segmentation; finally, Gaussian smoothing is applied to the mask, and the weight of the background region is set to 0.05 to obtain the final mask M(i), which is then superimposed on the dense depth map for depth-supervised training.

3. The novel viewpoint synthesis method based on Gaussian depth regularization using layered masks according to claim 2, characterized in that: Step 7 is as follows: In each training iteration, a training viewpoint is randomly sampled; using the current 3DGS model parameters, standard alpha color rendering and depth rendering based on random masks are performed respectively through differentiable rasterization technology to obtain the rendered RGB image. With rendering depth map The specific dual-track rendering logic is as follows: (1) RGB color rendering path: To synthesize realistic new perspective images and calculate photometric errors, the system preserves the contribution of all activated Gaussian units within the view frustum. For any pixel p on the image plane, its rendered color value is obtained by performing traditional alpha blending calculations on the set N of Gaussian units ordered by spatial depth on intersecting rays. : （1）； in, This represents the color feature vector of the i-th Gaussian sphere; This represents the fundamental opacity parameter of the i-th Gaussian sphere; denoted by , the transmittance of the i-th Gaussian sphere, which physically represents the proportion of light energy remaining after passing through all the Gaussian spheres in front of it without being blocked before reaching the i-th Gaussian sphere. Introduce a random drop mask; For any camera view that needs to render a depth map Generate a binary mask with the same number of Gaussian ellipsoids. Each Gaussian ball is discarded with probability p, and then only the masked random subset i of Gaussian balls is preserved. Perform depth rendering, and render the result using random Gaussian depth rendering. Monocular depth prior supervision and multi-view depth consistency constraints for scene applications; （2）； in, Represents a specific pixel point on a two-dimensional image plane; This represents the generated binary mask set, containing the mask states of all Gaussian ellipsoids; N represents the set of Gaussian primitives that overlap at pixel p and participate in rendering. This represents the mask state value of the i-th Gaussian sphere, which can be either 0 or 1. Let represent the transmittance of the i-th Gaussian sphere, and let represent the energy remaining after the light passes through all the Gaussian spheres in front of it. Let represent the opacity of the i-th Gaussian sphere; This represents the depth value of the center of the i-th Gaussian sphere along the optical axis in the current camera coordinate system; Random depth supervision is achieved by adding a random mask M to the transparency of the Gaussian set G, and obtaining... By using the rendering depth of random Gaussians to impose geometric constraints on the scene, the correct Gaussian primitives can still maintain the consistency of their positions under different random subsets.

4. The novel viewpoint synthesis method based on Gaussian depth regularization using layered masks according to claim 3, characterized in that: Step 8 is as follows: By utilizing motion reconstruction structures to obtain sparse point clouds and combining them with camera intrinsic and extrinsic parameters from each viewpoint, these 3D points are reprojected onto the corresponding 2D image plane. This process generates depth maps projected from the sparse point clouds at each viewpoint. For each perspective, only consider The set of pixels with valid depth values As shown in formula (3), in The RANSAC linear regression method is used on the ensemble to calculate the monocular depth estimate. Depth to sparse SFM projection points Optimal scale factor s and translation bias t: （3）； in Let s and t be variables that minimize the summation expression that follows them; Represents the sparse SfM projection depth map A set of pixels with valid numerical values; Represents a set A specific pixel in the image; This indicates the depth prior map of a single camera at the pixel level. Predicted depth at that location; This indicates the sparse SfM projection depth map at the pixel level. The true reference depth value at the location; s represents the global scale factor to be determined, used to correct the scaling error of monocular depth prediction; t represents the translation deviation to be determined, used to correct the reference offset error of monocular depth prediction. Then, based on the s and t obtained from formula (3), the dense depth prior map after global scale calibration is obtained in formula (4). : （4）； in It is a depth map estimated by a monocular depth prior model; Next, the calibrated dense depth map Depth supervision for 3DGS, calculation and The depth loss between , as in equation (5) （5）； in, The mask representing the current viewpoint guides the hierarchical depth supervision loss value; It represents the set of all valid pixels in the current image plane; express A single pixel within the set; This represents the validity weight mask value at pixel p, used to assign high weight to foreground objects and low weight to background objects, while filtering out unreliable depth estimation regions. This represents the depth value at pixel p obtained from 3DGS rendering. This represents the true supervisory value of the scale-aligned calibrated depth map at pixel p; The denominator term for loss normalization represents the sum of all effective weights or effective pixels in the entire image. It is used to average the accumulated loss value across all pixels to maintain the magnitude of the loss during gradient optimization.

5. The novel viewpoint synthesis method based on Gaussian depth regularization using layered masks according to claim 4, characterized in that: Step 9 is as follows: First, use camera parameters to calculate the relationship with the camera. The camera with the closest Euclidean distance and the smallest angle. ; （6）； in express Camera position vector, Indicates distance to camera The set of the k nearest cameras is given by equation (7): （7）； in, Let S represent the camera orientation vector; S represents the set of all cameras. Then for the view Generate two different random masks And M(j), random Gaussian depth rendering is performed on the two viewpoints using formula (2) to obtain the rendering depth map. and ; Will View depth map reprojection to Calculate the reprojection depth from the viewpoint The robust reprojection loss is optimized; the reprojection error is easily affected by outliers; therefore, as shown in Equation (8), a Gaussian weighting mechanism based on the error magnitude is designed to evaluate the geometric consistency between views; this mechanism assigns higher weights to regions with high matching degree and smooth depth changes. Thus, the influence of the occluded area is automatically suppressed during the optimization process, and its calculation method is as shown in equation (8). （8）； in These are Gaussian weights that reflect the confidence level of geometric consistency. Hyperparameters for controlling weight sensitivity; This represents the depth reprojection error at pixel p, which is the difference between the original rendered depth value from the target viewpoint and the depth value reprojected from the source viewpoint. Finally, as shown in equation (9), the multi-view geometric consistency loss is calculated: （9）； in, and These represent the depth observation value from the target's perspective and the depth value after reprojection from the source's perspective, respectively. To prevent numerical stability constants with a denominator of zero; For valid pixel regions; To calculate the robust loss function.

6. The novel viewpoint synthesis method based on Gaussian depth regularization using layered masks according to claim 5, characterized in that: Step 10 is as follows: Render the source view Ci image Projected onto target view On the image plane, a distorted image is obtained using bilinear sampling. Then, calculate From the perspective of the target Rendered image Differences in structural similarity between them; To enhance the robustness of the optimization, an adaptive Gaussian weighting mechanism is used to allocate loss weights; the texture consistency loss is defined as follows: （10）； （11）； in, This represents the result of reprojecting the source view image to the target view image; This represents the structural difference loss at pixel p; This is a sensitivity hyperparameter for texture consistency.

Images