New visual angle synthesis method for three-dimensional Gaussian sputtering based on double-visual-angle guidance

Abstract

The invention relates to the technical field of computer vision graphics and neural rendering, in particular to a new visual angle synthesis method of three-dimensional Gaussian sputtering based on double visual angle guidance, which comprises the following steps of: selecting a main visual angle pair and an auxiliary visual angle pair to perform joint optimization in each round of training, and performing attention enhancement on common-view anchor point features; meanwhile, multi-view projection consistency loss and multi-scale intensity loss are introduced, so that the global structure stability and the local texture detail quality are improved; through combined training of visual angle pairs with high overlap and moderate parallax, multi-visual-angle complementary information is fully utilized, and single-visual-angle offset is reduced; through common-view anchor point attention enhancement, explicitly modeling the relationship between anchor points, and improving the local texture and structure expression ability; anchor drift is inhibited through projection consistency loss, and details and a contrast structure are improved through multi-scale strength loss, so that rendering quality is improved and geometric consistency is enhanced; the problems of poor geometric consistency, unstable anchor points and detail artifacts caused by existing single-view training are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision graphics and neural rendering technology, and in particular to a novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering. Background Technology

[0002] Novel perspective synthesis tasks typically generate unobserved perspective images from multi-view input images. In recent years, 3D Gaussian Splatting (3DGS) has demonstrated outstanding performance in rendering quality and speed; however, its training paradigm often employs single-view supervision: each iteration randomly selects a perspective, and the parameters are updated using only that perspective to reconstruct the error. This strategy easily leads to excessive bias in model parameters towards the training perspective, causing oscillations, drift, and redundancy in the 3D Gaussian or anchor points under multi-view conditions. Ultimately, this results in detail artifacts and unreasonable floating ellipsoids in the rendering results, while the consistency of the underlying structure remains insufficient.

[0003] Furthermore, while anchor-based 3DGS methods (such as introducing anchors and neural Gaussians through voxel meshes and using MLP to predict neural Gaussian properties) can alleviate Gaussian overexpansion and improve structure distribution, if single-view training is still used, it is still difficult to fully utilize the complementary information from multiple views, especially under occluded, blurred, or low-quality views where supervision is insufficient.

[0004] In view of this, we propose a novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering to solve the existing problems. Summary of the Invention

[0005] The purpose of this invention is to provide a novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering, comprising the following steps:

[0007] S1. Scene initialization and anchor point representation construction: Obtain multi-view input images and camera parameters; initialize a 3D Gaussian based on sparse point cloud, or construct an anchor point set based on voxel mesh; each anchor point carries a feature vector and several offsets to generate a neural Gaussian; predict the properties of the neural Gaussian through a neural network and perform differentiable rendering;

[0008] S2, Viewpoint Pair Selection: In each round of training, for the current main viewpoint, auxiliary viewpoints are selected from the other viewpoints to perform main and auxiliary viewpoint matching.

[0009] S3: Dual-view joint training and co-view anchor labeling: Based on the selected primary and secondary view pairs, a training iteration is jointly performed; anchors or neural Gaussian sets that are commonly visible in both views are labeled for subsequent cross-view consistency constraints and feature enhancement.

[0010] S4: Co-view Anchor Point Attention Enhancement: The anchor point feature enhancement module aggregates the co-view anchor point feature vectors, uses an attention mechanism to model the deep dependencies between anchor points, and fuses the geometric and appearance information of the primary and secondary perspectives;

[0011] S5: Loss function construction and parameter update: Construct a total loss function that includes at least two types of losses: multi-view projection consistency loss and multi-scale intensity loss, and update the parameters.

[0012] S6: New Perspective Rendering Output: After training, use optimized anchor points or neural Gaussian parameters to perform differentiable rasterization rendering of the new perspective, and output the new perspective synthetic image.

[0013] Furthermore, in S1, the properties of the neural Gaussian include at least color, opacity, rotation, and scaling.

[0014] Furthermore, in S2, the selection principle for the primary and secondary viewpoints simultaneously satisfies the following: there is visible anchor point or point cloud overlap between the primary and secondary viewpoints, and there is parallax; and the viewpoint pair with the highest score can be selected through a scoring function of visibility correlation degree and viewpoint difference penalty, while filtering viewpoint pairs outside the set parallax range.

[0015] Furthermore, in S4, the anchor feature enhancement module only applies to the training optimization phase.

[0016] Furthermore, in S5, the multi-view projection consistency loss constrains the projection position of the same anchor point under the primary and secondary views.

[0017] Furthermore, in S5, multi-scale intensity loss converts the rendered image and the real image into a luminance image, aligns the luminance at multiple scales, and extracts the luminance edge gradient and calculates edge consistency.

[0018] Furthermore, in S1, after obtaining the set of multi-view input images and their camera intrinsic and extrinsic parameters, the projection operator for each viewpoint is calculated.

[0019] Furthermore, in S1, each anchor point contains a three-dimensional position and a learnable feature vector. Several learnable offset vectors are set for each anchor point, an attribute prediction network is constructed, and a differentiable renderer is initialized to perform Gaussian sputtering rasterization to obtain a synthetic image.

[0020] Furthermore, in S4, the co-visible anchor features are taken and input into the attention enhancement module to model the dependencies between anchors, thus obtaining enhanced features.

[0021] Furthermore, between S1 and S2, a visibility mask and viewpoint correlation matrix are constructed, and the viewpoint difference is calculated.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] This invention selects primary and secondary viewpoint pairs for joint optimization in each training round, enhances attention on co-view anchor point features, and introduces multi-view projection consistency loss and multi-scale intensity loss to improve global structural stability and local texture detail quality. Through joint training with highly overlapping and moderately disparate viewpoint pairs, it fully utilizes complementary information from multiple views to reduce single-view bias. By enhancing attention on co-view anchor points, it explicitly models the relationships between anchor points, improving the expressive power of local texture and structure. It suppresses anchor point drift through projection consistency loss and enhances detail and contrast structure through multi-scale intensity (brightness or edge) loss, thereby improving rendering quality and enhancing geometric consistency. This addresses the problems of poor geometric consistency, unstable anchor points, and detail artifacts caused by existing single-view training methods. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the overall process of a novel perspective synthesis method for three-dimensional Gaussian sputtering based on dual-view guidance, as described in this invention.

[0025] Figure 2 This is a schematic diagram illustrating the selection of viewpoint pairs in a novel viewpoint synthesis method based on dual-viewpoint guided 3D Gaussian sputtering according to the present invention. Detailed Implementation

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] A novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering includes the following steps:

[0029] S1. Scene initialization and anchor point representation construction: Obtain multi-view input images and camera parameters; initialize a 3D Gaussian based on sparse point cloud, or construct an anchor point set based on voxel mesh; each anchor point carries a feature vector and several offsets to generate a neural Gaussian; predict the properties of the neural Gaussian through a neural network and perform differentiable rendering; wherein, the properties of the neural Gaussian include at least color, opacity, rotation, and scaling.

[0030] S2. Viewpoint Pair Selection: In each round of training, for the current primary viewpoint, auxiliary viewpoints are selected from the remaining viewpoints to perform primary and auxiliary viewpoint matching. The selection principle for primary and auxiliary viewpoints simultaneously satisfies the following: there is a high degree of visible anchor points or point cloud overlap between the primary and auxiliary viewpoints, and there is an appropriate disparity to avoid information redundancy or excessive difficulty in achieving consistency. The viewpoint pair with the highest score can be selected through a scoring function of "visibility correlation + viewpoint difference penalty", while filtering viewpoint pairs with too small or too large disparity.

[0031] S3: Dual-view joint training and co-view anchor labeling: Based on the selected primary and secondary view pairs, a training iteration is jointly performed; anchors or neural Gaussian sets that are commonly visible in both views are labeled for subsequent cross-view consistency constraints and feature enhancement.

[0032] S4: Co-view Anchor Attention Enhancement: The anchor feature enhancement module aggregates the feature vectors of co-view anchors, uses an attention mechanism to model the deep dependencies between anchors, and fuses the geometric and appearance information of the primary and secondary perspectives to enhance the expressive power of anchors; the anchor feature enhancement module only applies to the training optimization phase;

[0033] S5: Loss Function Construction and Parameter Update: Construct a total loss function that includes at least two types of losses: multi-view projection consistency loss and multi-scale intensity loss, and update the parameters. Multi-view projection consistency loss constrains the projection positions of the same anchor point in the main and secondary views to meet consistency, so as to suppress multi-view oscillation and drift. Multi-scale intensity loss converts the rendered image and the real image into brightness (intensity) images, aligns the brightness at multiple scales, and uses operators such as Sobel to extract the brightness edge gradient and calculate edge consistency, so as to optimize both global structure and local details.

[0034] S6: New Perspective Rendering Output: After training, use optimized anchor points or neural Gaussian parameters to perform differentiable rasterization rendering of the new perspective, and output the new perspective synthetic image.

[0035] The working principle of the novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering in Embodiment 1 is as follows:

[0036] like Figure 1 As shown, after acquiring multi-view images and camera poses, view pair selection is performed. Common view anchors or overlap rates are calculated using visibility or overlap estimation. Then, target disparity scheduling is performed using disparity scoring. The first K view pairs (p, q) are selected, and neural Gaussian generation is performed using these selected view pairs. After initializing anchors, a common view anchor set (V) is obtained. p ∩V qThis process selects co-view anchors for view pairs and then guides anchor feature enhancement through dual views. Attention-based co-view anchor feature interaction is only used during the training phase. A neural Gaussian is generated, and Gaussian parameters, including position, scale, rotation, opacity, and color, are predicted from the enhanced anchor features. A differentiable Gaussian sputtering renderer generates rendering views p and q, and iterative optimization is performed through parameter updates to re-initialize the anchors. Multi-scale intensity loss and multi-view projection consistency loss are calculated using the rendering views to initialize the anchors. The multi-scale intensity loss converts the rendered image and the real image into a brightness map, aligning brightness at multiple scales. Simultaneously, operators such as Sobel are used to extract brightness edge gradients and calculate edge consistency. The perceptual structure is robust to low-texture and complex lighting conditions. The multi-view projection consistency loss obtains multi-view... Figure 1 To reduce drift or oscillation, a total loss is constructed that includes at least two types of losses: multi-view projection consistency loss and multi-scale intensity loss, and the parameters are updated. The key contributions of dual-view guided Gaussian sputtering are adaptive view pair selection for overlap and disparity, joint training of dual views with stable structures, and enhanced anchor representation and perceptual loss.

[0037] like Figure 2 As shown, the common visible set of the main view and the auxiliary view is obtained from the anchor point or Gaussian point. The disparity angle is obtained through the visibility overlap. Filtering rules are applied, and then the comprehensive score is calculated. The auxiliary view with the highest score is selected to obtain the selected view pair.

[0038] In the data and camera parameter preparation for S1: acquiring a set of multi-view input images and its camera internal parameters With external references , , ; Calculate the projection operator for each viewpoint Used to convert three-dimensional points Projected to pixel coordinates .

[0039] In the anchor set and neural Gaussian initialization of S1: the anchor set is constructed based on sparse reconstructed point cloud or voxel mesh. Each anchor point Includes three-dimensional position With learnable feature vectors : Set K learnable offset vectors for each anchor point. k=1,...,K, used to generate K neural Gaussian centers. ; Constructing an attribute prediction network For each Gaussian neural network, the input may contain ,in The distance from the anchor point to the center of the camera. To determine the viewing direction, the Gaussian color (c), opacity (α), rotation (q), and scale (s) are output and used for differentiable rendering. Initialize the differentiable renderer to perform Gaussian sputtering rasterization to obtain the composite image.

[0040] Between S1 and S2, the visibility mask and view correlation matrix are constructed, and the view difference is calculated.

[0041] In the construction of the visibility mask and viewpoint correlation matrix: for each viewpoint v, the anchor point visibility mask is determined based on the current anchor point or Gaussian distribution and the camera view frustum. , Indicates anchor point Visible anchor point index set from viewpoint v: ; Calculate the visibility overlap of any viewpoint pair (v, w): ;Will Write it into the correlation matrix C. .

[0042] In the calculation of viewpoint difference: the optical axis direction vector for each viewpoint is calculated from the extrinsic parameters, and the unit vector of the optical axis in the camera coordinate system is... Then, the direction of the optical axis in world coordinates is: ; Calculate the difference in viewpoints And used as a disparity metric.

[0043] In the selection of primary and secondary perspectives in S2: a primary perspective is selected in each iteration. ; Filter out view pairs that are too small or too large from the candidate viewpoints w to avoid information redundancy or difficulty in achieving consistency. ; Calculate a comprehensive score for the remaining candidates: ,in Let w be the target parallax angle (which can be set to 10°-20°), α and β be weights, and select the w with the highest score as the auxiliary viewpoint. .

[0044] In determining the common view anchor set in S3: Calculate the common view set. And record its anchor point index for subsequent cross-view constraints.

[0045] In the co-view anchor attention enhancement of S4: extract co-view anchor features. The attention enhancement module is used to model the dependencies between anchor points, resulting in enhanced features. , ; Use enhanced features for attribute prediction: for co-visual anchors : .

[0046] In dual-view rendering: using the current parameters, a composite image is obtained by rendering in both the primary and secondary viewpoints. , Take corresponding real images from the dataset as supervision. , .

[0047] In the loss function calculation of S5: reconstruction loss, pixel-level errors are calculated and summed for both viewpoints: Projection consistency loss, for common-view anchor points Calculate its projections in the two viewpoints and constrain their consistency: , , Multi-scale intensity loss converts the image into a brightness map. Construct a multi-scale pyramid (e.g., s∈{1,1 / 2,1 / 4}) and calculate... , , , , .

[0048] The above specific embodiments are merely several preferred embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering, characterized by the following steps: include: S1. Scene initialization and anchor point representation construction: Obtain multi-view input images and camera parameters; initialize a 3D Gaussian based on sparse point cloud, or construct an anchor point set based on voxel mesh; each anchor point carries a feature vector and several offsets to generate a neural Gaussian; predict the properties of the neural Gaussian through a neural network and perform differentiable rendering; S2, Viewpoint Pair Selection: In each round of training, for the current main viewpoint, auxiliary viewpoints are selected from the other viewpoints to perform main and auxiliary viewpoint matching. S3: Dual-view joint training and co-view anchor labeling: Based on the selected primary and secondary view pairs, a training iteration is jointly performed; anchors or neural Gaussian sets that are commonly visible in both views are labeled for subsequent cross-view consistency constraints and feature enhancement. S4: Co-view Anchor Point Attention Enhancement: The anchor point feature enhancement module aggregates the co-view anchor point feature vectors, uses an attention mechanism to model the deep dependencies between anchor points, and fuses the geometric and appearance information of the primary and secondary perspectives; S5: Loss function construction and parameter update: Construct a total loss function that includes at least two types of losses: multi-view projection consistency loss and multi-scale intensity loss, and update the parameters. S6: New Perspective Rendering Output: After training, use optimized anchor points or neural Gaussian parameters to perform differentiable rasterization rendering of the new perspective, and output the new perspective synthetic image.

2. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S1, the properties of a neural Gaussian include at least color, opacity, rotation, and scaling.

3. The novel perspective synthesis method based on dual-viewpoint guided 3D Gaussian sputtering according to claim 1, characterized in that, In S2, the selection principle of the primary and secondary viewpoints simultaneously satisfies the following: there is visible anchor point or point cloud overlap between the primary and secondary viewpoints, and there is parallax; and the viewpoint pair with the highest score can be selected through the scoring function of visibility correlation degree and viewpoint difference penalty, while filtering viewpoint pairs outside the set parallax range.

4. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S4, the anchor feature enhancement module only applies to the training optimization phase.

5. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S5, the multi-view projection consistency loss constrains the projection position of the same anchor point under the primary and secondary views.

6. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S5, multi-scale intensity loss transforms the rendered image and the real image into a luminance image, aligns the luminance at multiple scales, and extracts the luminance edge gradient and calculates edge consistency.

7. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S1, after obtaining the set of multi-view input images and their camera intrinsic and extrinsic parameters, the projection operator for each viewpoint is calculated.

8. The novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S1, each anchor point contains a 3D position and a learnable feature vector. Several learnable offset vectors are set for each anchor point. An attribute prediction network is constructed, and a differentiable renderer is initialized to perform Gaussian sputtering rasterization to obtain a synthetic image.

9. A novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: In S4, common anchor features are taken and input into the attention enhancement module to model the dependencies between anchors, thus obtaining enhanced features.

10. A novel perspective synthesis method based on dual-viewpoint guided three-dimensional Gaussian sputtering according to claim 1, characterized in that: Between S1 and S2, the visibility mask and view correlation matrix are constructed, and the view difference is calculated.

Images