Robust three-dimensional Gaussian sputtering new view angle synthesis method in presence of interferent

Abstract

The invention discloses a robust three-dimensional Gaussian sputtering new view angle synthesis method in the presence of interferents. The method comprises the following steps: acquiring scene images of a plurality of view angles and corresponding camera parameters; based on the scene image and the camera parameters, carrying out three-dimensional scene modeling by utilizing three-dimensional Gaussian sputtering, and carrying out iterative optimization to construct a three-dimensional Gaussian model; and using the three-dimensional Gaussian model to render and generate a new view angle image. Interferents in a scene can be automatically identified and ignored, and manual labeling is not needed; by learning a neural decision boundary and introducing object perception, the accuracy of interferent identification is improved; a high-quality new view angle synthesis result is obtained in a scene with interferents, and meanwhile, good background reconstruction quality is kept; the method has good generalization and can process different types and numbers of interferents.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and graphics, and in particular to a robust three-dimensional Gaussian sputtering new viewing angle synthesis method in the presence of interference objects. Background Art

[0002] New perspective synthesis technology is widely used in virtual reality, autonomous driving, games and other fields. Existing methods such as Neural Radiance Field (NeRF) and 3D Gaussian sputtering can achieve high-quality new perspective synthesis, but they all rely on the assumption that the scene is static. However, in practical applications, there are often some dynamic objects in the scene that are not desired to be reconstructed, such as pedestrians, moving vehicles, etc. These objects are called interference objects.

[0003] Existing technologies have difficulty effectively handling scenes with distractors. If the effects of distractors are not considered, floating artifacts and blur will appear in the synthesized image. Manually labeling distractors is very time-consuming. Therefore, a method is needed to automatically identify and ignore distractors to obtain clear scene reconstruction results. Summary of the invention

[0004] According to an embodiment of the present invention, a robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference is provided, comprising the following steps:

[0005] Obtain scene images from multiple perspectives and corresponding camera parameters;

[0006] Based on the scene image and camera parameters, 3D scene modeling is performed using 3D Gaussian sputtering, and the 3D Gaussian model is constructed by iterative optimization.

[0007] Generate new perspective images using 3D Gaussian model rendering.

[0008] Furthermore, the three-dimensional scene modeling is performed using three-dimensional Gaussian sputtering, and the three-dimensional Gaussian model is constructed by iterative optimization, which includes the following steps:

[0009] 3D Gaussian sputtering modeling and rendering using scene description parameters;

[0010] Generate distractor masks to identify and ignore distractor regions in scene images;

[0011] Compute a loss function using distractor masks to evaluate the quality of the current scene description parameter settings;

[0012] Update the scene description parameters according to the loss function;

[0013] The above steps are repeated until the model converges or reaches the preset number of iterations to obtain the optimized three-dimensional Gaussian model.

[0014] Furthermore, the scene description parameters include: position, covariance matrix, opacity and spherical harmonic coefficients.

[0015] Furthermore, generating the distractor mask comprises the following steps:

[0016] Calculate the residual between the scene image and the current rendering result to obtain a residual image;

[0017] Optimize the residual image to obtain the feature image;

[0018] Classify each pixel in the feature image as an interferer or a non-interferer, and obtain pixel-level classification results;

[0019] Perform object segmentation on the scene image to obtain the object segmentation result;

[0020] Intersect the pixel-level classification results and the object segmentation results to obtain the distractor mask.

[0021] Furthermore, the optimization processing of the residual image includes median normalization processing, spatial smoothing processing, local aggregation processing and feature aggregation processing.

[0022] Furthermore, a logistic regression classifier is used to classify each pixel in the feature image as a distractor or a non-distractor.

[0023] Furthermore, the output formula of the logistic regression classifier is:

[0024]

[0025] in, is the output mask corresponding to each pixel in the image R; σ is the sigmoid activation function; W is the learnable weight matrix; ω3(R) is the feature obtained after a series of processing on the input image R; b is the learnable bias term.

[0026] Furthermore, the pre-trained SegmentAnything model is used to perform object segmentation on the scene image.

[0027] Furthermore, the formula for obtaining the interferer mask is:

[0028]

[0029] W(R)=∑(m∈I(M(R)))(1-m);

[0030] Where M(R) represents the set of object segmentation masks of image R, and each mask m represents an independent object in the image; is the predicted distractor mask in complement form; is the object mask m and the complement distractor mask The element-wise product of (Hadamard product); yes Jaccard index between and m; Used to filter object masks. Only when more than 40% of the pixels in the object mask are predicted to be distractors, the object will be included in I(M(R)). I(M(R)) is the condition that is the set of object masks; 1-m is the complement of object mask m; ∑(m∈I(M(R)))(1-m) is the sum of the complements of all object masks in I(M(R)); W(R) represents the final generated interferer mask.

[0031] Furthermore, the formula of the loss function is:

[0032]

[0033] in, is the 3D Gaussian rendering loss, which measures the difference between the rendered image R and the real image G after filtering with the mask W. represents the Hadamard product (element-wise product); λ is the regularization coefficient; ∑(1-W) is the sum of the complements of the mask W; L is the total mask loss, which is a combination of the 3D Gaussian rendering loss and the regularization term.

[0034] According to the robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interferers according to the embodiments of the present invention, interferers in the scene can be automatically identified and ignored without manual labeling; the accuracy of interferer identification is improved by learning neural decision boundaries and introducing object perception; high-quality new perspective synthesis results are obtained in scenes with interferers while maintaining good background reconstruction quality; it has good generalization and can handle different types and quantities of interferers.

[0035] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is an overall flow chart of a robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference according to an embodiment of the present invention;

[0037] Figure 2 A flowchart of using three-dimensional Gaussian sputtering to model a three-dimensional scene in the robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference according to an embodiment of the present invention, and iteratively optimizing and constructing a three-dimensional Gaussian model;

[0038] Figure 3The present invention is a flowchart of generating an interferer mask in a robust three-dimensional Gaussian sputtering new perspective synthesis method when interferers exist according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings to further illustrate the present invention.

[0040] First, combine Figures 1 to 3 A robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interferers according to an embodiment of the present invention is described, which can automatically identify and ignore interferers in the scene, thereby obtaining a high-quality new perspective image.

[0041] like Figures 1 to 3 As shown, the robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference objects of an embodiment of the present invention includes the following steps: obtaining scene images taken from multiple perspectives and corresponding camera parameters, some scene images contain some dynamic objects that do not want to be reconstructed, namely interference objects; based on the scene images and camera parameters, three-dimensional Gaussian sputtering is used to model the three-dimensional scene, and a three-dimensional Gaussian model is constructed by iterative optimization; a new perspective image is generated by rendering the three-dimensional Gaussian model, and the optimized three-dimensional Gaussian model can be used to render images of any new perspective.

[0042] Further, the three-dimensional scene modeling is performed using three-dimensional Gaussian sputtering, and the iterative optimization construction of the three-dimensional Gaussian model includes the following steps: the three-dimensional Gaussian sputtering modeling and rendering are performed using scene description parameters. Specifically, a set of three-dimensional Gaussian functions are used to describe the scene, and each Gaussian function is described by the scene description parameters. The scene description parameters include: position, covariance matrix, opacity and spherical harmonic coefficients, wherein the position describes the specific position of the Gaussian function in the three-dimensional space; the covariance matrix describes the shape and direction of the Gaussian function in the three-dimensional space; opacity is used to control the influence of the Gaussian function on the final rendering result, and plays a role in adjusting the light transmittance during the rendering process, thereby affecting the visualization effect of the scene; the spherical harmonic coefficients are used to describe the color attributes and global illumination effects of the Gaussian function; and the scene image is fitted by optimizing these parameters. In the optimization process, an interference mask is generated to identify and ignore the interference area in the scene image; the interference mask is used to calculate the loss function to evaluate the quality of the current scene description parameter setting; the scene description parameters are updated according to the loss function; the above steps are repeated until the model converges or reaches the preset number of iterations, and the optimized three-dimensional Gaussian model is obtained.

[0043] Furthermore, generating the interference mask includes the following steps: calculating the image residual: calculating the residual between the scene image and the current rendering result to obtain the residual image; residual processing: optimizing the residual image to obtain the feature image; neural decision boundary: using a logistic regression classifier to classify each pixel in the feature image as an interference or a non-interference object to obtain a pixel-level classification result. The output formula of the logistic regression classifier is: in, is the output mask, corresponding to each pixel in the image R; σ is the sigmoid activation function, which compresses the input to between 0 and 1, making the output suitable as a mask value; W is a learnable weight matrix; ω3(R) is the feature obtained after a series of processing on the input image R; b is a learnable bias term. This formula implements a learnable neural decision boundary for distinguishing distractor and non-distractor pixels. Through training, the model can automatically learn the optimal W and b parameters to more accurately identify distractors. The sigmoid function maps the linear combination Wω3(R)+b to a value between 0 and 1, which can be interpreted as the probability that the pixel belongs to a distractor. Generally, values ​​greater than 0.5 are considered distractors, and values ​​less than 0.5 are considered not distractors. The parameters of the classifier are optimized by backpropagation. Object perception: Use the pre-trained SegmentAnything model to segment the scene image and obtain the object segmentation results to provide object-level perception information; mask generation intersects the pixel-level classification results and the object segmentation results to obtain the distractor mask. Only when more than 40% of the pixels in an object are classified as distractors, the entire object will be marked as a distractor. Specifically:

[0044] Formula 1:

[0045] Formula 2: W(R) = ∑(m∈I(M(R)))(1-m);

[0046] Formula 1 is used to determine which image masks should be considered in the final distractor mask. M(R) represents the set of object segmentation masks of image R, and each mask m represents an independent object in the image. is the predicted distractor mask in complement form, i.e. Here W - (R) is the original distractor mask calculated from the residuals by logistic regression; is the object mask m and the complement distractor mask The element-wise product (Hadamard product) of is used for pixels predicted as distractors in the object mask; yes The Jaccard index between and m. The Jaccard index is a way to measure the similarity between two sets, defined as the size of the intersection of the two sets divided by the size of their union. In this case, it measures the proportion of pixels in the object mask that are predicted to be distractors; Used to filter object masks. Only when more than 40% of the pixels in the object mask are predicted to be distractors, the object will be included in I(M(R)). I(M(R)) is the condition that These masks are used to further refine the distractor mask to ensure that only true distractors are ignored while retaining as much static scene content as possible. In this way, I(M(R)) ensures that only objects that have significant overlap with the predicted distractor mask are considered in the final distractor mask, helping to improve the quality and accuracy of new view synthesis.

[0047] Formula 2 is used to generate the final distractor mask, which combines the original mask calculated from the residual and the object segmentation mask obtained by the pre-trained segmentation network. 1-m is the complement of the object mask m. If a pixel in m belongs to an object, then this pixel will be marked as 0 (not a distractor) in 1-m, otherwise it will be marked as 1 (a distractor); ∑(m∈I(M(R)))(1-m) is the sum of the complements of all object masks in I(M(R)), and all objects identified as distractors are merged into the final mask W(R). If the complement of any object mask marks a pixel as 1, then the final mask W(R) will also mark this pixel as 1; W(R) represents the final generated distractor mask. In this way, the final distractor mask W(R) can accurately identify distractors in the image while retaining as much static scene content as possible. This mask is subsequently used in the loss function to ignore the impact of distractors on the 3D scene reconstruction, thereby improving the quality of new view synthesis.

[0048] This formulation leverages a pre-trained segmentation network to improve mask accuracy, ensuring that only true distractors are ignored and not parts of the static scene, improving the robustness and accuracy of synthesis from new perspectives.

[0049] Furthermore, the optimization process of the residual image includes median normalization, spatial smoothing, local aggregation and feature aggregation. Specifically: median normalization reduces the global brightness deviation in the image by subtracting the median of the residual of each color channel from the residual of the channel; spatial smoothing applies a 3×3 mean filter to smooth the normalized residual to reduce noise and small local fluctuations; local aggregation aggregates the smoothed residual in a larger local area (such as a 16×16 neighborhood) to further reduce noise and enhance the coherence of local features, which helps to more accurately identify the interference area in subsequent steps; feature aggregation adds the normalized residual, the smoothed residual and the locally aggregated residual to form a feature map containing multi-level information. The specific formula is as follows:

[0050] Median Normalization:

[0051] Spatial Smoothing:

[0052] Partial polymerization:ω2(R8(R)) (i) =(1 / 16 2 )*Σ(s∈R 16 (R))ω1(s) (i) ;

[0053] Feature Aggregation:

[0054] Where R represents the image, i represents the color channel; ε(R) (i) represents the residual of the i-th channel, and is centered by taking the median of the residual of each channel; is the centered residual, which reduces the impact of outliers by subtracting the median from the original residual; B 3×3 represents a 3×3 mean filter, using a 3×3 box kernel to capture local smoothness and improve local continuity; R N (R) describes an N×N neighborhood around R, ω2(R8(R)) (i) is the result of aggregating the ω1 values ​​of all 8×8 blocks in a 16×16 neighborhood to capture more continuous behavior of the interferer; ω3(R) (i)is the final mask value, indicating whether the pixel of the i-th channel may be an interferer. The core idea of ​​the above formula is to combine information from different sources (original residual, residual after local smoothing, and aggregated information of neighborhood blocks) to obtain a more comprehensive interferer mask. Such a mask can more accurately identify anomalies in the image caused by interferers, so that these parts can be ignored in subsequent 3D scene reconstruction to improve rendering quality; it can dynamically adapt to different interferers and scenes because it is not based on a fixed threshold, but on patterns learned from the data. This makes the method very robust when dealing with various complex scenes.

[0055] Furthermore, when calculating the reconstruction loss, the distractor mask is used to ignore the areas identified as distractors. Specifically, the loss function can be expressed as: Represents the mask loss function used in the training procedure, which combines the 3D Gaussian rendering loss and the regularization term. It is as follows:

[0056] W is the interferer mask (0 for interferer, 1 for non-interferer), is the 3D Gaussian rendering loss, which measures the difference between the rendered image R and the real image G after filtering with the mask W. Represents the Hadamard product (element-wise product), which is used to apply the mask to the image; λ is the regularization strength, a hyperparameter that controls the impact of the regularization term and helps balance the 3D Gaussian rendering loss and the regularization term; ∑(1-W) is the sum of the complement of the mask W, 1-W represents the distractor pixels in the mask, and summing these pixels provides a measure of the number of distractors in the mask; L is the total mask loss, which is a combination of the 3D Gaussian rendering loss and the regularization term. This loss function is used to optimize the 3D Gaussian rendering model and the distractor mask during training. The purpose of the mask loss function is to ensure that the mask can effectively identify and ignore distractors while minimizing the difference between the rendered image and the real image. The regularization term helps prevent the mask from being too complex or overfitting the training data, thereby improving the model's generalization ability in synthesizing new perspectives.

[0057] Example: This method is applied to the reconstruction of indoor scenes containing moving objects. The input includes 63 training images and 202 test images, where the training images contain various household items as interference objects. The NVIDIA RTX 3090 graphics card is used for training and iterates 30,000 times. In terms of indicators such as PSNR (peak signal-to-noise ratio), SSIM (structural similarity index) and LPIPS (image feature similarity), this method is superior to the existing three-dimensional Gaussian sputtering method in static scenes. The experimental results are as follows:

[0058] method PSNR SSIM LPIPS Static 3D Gaussian sputtering 26.53 0.87253 0.1568 This method 28.39 0.8875 0.1321

[0059] Compared with the static 3D Gaussian sputtering method, this method can effectively remove artifacts caused by interferences while maintaining good background reconstruction quality. This method can also handle interferences of different proportions. Experiments show that even when 80% of the images contain interferences, this method can still maintain stable reconstruction quality. In the process of increasing the proportion of interferences from 0% to 100%, the LPIPS index of this method only increases from 0.14 to 0.16, while the static Gaussian sputtering method increases from 0.159 to 0.182. In addition, this method can also maintain good performance in clean scenes without interferences. In three different clean scene tests, the average PSNR of this method is 30.14, which is comparable to 30.43 of the static Gaussian sputtering method.

[0060] Above, refer to Figures 1 to 3 A robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interferers according to an embodiment of the present invention is described, which can automatically identify and ignore interferers in the scene without manual labeling; the accuracy of interferer identification is improved by learning neural decision boundaries and introducing object perception; high-quality new perspective synthesis results are obtained in scenes with interferers while maintaining good background reconstruction quality; it has good generalization and can handle different types and numbers of interferers.

[0061] It should be noted that, in this specification, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0062] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference, characterized in that: The following steps are included: Obtain scene images from multiple perspectives and corresponding camera parameters; Based on the scene image and the camera parameters, three-dimensional scene modeling is performed using three-dimensional Gaussian sputtering, and a three-dimensional Gaussian model is constructed by iterative optimization; The three-dimensional Gaussian model is used to render and generate an image with a new viewing angle.

2. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 1, characterized in that: The method of using three-dimensional Gaussian sputtering to perform three-dimensional scene modeling and iteratively optimizing and constructing a three-dimensional Gaussian model comprises the following steps: 3D Gaussian sputtering modeling and rendering using scene description parameters; generating a distractor mask to identify and ignore distractor regions in the scene image; Compute a loss function using distractor masks to evaluate the quality of the current scene description parameter settings; Updating the scene description parameters according to the loss function; The above steps are repeated until the model converges or reaches the preset number of iterations to obtain the optimized three-dimensional Gaussian model.

3. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 2, characterized in that: The scene description parameters include: position, covariance matrix, opacity and spherical harmonic coefficients.

4. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 2 or 3, characterized in that: The generating of the interferer mask comprises the following steps: Calculating the residual between the scene image and the current rendering result to obtain a residual image; Optimizing the residual image to obtain a feature image; Classifying each pixel in the feature image as an interference object or a non-interference object to obtain a pixel-level classification result; Performing object segmentation on the scene image to obtain an object segmentation result; The pixel-level classification result and the object segmentation result are intersected to obtain an interference mask.

5. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 4, characterized in that: The optimization process of the residual image includes median normalization process, spatial smoothing process, local aggregation process and feature aggregation process.

6. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 4, characterized in that: A logistic regression classifier is used to classify each pixel in the feature image as an interferer or a non-interferer.

7. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 6, characterized in that: The output formula of the logistic regression classifier is: in, is the output mask corresponding to each pixel in the image R; σ is the sigmoid activation function; W is the learnable weight matrix; ω3(R) is the feature obtained after a series of processing on the input image R; b is the learnable bias term.

8. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 4, characterized in that: Use the pre-trained SegmentAnything model to perform object segmentation on the scene image.

9. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 4, characterized in that: The formula for obtaining the distractor mask is: W(R)=∑(m∈I(M(R)))(1-m); Where M(R) represents the set of object segmentation masks of image R, and each mask m represents an independent object in the image; is the predicted distractor mask in complement form; is the object mask m and the complement distractor mask The element-wise product of (Hadamard product); yes Jaccard index between and m; Used to filter object masks. Only when more than 40% of the pixels in the object mask are predicted to be distractors, the object will be included in I(M(R)). I(M(R)) is the condition that is the set of object masks; 1-m is the complement of object mask m; ∑(m∈I(M(R)))(1-m) is the sum of the complements of all object masks in I(M(R)); W(R) represents the final generated interferer mask.

10. The robust three-dimensional Gaussian sputtering new perspective synthesis method in the presence of interference as claimed in claim 2, characterized in that: The formula of the loss function is: L=LGS(RοW,GοW)+λ*∑(1-W); Among them, LGS(RοW,GοW) is the 3D Gaussian rendering loss, which measures the difference between the rendered image R and the real image G after filtering with mask W, ο represents the Hadamard product (element-wise product); λ is the regularization coefficient; ∑(1-W) is the sum of the complement of mask W; L is the total mask loss, which is a combination of the 3D Gaussian rendering loss and the regularization term.

Images