Gaussian splatting with gradient-based pruning and semantically aware-robust optimization

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a 3D representation of a scene. In particular, one of the methods includes obtaining a plurality of training images of a scene; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps, updating, using rendered images rendered using the set of Gaussians and corresponding training images, the set of Gaussian distributions. As part of the updating, gradient-based pruning, semantically-aware optimization, or both can be performed.

Description

[0001] GAUSSIAN SPLATTING WITH GRADIENT-BASED PRUNING AND SEMANTICALLY-AWARE ROBUST OPTIMIZATION

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims priority to Application Serial No. 20240100430. filed on June I E 2024, with the Hellenic Industrial Property Organisation. The contents of the foregoing application are hereby incorporated in their entirety.

[0004] BACKGROUND

[0005] This specification relates to learning a three-dimensional (3D) representation of a scene from a set of images of the scene.

[0006] A 3D representation of a scene is data that represents 3D properties of the scene. 3D representations can represent a scene in a variety of ways.

[0007] For example, Neural Radiance Field (NeRF) techniques encode information about the scene in the trained parameters of one or more multi-layer perceptrons (MLPs).

[0008] As another example, point clouds represent the scene as a collection of 3D points.

[0009] As yet another example, meshes represent a 3D scene as vertices and edges that define the faces of the mesh.

[0010] SUMMARY

[0011] This specification describes a system implemented as computer programs on one or more computers in one or more locations that learns a 3D representation of a scene from a set of images of the scene. In particular, the 3D representation of the scene is represented as a set of Gaussian distributions, e.g., 3D anisotropic Gaussian distributions (“splats”).

[0012] Throughout this specification, a ‘‘scene” can refer to, e.g., a real-world environment, or a simulated environment (e.g.. a simulation of real-world environment, e g., such that the simulated environment is a synthetic representation of a real-world scene).

[0013] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0014] 3D Gaussian Splatting (3DGS) has emerged as a promising technique for 3D reconstruction due to its highly efficient training and inference, enabling many real-time applications.

[0015] However, most practical real-world captures, i.e., sets of photos of real-world scenes that can be used to generate a 3D representation, include transient distractors such as people, pets, or shadows that violate 3DGS’s fundamental inter-view consistency assumption, reducing reconstruction quality, i.e., reducing the quality’ of the final 3D representations generated by the 3DGS technique. That is, existing 3DGS techniques struggle to accurately represent a 3D scene when the training images used to leam the 3D representation include occlusions, dynamic objects, or other transient distractors.

[0016] This specification describes various techniques that improve the quality of the final 3D representations, e.g., when the training images used to leam the 3D representation include transient distractors, relative to existing 3DGS techniques.

[0017] For example, this specification describes how to, instead of or in addition to using gradient-based pruning, employ semantically-aware optimization in order to account for distractors or transient effects in the set of the training images, i.e., to account for the portions of certain training images potentially depicting dynamic or noisy aspects of the scene. When performing semantically-aware optimization, at each update step, the system “masks ouf’ pixels that have been determined to be outlier pixels from contributing to the update at the update step, where an outlier pixel is a pixel that depicts a portion of the scene that is dynamic, noisy, or otherwise transient. Thus, by employing semantically-aware optimization as described in this specification, the system can effectively account for certain training images potentially depicting dynamic or noisy aspects of the scene, i.e., can still generate high-quality, accurate 3D representations even when training images depict dynamic or noisy aspects of the scene that should not be reflected in the generated 3D representations.

[0018] As another example, this specification describes how to employ gradient-based pruning when updating the Gaussian distributions across the 3DGS update steps. Employing gradient-based pruning can result in improved computational efficiency during the generation process (whether or not the images include transient distractors). In particular, in this example, at each update step, the system can determine whether a pruning criterion is satisfied. When the pruning criterion has been satisfied, the system determines whether to remove any of the Gaussian distributions from the set based on respective utilization scores for each of the Gaussian distributions in the set. Generally, the respective utilization score for each of the Gaussian distributions measures how much the Gaussian distribution has contributed to rendered images that have been rendered since the preceding update step at which the pruning criterion was satisfied. For example, the respective utilization score can be based on. for each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, a gradient of rendered colors for pixels in the rendered image with respect to a projected position of the Gaussian distribution for the rendered image. This pruning technique is effective at pruning unnecessary Gaussian distributions and also achieves excellent compression in terms of distribution count with equivalent reconstruction quality. That is, compared to existing techniques for pruning Gaussian distributions while learning the representation, this pruning technique achieves superior compression in terms of particle count with equivalent or better reconstruction quality’, significantly improving the computational efficiency of generating the final 3D representation. In other words, because this technique more effectively prunes Gaussian distributions during generation, this techniques allows generating more accurate reconstructions while storing fewer Gaussian distributions during the reconstruction process and as part of the final representation, resulting in a more computationally efficient and compressive process and representation.

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0020] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0021] BRIEF DESCRIPTION OF DRAWINGS

[0022] FIG. 1 shows an example scene representation system.

[0023] FIG. 2 is a flow diagram of an example process for generating a 3D representation of a scene.

[0024] FIG. 3 is a flow’ diagram of an example process for generating a 3D representation of a scene when the sy stem makes use of gradient-based pruning.

[0025] FIG. 4 is a flow diagram of an example process for generating a 3D representation of a scene when the system makes use of semantically-aware optimization.

[0026] FIG. 5 is a flow’ diagram of another example process for generating a 3D representation of a scene when the system makes use of semantically-aware optimization.

[0027] FIG. 6 shows an example of the performance of the described techniques.

[0028] FIG.7 shows an example of the performance of the described techniques when making use of gradient-based pruning.

[0029] Like reference numbers and designations in the various drawings indicate like elements.

[0030] DETAILED DESCRIPTION FIG. 1 shows an example scene representation system 100. The scene representation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0031] The system 100 learns a 3D representation 150, i.e., generates a final 3D representation 150, of a scene in an environment from a set of training images 102 of the scene.

[0032] For example, the images can be real-world images captured by one or more cameras of a real-world environment that includes the scene.

[0033] In another example, the images can be synthetic images generated of a scene based on sensor data of a real-world environment captured by one of more sensors, where the sensor data may not be in the form of an image. For example, the sensor data may describe the positions and orientations of objects having known shape. In both these examples, the images characterize the real-world environment.

[0034] As another example, the images can be synthetic images generated of a scene in a simulated environment.

[0035] As will be described in more detail below, the representation 150 represents the scene as a set of Gaussian distributions 140, e.g., 3D anisotropic Gaussian distributions (“splats”). The set of Gaussian distributions can be used to render an image of the scene from an arbitrary camera poses.

[0036] After generating the final representation 150, the system 100 can use the final representation 150 for any of a variety of purposes.

[0037] For example, after generating the final 3D representation 150 of the scene, the system 100 can receive data specifying a new camera pose and generate a new image of the scene that appears to be taken by a camera having the new camera pose using the final 3D representation 150 of the scene.

[0038] As another example, after generating the final 3D representation 150 of the scene, the system 100 can use the final 3D representation 150 of the scene to perform 3D modeling of the scene, e.g. using a 3D printer to generate a real-world object which is a model of the scene.

[0039] As another example, the system 100 can be part of a computer graphics pipeline and the final 3D representation 150 can be provided to another component of the pipeline, e.g., to be included in an augmented reality (AR) or virtual reality (VR) environment, or to be included in another computer graphics application, e.g., a video game or a video that incorporates computer graphics.

[0040] As another example, after generating the final 3D representation 150 of the scene, the system 100 can control a robot (e.g. a real-world electromechanical robot which interacts with a real-world environment that includes the scene) based on the final 3D representation 150 of the scene. For example, the system 100 can use the 3D representation 150 to generate a simulation of the scene and can then use the simulation for planning during the control of the robot, e.g., using model predictive control (MPC) or another technique. For example, the system 100 can use the 3D representation 150 to select actions for the robot to perform, e.g. to perform a task in the real-world environment.

[0041] As another example, after generating the final 3D representation 150 of the scene, the system 100 can train a control policy for a robot using the final 3D representation 150 of the scene. For example, the system 100 can use the 3D representation 150 to generate a simulation of the scene and can then use the simulation for simulating the impact of actions performed by the robot during training, e.g., through reinforcement learning. The trained control policy may be used to select actions for a real-world electromechanical robot to perform in a real -world environment, e.g. to perform a navigation and / or manipulation task in the real-world environment.

[0042] As another example, after generating the final 3D representation 150 of the scene, the system can generate images of the scene using the final 3D representation 150 of the scene and use the generated images to train a machine learning model to perform an image processing task, e.g., to train a machine learning model that performs image classification (e.g. assigning an image of the scene to one of a plurality of categories; for example, the images may be views of the scene from a variety of camera poses, and the image classification may be used to classify a given image of the scene into a corresponding one of plurality of possible ranges for the camera pose; as another example, the scene may include one or more objects, and the image classification may be used to classify a given image into one or more of a plurality of object classes that identify the class of object depicted in the image) or that performs image generation, e.g., image generation conditioned on some conditioning input (such as an image which shows how an environment - e.g. a real-world environment - including the scene would be modified by an influence on the environment characterized by the conditioning input; e.g. how the scene would appear if a lighting source of the environment were changed by an amount characterized by the conditioning input). While the above description describes determining a 3D representation of a scene from images of the scene, the described techniques can be used to generate any of a variety of representations. For example, the described techniques can be used for molecular reconstruction, i.e., to reconstruct the 3D structure of a molecule. As another example, the described techniques can be used for volumetric image reconstruction, i.e., to reconstruct a volumetric image from 2D “slices" of the image (e.g. a volumetric medical image from 2D medical images, such as X-ray images).

[0043] In particular, the 3D representation 150 of the scene is represented as a set of Gaussian distributions 140, e.g., 3D anisotropic Gaussian distributions (“splats”). A 3D anisotropic Gaussian distribution is defined by a covariance matrix that is, in general, not a multiple of the identity matrix, so that the distribution has different spreads along different axes.

[0044] In other words, the system 100 maintains, as the 3D representation 150 of the scene, data defining a set of Gaussian distributions 140.

[0045] For example, for each Gaussian distribution / , the data defining the Gaussian distribution can include a mean Hi, a covariance matrix, e.g., a positive semi-definite covariance matrix an opacity at, and a view dependent color, e.g., parameterized by spherical harmonics coefficients cf.

[0046] The system 100 receives a set of training images 102 of the scene and uses the set of images 102 of the scene to repeatedly update the 3D representation 150 (e.g. starting from an initialized 3D representation, as described below), i.e., to update the set of Gaussian distributions 140, to generate a final 3D representation 150 of the scene. That is, the final 3D representation 150 of the scene is the data defining the set of Gaussian distributions 140 after being repeatedly updated by the system 100.

[0047] More specifically, the system 100 obtains a plurality of training images 102 of a scene. Generally, the system 100 also obtains, e.g., receives or generates, a respective camera pose for each of the training images 102. The camera pose for a given training image 102 specifies the pose of the camera when the given training image 102 was captured.

[0048] The camera pose can represent the pose of the camera in any of a variety of ways.

[0049] For example, the camera pose can include an estimated location of an origin in a camera reference frame of the camera. As another example, the camera pose can include an estimated azimuth of the camera. For example, the azimuth can be represented as an angle between 0 degrees and 360 degrees or between 0 and radians.

[0050] As another example, the camera pose can include an estimated elevation of the camera. For example, the elevation can be selected from a specified range, e.g., between - 7i / 2 and + K / 2, inclusive.

[0051] As yet another example, the camera pose can include an estimated camera roll of the camera. For example, the camera roll of the camera can be represented as an angle between 0 degrees and 360 degrees or between 0 and 2 radians.

[0052] The system 100 initializes 120 a three-dimensional (3D) representation 150 of the scene. As described above, the 3D representation 150 includes a set of Gaussian distributions 140.

[0053] The system 100 can initialize the set of Gaussian distributions 140 in any of a variety' of ways.

[0054] For example, the system 100 can initialize the set of Gaussian distributions 140 randomly.

[0055] As another example, the system 100 can initialize the set of Gaussian distributions 140 using the set of training images and their corresponding poses using any appropriate Gaussian splatting initialization technique.

[0056] As a particular example, the system 100 can initialize the set of Gaussian distributions 140 by representing the training images as a sparse point cloud using the corresponding camera poses and then initializing a respective Gaussian distribution for each point in the sparse point cloud.

[0057] The system 100 generates a final 3D representation 150 of the scene. To generate the final representation 150, the system 100 updates 160 the set of Gaussian distributions 140 at each of a plurality of update steps.

[0058] At any given update step, the system 100 renders, using the set of Gaussian distributions 140, a respective rendered image 130 of the scene corresponding to each of one or more of the training images 102. A given rendered image 130 corresponds to a given training image 102 when it is rendered from the same camera pose as the camera pose from which the training image was taken.

[0059] The system 100 can perform this rendering using the set of Gaussian distributions 140 using any appropriate Gaussian splatting rendering technique. One example of such a technique is described below. The system 100 then updates 160, using the respective rendered image(s) 130 and the corresponding training image(s) 102. the set of Gaussian distributions 140.

[0060] Generally, at any given update iteration, the system 100 updates 160 some or all of the Gaussian distributions in the set of Gaussian distributions 140 produced in the previous update iteration (if any; if not, the Gaussian distributions 140 of the initialized 3D representation 150) using errors between the rendered image(s) 130 and the corresponding training image(s) 102.

[0061] At some or all of the update iterations, the system 100 can also optionally perform one or more of: splitting one Gaussian distribution currently in the set into multiple Gaussian distributions, adding a new Gaussian distribution to the set, or removing (‘"pruning7’) one or more of the Gaussian distributions currently in the set.

[0062] During the updating, the system 100 can employ one or more techniques to improve the quality of the final 3D representation 150.

[0063] As one example, the system 100 can employ gradient-based pruning. In this example, at each update step, the system 100 can determine whether a pruning criterion is satisfied.

[0064] When the pruning criterion has been satisfied, the system 100 determines whether to remove any of the Gaussian distributions 140 from the set based on respective utilization scores for each of the Gaussian distributions 140 in the set.

[0065] Generally, the respective utilization score for each of the Gaussian distributions 140 measures how much the Gaussian distribution 140 has contributed to rendered images 130 that have been rendered since the preceding update step at which the pruning criterion was satisfied.

[0066] For example, the respective utilization score can be based on, for each of the rendered images 130 that have been rendered since a preceding update step at which the pruning criterion was satisfied, a gradient of rendered colors for pixels in the rendered image with respect to a projected position of the Gaussian distribution 140 for the rendered image 130. The projected position may be the center of the Gaussian distribution 140 in the 3D space projected according to the pose of the rendered image.

[0067] This pruning technique is effective at pruning unnecessary Gaussian distributions and also achieves excellent compression in terms of particle count with equivalent reconstruction quality. That is, compared to existing techniques for pruning Gaussian distributions while learning the representation, this pruning technique achieves superior compression in terms of particle count with equivalent or better reconstruction quality, significantly improving the computational efficiency of generating the final 3D representation.

[0068] Gradient-based pruning is described in more detail below with reference to FIG. 3.

[0069] As another example, instead of or in addition to using gradient-based pruning, the system 100 can employ semantically-aware optimization in order to account for distractors or transient effects in the set of the training images, i.e., to account for the portions of certain training images potentially depicting dynamic or noisy aspects of the scene.

[0070] When performing semantically-aware optimization, at each update step, the system 100 “masks out” pixels that have been determined to be outlier pixels from contributing to the update at the update step, where an outlier pixel is a pixel that depicts a portion of the scene that is dynamic, noisy, or otherwise transient.

[0071] In some cases, the system 100 performs semantically-aware optimization by clustering the pixels in each training image. Performing semantically-aware optimization using pixel clusters is described in more detail below with reference to FIG. 4.

[0072] In some other cases, the system 100 performs semantically-aware optimization using an outlier prediction neural network. Performing semantically-aware optimization using the outlier prediction neural network is described in more detail below with reference to FIG. 5.

[0073] FIG. 2 is a flow diagram of an example process 200 for generating a 3D representation of a scene. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a scene representation system, e.g., the scene representation system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 200.

[0074] The system obtains a plurality of training images of a scene (step 202). Generally, the training images can be “casually captured” images, i.e., the system does not require that the training images be of a controlled environment, e.g., with consistent lighting, or free of occlusions or transient distractors.

[0075] The system also obtains, e.g.. receives or generates, respective camera poses for each of the training images. For example, the system can receive the camera poses as input along with the training images. As another example, the system can apply a structure from motion (SfM) technique to generate the camera poses. As yet another example, the system can predict the camera pose of each training image by making use of a pose prediction neural network that predicts the camera poses of input images. In some implementations, e.g., when the system performs semantically-aware optimization by clustering the pixels in each training image, prior to generating the 3D representation and for each training image, the system clusters the pixels in the training image into a plurality of clusters.

[0076] For example, the system can perform this clustering for a given training image by obtaining a feature map of features of the training image and then performing an appropriate clustenng technique, e.g., agglomerative clustering, on the feature map to generate the plurality of clusters.

[0077] The system can obtain the feature map of features of the training image using any appropriate technique.

[0078] For example, the feature map of features of the training image can be generated by processing the corresponding training image using a trained diffusion model.

[0079] That is, the system or another system can process the corresponding training image using the trained diffusion model to generate the feature map. As a particular example of this, the trained diffusion can be a latent diffusion model that performs a denoising process in a latent space. An example of such a diffusion model is mobilediffusion.

[0080] An example technique for generating image features using a latent diffusion model is described in Luming Tang, Menglin Jia, QianqianWang, Cheng Pemg Phoo, and Bharath Hariharan, 2023. Emergent Correspondence from Image Diffusion. In NeurlPS.

[0081] The system initializes a three-dimensional (3D) representation of the scene (step 204).

[0082] As described above, the 3D representation generally includes a set of Gaussian distributions. Thus, the system initializes data defining each of the Gaussian distributions in the set. Example techniques for initializing the 3D representation are described above with reference to FIG. 1.

[0083] The system generates a final 3D representation of the scene (step 206). Generally, the system generates the final 3D representation by updating the 3D representation at each of a plurality of update steps.

[0084] In particular, at each update step, the system renders, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images (step 208).

[0085] That is, the system selects, e.g.. randomly, one or more of the training images, and then, for each selected training image, renders an image of the scene from the camera pose of the selected training image using the set of Gaussian distributions. The system can generally render the image using any appropriate differentiable rendering technique.

[0086] For example, the 3D scene representation can be rendered to screen space (i.e. a 2D view of the scene from a specific camera pose, e.g. looking along a specific view direction) by rasterization. The screen space (and camera pose) may be defined by a perspective transformation matrix W and a projection matrix J. The splat positions / means can be rasterized to screen coordinates via classical projective geometry. To rasterize the covariance matrix of each splat given W, the perspective transformation matrix generated from the camera pose, the projection of the 3D covariance to 2D screen space can be approximated as S= JWEW'J'. where J is the Jacobian of the projection matrix, which provides a linear approximation to the non-linear projection process. To ensure 2 represents covariance throughout optimization (i.e., positive semidefmite), the covariance matrix can be parameterized as S = RSS' R7. where S and R are respectively a “scale” matrix and a rotation matrix. The scale S=diag(s) with sGR3, and rotation R is computed from a unit Quaternion q. Once splat positions and covariances in screen spaces are computed, the image formation process executes volume rendering as alpha-blending, including sorting the splats along the view direction.

[0087] In other words, the system can render the image by projecting each Gaussian distribution into 2D from the camera perspective, i.e., according to the camera pose. The system can then sort the Gaussian distributions by depth and, for each pixel, iterate over each Gaussian distribution front-to-back, blending the Gaussian distributions together, e.g., through alpha-blending, to determine the color of the pixel in the rendered image.

[0088] Rendering images in this manner is described in more detail in Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM TOG (Proc. SIGGRAPH) (2023).

[0089] The system then updates, using the respective rendered images and the corresponding training images, the set of Gaussian distributions (step 210).

[0090] For example, when the system does not perform semantically-aware optimization, the system can update the set of Gaussian distributions to minimize (that is, iteratively reduce) an objective that includes a term that measures an error, e.g., an LI error, between each rendered image and the corresponding training image. That is. the objective can include the following term for each rendered image: where Inis the corresponding training image and is the rendered image corresponding to the training image at iteration t.

[0091] Because the rendering is differentiable, the system can update the set of Gaussian distributions using gradient descent by backpropagating through the rendering process.

[0092] When the system does perform semantically-aware optimization, the system can modify the objective to account for outlier pixels within the training images. This is described below with reference to FIGS. 4 and 5.

[0093] At some or all of the iterations, the system can also update the set of Gaussian distributions in other ways, e.g., in order to densify the initially sparse set of Gaussian distributions, to prune unnecessary Gaussian distributions, and so on.

[0094] For example, the system can determine to modify a given Gaussian based on the gradient of the objective with respect to the given Gaussian. As a particular example, if the gradient is larger than a threshold and the Gaussian is larger than a threshold, the system can split the Gaussian into two separate Gaussian distributions. As another particular example, if the gradient is larger than a threshold and the Gaussian is smaller than a threshold, the system can clone the Gaussian to generate two Gaussian distributions.

[0095] As another example, the system can determine to prune a given Gaussian if the opacity of the Gaussian is below a threshold.

[0096] Examples of these and other techniques for updating Gaussians are described in more detail in Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM TOG (Proc. SIGGRAPH) (2023).

[0097] In some cases, the system modifies the above process to make use of gradient-based pruning. This is described below with reference to FIG. 3.

[0098] FIG. 3 is a flow diagram of an example process 300 for generating a 3D representation of a scene when the system makes use of gradient-based pruning. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a scene representation system, e.g., the scene representation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. Generally, the system can perform the process 300 at some or all of the update steps, e.g., after updating the set of Gaussian distributions using the respective rendered image(s) and the corresponding training image(s) for the update step.

[0099] The system determines whether a pruning criterion is satisfied (step 302).

[0100] The system can determine whether the pruning criterion has been satisfied in any of a variety of ways.

[0101] As one example, the system can determine that the pruning criterion is satisfied when a threshold number of update steps have been performed since the preceding update step at which the pruning criterion was satisfied.

[0102] That is, the system can determine that a threshold number of update steps have been performed since the preceding update step (if any) at which the pruning criterion was satisfied and, in response, determine that the pruning criterion has been satisfied. If there was no preceding update step at which the pruning criterion was satisfied, the system can determine that the pruning criterion is satisfied if the total number of update steps which have been performed is at least a second threshold number of update steps, which may be the same or different from the first threshold number of update steps.

[0103] As another example, the system can determine that the pruning criterion is satisfied when at least a threshold amount of wall clock time has elapsed since the preceding update step at which the pruning criterion was satisfied. If there was no preceding update step at which the pruning criterion was satisfied, the system can determine that the pruning criterion is satisfied if at least a second threshold amount of wall clock time has elapsed since the iterative updates steps began. The second threshold amount of wall clock time may be the same or different from the first threshold amount of w all clock time.

[0104] When the pruning criterion has not been satisfied, the system determines not to remove any of the Gaussian distributions from the set.

[0105] When the pruning criterion has been satisfied, the system determines whether to remove any of the Gaussian distributions from the set based on respective utilization scores for each of the Gaussian distributions in the set (step 304).

[0106] Generally, the respective utilization score for each of the Gaussian distributions measures how much the Gaussian distribution has contributed to rendered images that have been rendered since the preceding update step at which the pruning criterion w as satisfied.

[0107] In particular, the respective utilization score for a given Gaussian distribution can be based on, for each of the rendered images that have been rendered since the preceding update step at which the pruning criterion was satisfied, a gradient of rendered colors for pixels in the rendered image with respect to a projected position of the Gaussian distribution for the rendered image. More specifically, the projected position of the Gaussian distribution for the rendered image is the projected position of the Gaussian distribution in a screen space of the rendered image (i.e. a 2D field of view associated with the rendered image, based on the corresponding camera pose of the rendered image).

[0108] As one example, the respective utilization score can be based on the sum of, for each pixel of each of the rendered images that have been rendered since the preceding update step at which the pruning criterion was satisfied, the respective gradient of the rendered color for the pixel in the rendered image with respect to the projected position of the Gaussian distribution for the rendered image.

[0109] As another example, the respective utilization score can (alternatively or additionally) be based on the sum of, for each pixel of each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, a product between (i) a mask value for the pixel and (ii) the respective gradient of the rendered color for the pixel in the rendered image with respect to the projected position of the Gaussian distribution for the rendered image. For example, the utilization score ugfor a Gaussian g as of an update step t can be computed as follows: where Ar(t) is the set of update steps since the preceding update step at which the pruning criterion was satisfied (or since the first iteration, in the case that the pruning criterion has not previously been satisfied), h and w are spatial coordinates of the images, Ew his the expectation operator over the spatial coordinates h and w, is the mask value for the pixel having coordinates h,w and can be set to 0 when the pixel is predicted to be an outlier pixel within the rendered image, is the pixel having coordinates h and w in the rendered image at the update step t, is the projected position of the Gaussian distribution for the rendered image, e.g., that is based on the camera pose of the rendered image, e.g., the matrices J and W described above, and the mean of the Gaussian distribution.

[0110] For example, the system can maintain respective utilization scores for each of the Gaussian distributions in the set and. at each update step and for each Gaussian distribution that is in the set as of the update step, update, using the rendered images for the update step. the respective utilization score for the Gaussian distribution, e g. by adding to the maintained score from the preceding update step.

[0111] As a particular example, the system can generate the mask value by processing features of a corresponding pixel in the corresponding training image for the rendered image using an outlier prediction neural network. For example, the features of the corresponding pixel can be the features for the pixel in the feature map described above, i.e., the feature map generated using the trained diffusion model.

[0112] Generally, the outlier prediction neural network is trained jointly with generating the final 3D representation. The outlier prediction neural network, generating mask values using the outlier prediction neural network, and the joint training of the outlier prediction neural network are described in more detail below.

[0113] As another example, the system can generate the mask value based on color residuals. This is described in more detail below.

[0114] For example, the system can determine to remove one or more Gaussian distributions from the set when one or more Gaussian distributions have respective utilization scores that are below a threshold score.

[0115] As a particular example, the system can determine to remove any Gaussian distributions that have a respective utilization score that is below the threshold score. In other words, the system can determine to remove each Gaussian distribution that has a respective utilization score that is below the threshold score.

[0116] As another example, the system can determine to remove, from among the Gaussian distributions that have a respective utilization score that is below the threshold score, at most a fixed number of Gaussian distributions that have the lowest respective utilization scores.

[0117] If the system determines to remove one or more of the Gaussian distributions, the system updates the set of Gaussian distributions by removing the determined Gaussian distribution(s) from the set (step 306).

[0118] The system can also set the respective utilization score for each of the Gaussian distributions that are not removed from the set to zero. That is, the system can “re-set” the utilization scores for the Gaussian distributions in the set each time gradient-based pruning is applied. FIG. 4 is a flow diagram of an example process 400 for generating a 3D representation of a scene when the system makes use of semantically-aware optimization. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a scene representation system, e.g., the scene representation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0119] In particular, the system can perform the process 400 at each update step in order to update, using the respective rendered images and the corresponding training images for the update step, the set of Gaussian distributions.

[0120] Moreover, in the example of FIG. 4, prior to the example process 400. the system has clustered the pixels of each of the training images into respective clusters, e.g., as described above.

[0121] For each cluster of each corresponding training image, the system generates a respective outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels (step 402).

[0122] For example, the system can generate the outlier score for the cluster based on, for each pixel in the cluster, a color residual between the pixel of the training image and a corresponding pixel in the corresponding rendered image.

[0123] The color residual between the pixel of the training image and the corresponding pixel in the corresponding training image measures the difference between, for each of the color components, e g., the RGB values, of the pixel, the color component in the pixel of the training image and the color component of the corresponding pixel in the corresponding training image.

[0124] For example, the color residual can be a sum or average over the color components of an absolute difference between the respective color component for the pixel of the training image and a function of the respective color component of the pixel of the corresponding rendered image. For example, the color residual can be the 1-norm of the differences between the respective color components for the pixel.

[0125] To generate the outlier score for the cluster, the system can, for each pixel in the cluster, determine an initial mask value for the pixel based on the color residual between the pixel and the corresponding pixel in the corresponding rendered image.

[0126] For example, the matrix of initial mask values for a given training image n at update step t can be determined as follows: where R„)is the matrix of color residuals, 1 is an indicator function returning 1 if the predicate is true and 0 otherwise, p is a generalized median with T being a hyperparameter controlling the cut-off percentile, and B is a (normalized) 3 - 3 box filter that performs a morphological dilation via convolution (®).

[0127] The system can determine the generalized median of residual values in any of a variety of ways. For example, the system can determine the generalized median based on a moving average of color residuals for the pixels across update steps. As a particular example, the system can track residual magnitudes over training batches, i.e., over update steps. In particular, the system can discretize residual magnitudes into B histogram buckets of width equal to the lower bound of rendenng error (10-3). The system can update the likelihood of each bucket at each iteration via a discounted update to the bucket population, similar to fast median filtering. This maintains a moving estimate of residual distribution, with constant memory' consumption, from which the system can extract the generalized median value p as the T quantile in the histogram population.

[0128] Optionally, the system can determine an initial mask value for the pixel and then sample a final mask value for the pixel from a Bernoulli distribution that is generated using the initial mask value and a scheduling weight. For example, the system can determine the final mask values as follows: is the matrix of initial mask values and a is the scheduling weight.

[0129] Generally, the scheduling weight increases as update steps progress. For example, the scheduling weight for the update step can be an output of an increasing function applied to an index for the update step. As a particular example, the increasing function can be a staircase exponential function.

[0130] Generating the final mask values in this fashion can allow the system to apply masks gradually, because the outlier predictions can be poor at the beginning of the optimization, i.e., the outlier prediction neural network initially predicts random masks. Further, direct use of the outlier mask can overcommit to outliers, preventing valuable error back-propagation and learning from those regions. Making use of the increasing function allows for a ‘"warm- up” period at the beginning of optimization and can improve the quality of the final representation.

[0131] The system can then determine the outlier score for the cluster based on the fraction of the pixels in the cluster that have an initial (or final) mask value that indicates that the pixel is an outlier pixel. For example, denoting the pixels as p, a given cluster as c, and defining the function C[c,p] as having the value one or zero according to whether pixel p is, or respectively is not. in cluster c, the system can compute the outlier score P as follows:

[0132] For each corresponding training image, the system generates an image mask that masks out predicted outlier pixels using the outlier scores for the clusters of the corresponding training image (step 404). For example, the system can set the mask value equal to one, i.e., determine to not mask out the pixels in a given cluster, for the pixels in the given cluster only if the cluster has an outlier score that is greater than a threshold.

[0133] The system updates the set of Gaussian distributions using, for each respective rendered image, the error between the rendered image and the corresponding masked training image (step 406). The error is calculated based on the image mask. Specifically, it is calculated based on the pixels of the rendered image selected based on the image mask, e.g. as pixels which are not masked-out by the image mask. This may be done by forming the error for a given rendered image, as a sum over the pixels of the rendered image of the product of the image mask with a difference term indicative of an absolute difference of the rendered image and the corresponding training image.

[0134] That is, the system only uses the pixels of the training image that were not masked out to determine the error between the rendered image and the corresponding training image, i.e., pixels that are masked out by the image mask are not included in the loss.

[0135] That is, rather than optimizing the objective described above, the system instead optimizes an objective that, for the update iteration labelled by t, includes the following term for each rendered image n:

[0136] FIG. 5 is a flow diagram of an example process 500 for generating a 3D representation of a scene when the system makes use of semantically-aware optimization. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a scene representation system, e.g., the scene representation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0137] In particular, the system can perform the process 500 at each update step in order to update, using the respective rendered images and the corresponding training images for the update step, the set of Gaussian distributions.

[0138] For each pixel of each corresponding training image, the system processes features of the pixel of the corresponding training image (e.g. together with data indicating the location of the pixel in the training image) using an outlier prediction neural network to generate an outlier score for the pixel that indicates a likelihood that the pixel is an outlier pixel (step 502).

[0139] The outlier prediction neural network can generally have any appropriate architecture that allows the neural network to map a set of features to a scalar value. For example, the outlier prediction neural network can be a multi-layer perceptron (MLP).

[0140] For example, the features of the corresponding pixel can be the features for the pixel in the feature map described above, i.e., the feature map generated using the trained diffusion model.

[0141] For each corresponding training image, the system generates an image mask that masks out predicted outlier pixels using the outlier scores for the pixels of the corresponding training image (step 504).

[0142] For example, for each pixel, the system can generate a mask value for the pixel from the outlier score for the pixel in accordance with a scheduling weight for the update step.

[0143] As a particular example, the system can determine an initial mask value for the pixel from the outlier score for the pixel, e g., by setting the initial mask value to be equal to the outlier score, and then sample a final mask value for the pixel from a Bernoulli distribution P that is generated using the initial mask value and the scheduling weight. For example, the system can determine the final mask values as follows: is the matrix of initial mask values and a is the scheduling weight.

[0144] Generally, the scheduling weight increases as update steps progress. For example, the scheduling weight for the update step can be an output of an increasing function applied to an index for the update step. As a particular example, the increasing function can be a staircase exponential function.

[0145] Generating the final mask values in this fashion can allow the system to apply masks gradually, because the outlier predictions can be poor at the beginning of the optimization, i.e., the outlier prediction neural network initially predicts random masks. Further, direct use of the outlier mask can overcommit to outliers, preventing valuable error back-propagation and learning from those regions. Making use of the increasing function allows for a ‘"warmup” period at the beginning of optimization and can improve the quality of the final representation.

[0146] The system updates the set of Gaussian distributions using, for each respective rendered image, an error between the rendered image and the corresponding masked training image (step 506). The error for each rendered image is calculated based on the image mask. Specifically, it is calculated based on the pixels of the rendered image selected based on the image mask, e.g. as pixels which are not masked-out by the image mask. This may be done by forming the error for a given rendered image, as a sum over the pixels of the rendered image of the product of the image mask with a difference term indicative of an absolute difference of the rendered image and the corresponding training image. That is, the system only uses the pixels of the training image that were not masked out to determine the error between the rendered image and the corresponding training image, i.e., pixels that are masked out by the image mask are not included in the loss.

[0147] That is, rather than optimizing the objective described above, the system instead optimizes an objective that, for the update iteration labelled by t, includes the following term for each rendered image n:

[0148] Generally, at some or all of the update steps, the system also trains the outlier prediction neural network using, for each pixel of each corresponding training image for the update step, a color residual between the pixel and the corresponding pixel in the corresponding rendered image. That is, the system determines a target output for the outlier prediction neural network for a given pixel from the color residual between the pixel and the corresponding pixel in the corresponding rendered image.

[0149] In some cases, the system both trains the outlier prediction neural network and updates the set of Gaussian distributions at each update step. In some other cases, the system alternates between performing one or more update steps where the system trains the outlier prediction neural network and performing update steps where the system updates the set of Gaussian distributions.

[0150] For example, the system can train the outlier prediction neural network on a loss function that includes a term that encourages the outlier prediction neural network to map features of pixels with high color residuals to high outlier probability and to map features of pixels with low color residual error to low outlier probability. Each update to the output prediction neural network modifies a plurality of numerical parameters defining the output prediction neural network to reduce the value of the loss function.

[0151] The loss function can optionally also include one or more additional terms. For example, the one or more additional terms can include a regularization term that regularizes the training.

[0152] As a particular example, the training of the outlier prediction neural network can be represented as follows: where k is a hyper-parameter, e.g., .3., 5., .6, or .8, 9^ are the parameters of the outlier prediction neural network as of update step t, and U and L are self-supervision labels computed as follows: from e , (2) with T = ,5 from eq. (2) with r ~ .9 where eq. (2) is the equation for computing the initial mask values described above, i.e.,

[0153] Thus, the system directly supervises the outlier prediction neural network only on pixels for which the system can confidently determine the inlier status based on reconstruction residuals, and otherwise the system relies on semantic similarity in the feature space for supervising the outlier prediction neural network. The system can use any appropriate regularization loss as the loss Lreg in the equation above. For example, the regularization loss can be a loss that minimizes (that is, iteratively reduces) the Lipschitz constant.

[0154] FIG. 6 shows an example 600 of 3D representations learned from the described techniques (“Our SLS”) when performing semantically-aware optimization relative to an existing 3D Gaussian Splatting technique (“3DGS”) from a set of training images that includes transient occluders (in Fig. 6 the training image are at the left, and include people walking through a real-world environment including a static bicycle). As can be seen from the Figure, the described techniques effectively account for the transient occluders while the SDGS cannot and generates blurry images, both from viewpoints that match those of the training images and novel views.

[0155] FIG.7 shows an example 700 of the performance of the described techniques when making use of gradient-based pruning.

[0156] In particular, the example 700 shows the performance of the described techniques with different values of k, the threshold used to determine when to prune a Gaussian when performing gradient-based pruning as described above, across four different scenes: garden vase, stump, bicycle, vase deck, and tree hill. Higher values of k result in a larger compression factor, i.e., fewer “primitives.” i.e., splats, in the final set of Gaussians relative to the original number. As can be seen from the example 700, the Peak Signal -to-Noise Ratio (PSNR) of the rendered images does not experience a significant degradation even with compression factors of 4.5x relative to the original. This is shown visually for the garden vase data set in FIG. 7.

[0157] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0158] The term "‘data processing apparatus’7refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry , e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0159] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0160] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0161] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0162] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry7and one or more programmed computers.

[0163] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0164] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory. media and memory7devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory7devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g.. visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0165] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0166] Machine learning models can be implemented and deployed using a machine learning framework, e.g.. a TensorFlow framework or a Jax framework.

[0167] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0168] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0169] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0170] Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0171] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0172] Aspects of the present disclosure may be as set out in the following clauses: Clause 1. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images; updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions; determining whether a pruning criterion is satisfied: and when the pruning criterion has been satisfied: determining whether to remove any of the Gaussian distributions from the set based on respective utilization scores for each of the Gaussian distributions in the set, wherein the respective utilization score for each of the Gaussian distributions measures how much the Gaussian distribution has contributed to rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied.

[0173] Clause 2. The method of clause 1 , wherein determining whether a pruning criterion is satisfied comprises: determining that a threshold number of update steps have been performed since the preceding update step at which the pruning criterion was satisfied.

[0174] Clause 3. The method of clause 1 or clause 2, wherein determining whether to remove any of the Gaussian distributions from the set based on the respective utilization scores comprises: determining to remove any Gaussian distributions that have a respective utilization score that is below a threshold score.

[0175] Clause 4. The method of any preceding clause, wherein generating the final 3D representation of the scene further comprises, at each of the plurality of update steps: maintaining the respective utilization score for each of the Gaussian distributions in the set as of the update step; and for each Gaussian distribution in the set as of the update step, updating, using the rendered images for the update step, the respective utilization score. Clause 5. The method of clause 4. wherein generating the final 3D representation of the scene further comprises, at each of the plurality- of update steps: when the pruning criterion is satisfied, setting the respective utilization score for each of the Gaussian distributions that is not removed from the set to zero.

[0176] Clause 6. The method of any preceding clause, wherein, for each Gaussian distribution in the set as of the update step, the respective utilization score is based on, for each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, a gradient of rendered colors for pixels in the rendered image with respect to a projected position of the Gaussian distribution for the rendered image.

[0177] Clause 7. The method of clause 6, wherein the projected position of the Gaussian distribution for the rendered image is a projected position of the Gaussian distribution in a screen space of the rendered image.

[0178] Clause 8. The method of clause 6 or clause 7. wherein the respective utilization score is based on a sum of, for each pixel each of each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, the respective gradient of the rendered color for the pixel in the rendered image w ith respect to the projected position of the Gaussian distribution for the rendered image.

[0179] Clause 9. The method of clause 6 or clause 7, wherein the respective utilization score is based on a sum of, for each pixel each of each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, a product between (i) a mask value for the pixel and (ii) the respective gradient of the rendered color for the pixel in the rendered image w ith respect to the projected position of the Gaussian distribution for the rendered image.

[0180] Clause 10. The method of clause 9. wherein the mask value for the pixel is set to 0 when the pixel is predicted to be an outlier pixel within the rendered image.

[0181] Clause 11. The method of clause 10, further comprising: generating the mask value by processing features of a corresponding pixel in the corresponding training image for the rendered image using an outlier prediction neural network.

[0182] Clause 12. The method of clause 11, wherein outlier prediction neural network is trained jointly with generating the final 3D representation.

[0183] Clause 13. The method of clause 11 or clause 12, wherein the features of a corresponding pixel in the corresponding training image for the rendered image are generated by processing the corresponding training image using a trained diffusion model.

[0184] Clause 14. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; for each of the plurality of training images, clustering the pixels in the training image into a plurality of clusters; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images; updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions, comprising: for each cluster of each corresponding training image, generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels; for each corresponding training image, generating an image mask that masks out predicted outlier pixels using the outlier scores for the clusters of the corresponding training image and applying the image mask to the corresponding training image to generate a corresponding masked training image; and updating the set of Gaussian distributions using, for each respective rendered image, an error between the rendered image and the corresponding masked training image. Clause 15. The method of clause 14, wherein clustering the pixels in the training image into a plurality of clusters comprises: obtaining a feature map of features of the training image; and performing agglomerative clustering on the feature map to generate the plurality of clusters.

[0185] Clause 16. The method of clause 15, wherein the feature map of features of the training image is generated by processing the corresponding training image using a trained diffusion model.

[0186] Clause 17. The method of clause 16, wherein the trained diffusion model is a trained latent diffusion model.

[0187] Clause 18. The method of any one of clauses 14-17, wherein generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels comprises: generating the outlier score for the cluster based on, for each pixel in the cluster, a color residual between the pixel and a corresponding pixel in the corresponding rendered image.

[0188] Clause 19. The method of clause 18, wherein the color residual between the pixel and a corresponding pixel in the corresponding rendered image is based on a moving average of color residuals for the pixel across update steps.

[0189] Clause 20. The method of clause 18 or 19, wherein generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels comprises: for each pixel in the cluster, determining an initial mask value for the pixel based on the color residual between the pixel and a corresponding pixel in the corresponding rendered image; and determining the outlier score for the cluster based on a fraction of the pixels in the cluster that have an initial mask value that indicates that the pixel is an outlier pixel. Clause 21. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality7of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images; updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions, comprising: for each pixel of each corresponding training image, processing features of the pixel of the corresponding training image using an outlier prediction neural network to generate an outlier score for the pixel that indicates a likelihood that the pixel is an outlier pixel; for each corresponding training image, generating an image mask that masks out predicted outlier pixels using the outlier scores for the pixels of the corresponding training image and applying the image mask to the corresponding training image to generate a corresponding masked training image; and updating the set of Gaussian distributions using, for each respective rendered image, an error between the rendered image and the corresponding masked training image.

[0190] Clause 22. The method of clause 21, wherein the features of the pixel of the corresponding training image are generated by processing the corresponding training image using a trained diffusion model.

[0191] Clause 23. The method of clause 22, wherein the trained diffusion model is a trained latent diffusion model.

[0192] Clause 24. The method of any one of clauses 21-23, wherein generating an image mask that masks out predicted outlier pixels using the outlier scores for the pixels of the corresponding training image comprises: for each pixel, generating a mask value for the pixel from the outlier score for the pixel in accordance with a scheduling weight for the update step.

[0193] Clause 25. The method of clause 24, wherein the scheduling weight for the update step is an output of an increasing function applied to an index for the update step.

[0194] Clause 26. The method of clause 25, wherein the increasing function is a staircase exponential function.

[0195] Clause 27. The method of any one of clauses 21-26, further comprising, at each update step: training the outlier prediction neural network using, for each pixel of each corresponding training image for the update step, a color residual between the pixel and a corresponding pixel in the corresponding rendered image.

[0196] Clause 28. The method of clause 27, wherein the color residual between the pixel and a corresponding pixel in the corresponding rendered image is based on a moving average of color residuals for the pixel across update steps.

[0197] Clause 29. The method of clause 27 or clause 28, wherein training the outlier prediction neural network using, for each pixel of each corresponding training image for the update step, a color residual between the pixel and a corresponding pixel in the corresponding rendered image comprises: training the outlier prediction neural network on a loss function that comprises a term that encourages the outlier prediction neural network to map features of pixels with high color residuals to high outlier probability and to map features of pixels with low color residual error to low outlier probability.

[0198] Clause 30. The method of clause 29, wherein the loss function further comprises a regularization term.

[0199] Clause 31. The method of any preceding clause, further comprising: after generating the final 3D representation of the scene, receiving data specifying a new camera pose; and generating a new image of the scene that appears to be taken by a camera having the new camera pose using the final 3D representation of the scene.

[0200] Clause 32. The method of any preceding clause, further comprising: after generating the final 3D representation of the scene, using the final 3D representation of the scene to perform 3D modeling of the scene.

[0201] Clause 33. The method of any preceding clause, further comprising: after generating the final 3D representation of the scene, controlling a robot based on the final 3D representation of the scene.

[0202] Clause 34. The method of any preceding clause, further comprising: after generating the final 3D representation of the scene, training a control policy for a robot using the final 3D representation of the scene.

[0203] Clause 35. The method of any preceding clause, further comprising: after generating the final 3D representation of the scene, generating images of the scene using the final 3D representation of the scene and using the generated images to train a machine learning model.

[0204] Clause 36. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of clauses 1-35.

[0205] Clause 37. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of clauses 1-35.

Claims

WHAT IS CLAIMED IS:

1. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images; updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions; determining whether a pruning criterion is satisfied: and when the pruning criterion has been satisfied: determining whether to remove any of the Gaussian distributions from the set based on respective utilization scores for each of the Gaussian distributions in the set, wherein the respective utilization score for each of the Gaussian distributions measures how much the Gaussian distribution has contributed to rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied.

2. The method of claim 1 , wherein determining whether a pruning criterion is satisfied comprises: determining that a threshold number of update steps have been performed since the preceding update step at which the pruning criterion w as satisfied.

3. The method of claim 1 or claim 2, wherein determining whether to remove any of the Gaussian distributions from the set based on the respective utilization scores comprises: determining to remove any Gaussian distributions that have a respective utilization score that is below a threshold score.

4. The method of any preceding claim, wherein generating the final 3D representation of the scene further comprises, at each of the plurality of update steps: maintaining the respective utilization score for each of the Gaussian distributions in the set as of the update step; and for each Gaussian distribution in the set as of the update step, updating, using the rendered images for the update step, the respective utilization score.

5. The method of claim 4, wherein generating the final 3D representation of the scene further comprises, at each of the plurality of update steps: when the pruning criterion is satisfied, setting the respective utilization score for each of the Gaussian distributions that is not removed from the set to zero.

6. The method of any preceding claim, wherein, for each Gaussian distribution in the set as of the update step, the respective utilization score is based on. for each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, a gradient of rendered colors for pixels in the rendered image with respect to a projected position of the Gaussian distribution for the rendered image.

7. The method of claim 6, wherein the projected position of the Gaussian distribution for the rendered image is a projected position of the Gaussian distribution in a screen space of the rendered image.

8. The method of claim 6 or claim 7, wherein the respective utilization score is based on a sum of, for each pixel of each of the rendered images that have been rendered since a preceding update step at which the pruning criterion was satisfied, the respective gradient of the rendered color for the pixel in the rendered image with respect to the projected position of the Gaussian distribution for the rendered image.

9. The method of claim 6 or claim 7, wherein the respective utilization score is based on a sum of, for each pixel of each of the rendered images that have been rendered since apreceding update step at which the pruning criterion was satisfied, a product between (i) a mask value for the pixel and (ii) the respective gradient of the rendered color for the pixel in the rendered image with respect to the projected position of the Gaussian distribution for the rendered image.

10. The method of claim 9, wherein the mask value for the pixel is set to 0 when the pixel is predicted to be an outlier pixel within the rendered image.

11. The method of claim 10, further comprising: generating the mask value by processing features of a corresponding pixel in the corresponding training image for the rendered image using an outlier prediction neural network.

12. The method of claim 11, wherein outlier prediction neural network is trained jointly with generating the final 3D representation.

13. The method of claim 11 or claim 12, wherein the features of a corresponding pixel in the corresponding training image for the rendered image are generated by processing the corresponding training image using a trained diffusion model.

14. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; for each of the plurality of training images, clustering the pixels in the training image into a plurality of clusters; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images;updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions, comprising: for each cluster of each corresponding training image, generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels; for each corresponding training image, generating an image mask that masks out predicted outlier pixels using the outlier scores for the clusters of the corresponding training image; and updating the set of Gaussian distributions using, for each respective rendered image, an error between the rendered image and the corresponding masked training image, the error being based on the image mask.

15. The method of claim 14, wherein clustering the pixels in the training image into a plurality of clusters comprises: obtaining a feature map of features of the training image; and performing agglomerative clustering on the feature map to generate the plurality of clusters.

16. The method of claim 1 , wherein the feature map of features of the training image is generated by processing the corresponding training image using a trained diffusion model.

17. The method of claim 16, wherein the trained diffusion model is a trained latent diffusion model.

18. The method of any one of claims 14-17, wherein generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels comprises: generating the outlier score for the cluster based on, for each pixel in the cluster, a color residual between the pixel and a corresponding pixel in the corresponding rendered image.

19. The method of claim 18, wherein the color residual between the pixel and a corresponding pixel in the corresponding rendered image is based on a moving average of color residuals for the pixel across update steps.

20. The method of claim 18 or 19, wherein generating an outlier score for the cluster that indicates a likelihood that the pixels in the cluster are outlier pixels comprises: for each pixel in the cluster, determining an initial mask value for the pixel based on the color residual between the pixel and a corresponding pixel in the corresponding rendered image; and determining the outlier score for the cluster based on a fraction of the pixels in the cluster that have an initial mask value that indicates that the pixel is an outlier pixel.

21. A method performed by one or more computers, the method comprising: obtaining a plurality of training images of a scene; initializing a three-dimensional (3D) representation of the scene, the 3D representation comprising a set of Gaussian distributions; and generating a final 3D representation of the scene, comprising, at each of a plurality of update steps: rendering, using the set of Gaussian distributions, a respective rendered image of the scene corresponding to each of one or more of the training images; updating, using the respective rendered images and the corresponding training images, the set of Gaussian distributions, comprising: for each pixel of each corresponding training image, processing features of the pixel of the corresponding training image using an outlier prediction neural network to generate an outlier score for the pixel that indicates a likelihood that the pixel is an outlier pixel; for each corresponding training image, generating an image mask that masks out predicted outlier pixels using the outlier scores for the pixels of the corresponding training image; and updating the set of Gaussian distributions using, for each respectiverendered image, an error between the rendered image and the corresponding training image, the error being based on the image mask.

22. The method of claim 21, wherein the features of the pixel of the corresponding training image are generated by processing the corresponding training image using a trained diffusion model.

23. The method of claim 22, wherein the trained diffusion model is a trained latent diffusion model.

24. The method of any one of claims 21-23, wherein generating an image mask that masks out predicted outlier pixels using the outlier scores for the pixels of the corresponding training image comprises: for each pixel, generating a mask value for the pixel from the outlier score for the pixel in accordance with a scheduling weight for the update step.

25. The method of claim 24, wherein the scheduling weight for the update step is an output of an increasing function applied to an index for the update step.

26. The method of claim 25, wherein the increasing function is a staircase exponential function.

27. The method of any one of claims 21-26, further comprising, at each update step: training the outlier prediction neural network using, for each pixel of each corresponding training image for the update step, a color residual between the pixel and a corresponding pixel in the corresponding rendered image.

28. The method of claim 27, wherein the color residual between the pixel and a corresponding pixel in the corresponding rendered image is based on a moving average of color residuals for the pixel across update steps.

29. The method of claim 27 or claim 28, wherein training the outlier prediction neural network using, for each pixel of each corresponding training image for the update step, a color residual between the pixel and a corresponding pixel in the corresponding rendered image comprises: training the outlier prediction neural network on a loss function that comprises a term that encourages the outlier prediction neural network to map features of pixels with high color residuals to high outlier probability and to map features of pixels with low color residual error to low outlier probability.

30. The method of claim 29, wherein the loss function further comprises a regularization term.

31. The method of any preceding claim, further comprising: after generating the final 3D representation of the scene, receiving data specifying a new camera pose; and generating a new image of the scene that appears to be taken by a camera having the new camera pose using the final 3D representation of the scene.

32. The method of any preceding claim, further comprising: after generating the final 3D representation of the scene, using the final 3D representation of the scene to perform 3D modeling of the scene.

33. The method of any preceding claim, further comprising: after generating the final 3D representation of the scene, controlling a robot based on the final 3D representation of the scene.

34. The method of any preceding claim, further comprising: after generating the final 3D representation of the scene, training a control policy for a robot using the final 3D representation of the scene.

35. The method of any preceding claim, further comprising: after generating the final 3D representation of the scene, generating images of the scene using the final 3D representation of the scene and using the generated images to train a machine learning model.

36. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of claims 1-35.

37. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-35.

Images