3D scene generation using sparse voxel latent diffusion models

Abstract

The application describes generation of high-resolution 3D representations of objects or scenes at scale. The sparse nature of the 3D data can be leveraged using a sparse voxel hierarchical representation where only values for "active" voxels are stored. It discloses a method and processor that generates a first latent representation of a three-dimensional (3D) object at a first level of detai from which is generated a first sparse voxel hierarchical representation of the 3D object. From this representation a second latent representation of the 3D object at a second level of detail is generated wherein the second level of detail is greater than the first level of detail. From the second latent representation is generated a second voxel hierarchical representation of the 3D object which is provided as a generated 3D representation of the 3D object including geometry data and attribute data for one or more active voxels of the second voxel hierarchical representation. A machine learning model, such as a diffusion model, may be used to generate the latent representations with different levels of detail. An additional invention is claimed in which a system comprising one or more processors generates a sparse voxel hierarchical representation of a three-dimensional (3D) object by using a first machine learning model to generate latent representations at multiple levels of a sparse voxel hierarchy and a second machine learning model to generate respective hierarchical representations at the multiple levels, wherein generation of at least one of the latent representations is conditioned on at least one other of the latent representations having a coarser level of granularity. [Fig. 5]