Three-dimensional mesh generation using signed distance functions

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a mesh of a scene from a text description of the scene using signed distance functions.

Description

[0001] THREE-DIMENSIONAL MESH GENERATION USING SIGNED DISTANCE FUNCTIONS

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] This application is a non-provisional of and claims priority to Greece Application No. 20230100331, filed on April 19, 2023, the entire contents of which are hereby incorporated by reference.

[0004] BACKGROUND

[0005] This specification relates to generating representations of a scene using machine learning models.

[0006] As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance w ith current values of a respective set of weights.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers that generates a three-dimensional (3D) mesh of a scene from a text prompt.

[0009] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0010] The ability to generate highly realistic 2D images from mere text prompts has recently made huge progress in terms of speed and quality, thanks to the advent of image diffusion models.

[0011] However, methods that attempt to harness diffusion models, trained on 2D images, for supervision of 3D model generation using view dependent prompts have major drawbacks.

[0012] For example, rather than commonly used 3D meshes, they instead generate neural radiance fields (NeRFs), making them impractical for most real applications.

[0013] As another example, these approaches tend to produce over-saturated models, giving the output a cartoonish looking effect. To address these issues, this specification describes techniques for generation of highly realistic looking 3D meshes from text prompts.

[0014] To this end, the described techniques extend NeRF to employ a signed distance function (SDF) backbone, leading to improved 3D mesh extraction. That is, by using a 3D model of the scene that predicts signed distances for input points, the system can significantly improve the quali ty of the mesh that is extracted once the 3D model of the scene has been optimized for a given input text prompt.

[0015] In addition, this specification describes techniques for finetuning the extracted mesh texture, removing the effect of high saturation and improving the details of the output 3D mesh.

[0016] In particular, in some cases, the one or more computers that perform the mesh generation can have a constrained memory space, e.g., due to other processes running on the computers or due to the underlying computer hardware. However, optimizing the 3D model, e.g., using SDS gradients as described below, can require repeatedly rendering images bysubmitting a significant number of queries to a neural-network-based 3D model and performing inference using a pixel-level diffusion model. As a result, this optimization may' be memory- intensive. Thus, to stay within the memory- constraints of hardware on which the system is deployed, the system may need to use relatively low -resolution images and, in some case, a relatively high guidance weight for performing the optimization. When this occurs, the first 3D mesh can miss high resolution details, given that the optimization is performed at low- resolution. To account for this, the described techniques can fine-tune the extracted mesh texture as described in this specification, e.g., to remove the effect of high saturation and improve the details of the output 3D mesh, yielding a high quality final mesh while staying within the memory constraints.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below-.

[0018] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 is a block diagram of an example mesh generation system.

[0021] FIG. 2 is a flow diagram of an example process for generating a final mesh. FIG. 3 is a flow diagram of an example process for optimizing the parameters of the 3D model.

[0022] FIG. 4 is a flow diagram of an example process for generating the second mesh from the first mesh.

[0023] FIG. 5 is a flow diagram of an example process for generating the third mesh from the second mesh.

[0024] FIG. 6 shows an example of the operation of the system.

[0025] Like reference numbers and designations in the various drawings indicate like elements.

[0026] DETAILED DESCRIPTION

[0027] FIG. 1 is a block diagram of an example mesh generation system 100. The mesh generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0028] The system 100 receives a text prompt 102 and processes the text prompt 102 to generate a three-dimensional (3D) mesh 150 of a scene.

[0029] For example, the text prompt 102 can describe an object in the scene and the 3D mesh 150 generated by the system can define the shape, e.g., the geometry, of the object.

[0030] In other words, a three-dimensional mesh of a three-dimensional object is a collection of vertices and polygon faces that define the shape of the three-dimensional object.

[0031] Optionally, the mesh 150 can also specify a number of attributes at one or more of its vertices, faces, or edges. These can include, e.g., texture, e.g., texture coordinates or other texture data, vertex normals, reflection vectors, or material parameters.

[0032] In particular, the system 100 can receive a text prompt 102 that describes the scene and then generate a 3D mesh 150 of the scene from the text prompt 102. In the example of FIG. 1, the text prompt is “A red-eyed tree frog.”

[0033] The text prompt 102 can be any appropriate natural language text sequence and can be. e.g., received from a user of the system 150. That is, during operation, the system 100 can generate meshes 150 from arbitrary text prompts 102 without imposing requirements on the contents of the text prompts 102.

[0034] To generate the 3D mesh 150 from the text prompt 102, the system 100 optimizes parameters 114 of a 3D model 110 of the scene using a trained image diffusion neural network 120 that is conditioned on the text prompt 102. For example, as will be described in more detail below, the system 100 can optimize the parameters 114 of the 3D model 110 of the scene using score distillation sampling (SDS) gradients computed from images of the scene generated using the 3D model 110 of the scene and using the trained image diffusion neural network 120.

[0035] In particular, the 3D model 110 of the scene maps a point in the scene and a viewing direction from the point to a signed distance for the point and a color along the point - viewing direction pair.

[0036] The signed distance for the point can be, e.g., the signed distance from the point to the surface of the object in the scene that is described by the text prompt. That is, the signed distance can be positive if the point is outside the surface of the object and negative if the point is within the object.

[0037] The parameters of the 3D model 110 can include parameters of one or more neural networks, e.g., multi-layer perceptrons (MLPs), that are used as part of the mapping.

[0038] These signed distances and colors can then be used to synthesize an image of the scene from the viewing direction, e.g.. by adopting an SDF to density transformation that can be learned as part of the 3D model 110 and using a volume rendering operator to render the image.

[0039] One example of synthesizing an image of the scene from a viewing direction is described below.

[0040] Thus, by optimizing the 3D model 110 of the scene, the system 100 leams a signed distance function for the scene that maps points in the scene to signed distances.

[0041] The system 100 then generates, in accordance with the optimized parameters 114 of the 3D model 110 of the scene, a first 3D mesh 112.

[0042] That is. the system 100 generates data specifying the vertices and faces of the first 3D mesh 112 using the learned signed distance function for the scene. As described above, the system 100 can optionally also generate values of one or more attributes for one or more of the vertices in the mesh, the faces in the mesh, or the edges in the mesh, e.g., by generating textures for the mesh from the color values mapped by the 3D model.

[0043] In some cases, the system 100 uses the first 3D mesh 112 as the final 3D mesh 150.

[0044] In some other cases, a refinement engine 140 within the system 100 further refines the first 3D mesh 112 to generate the final 3D mesh 150.

[0045] For example, the refinement engine 140 can generate a second 3D mesh from the first 3D mesh and, further optionally, a third 3D mesh from the second 3D mesh, and then provide the further refined mesh as the final 3D mesh 150. In some of these cases, during the refinement, the system 100 can freeze the geometry' of the mesh 112 and only update one or more other attributes of the mesh 112, e.g., the texture of the mesh 1 12.

[0046] That is, when generating the second mesh from the first mesh, when generating the third mesh from the second mesh, or both, the refinement engine 120 can freeze the geometry7of the mesh while updating the texture of the mesh.

[0047] The system 100 can use the final 3D mesh 150 for any of a variety of purposes.

[0048] As one example, the system 100 can use the final 3D mesh 1 0 to render images of the scene from novel viewpoints. For example, as shown in FIG. 1, the system 100 can render an image 160 of the redeyed tree frog characterized in the prompt 102 from a specified viewpoint. That is, the final 3D mesh can be rendered as a 2D image. The 2D image can be rendered on a display associated with a computing apparatus.

[0049] As another example, the system 100 can be part of a computer graphics pipeline and the final 3D mesh 150 can be provided to another component of the pipeline, e g., to be included in an augmented reality (AR) or virtual reality (VR) environment, or to be included in another computer graphics application, e.g., a video game or a video that incorporates computer graphics.

[0050] FIG. 2 is a flow diagram of an example process 200 for generating a mesh of a scene. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a mesh generation system, e.g., the system 100 in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0051] The system obtains a text prompt (step 202).

[0052] The system then generates a final three-dimensional (3D) mesh of a scene that is described by the text prompt.

[0053] As part of generating the final mesh, the system generates a first 3D mesh of the scene (step 204).

[0054] In particular, the system optimizes the parameters of a 3D model of the scene using a trained image diffusion neural network that is conditioned on the text prompt (step 206).

[0055] As described above, the 3D model of the scene maps a point in the scene and a viewing direction from the point to a signed distance for the point and a color along the point - viewing direction pair.

[0056] Optimizing the 3D model is described in more detail below with reference to FIG. 3. After optimizing the 3D model, the system generates, in accordance with the optimized parameters of the 3d model of the scene, a first 3D mesh (step 206).

[0057] In particular, to generate the 3D mesh after optimizing the model, the system can extract the first 3D mesh as the surface at the zero-level set of signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene. The system can perform this extraction using any appropriate mesh generation technique. As one example, the system can perform this extraction using Marching Cubes.

[0058] In addition to extracting the geometry of the mesh, the system can also use the mesh generation technique to extract textures for the mesh using the colors mapped by the 3D model. For example, the system can determine colors for images corresponding to faces of the mesh using the colors assigned to points on the face of the mesh by the 3D model.

[0059] Since floaters (i.e. , areas of near 0 signed distance value away from the expected object surface) can occasionally remain within the volume, the system can select the largest mesh component that is closer to the center of the volume to create the mesh. In other words, the system can select a largest mesh component closer to a center of a volume defined by the signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene.

[0060] Optionally, the system can further refine the first 3D mesh.

[0061] In particular, the system can generate a second 3D mesh from the first 3D mesh (step 208). At a high level, when refining the first 3D mesh to generate the second 3D mesh, the system can fix the geometry of the mesh and refine one or more attributes of the mesh, e g., by fine-tuning the mesh texture to account for sub-optimal color saturation and lack of detail in the initial mesh.

[0062] Generating the second mesh from the first mesh is described in more detail below with reference to FIG. 4.

[0063] Further optionally, the system can generate a third 3D mesh from the second 3D mesh (step 210). At a high level, when refining the second 3D mesh to generate the third 3D mesh, the system can refine the second mesh to account for any misalignments that are reflected in the second 3D mesh.

[0064] Generating the third mesh is described in more detail below with reference to FIG. 5.

[0065] When the third mesh is generated, the system can then use the third 3D mesh as the final mesh of the scene. When only the second mesh is generated, the system can use the second mesh as the final mesh of the scene. When only the first mesh is generated, the system can use the first mesh as the final mesh of the scene. FIG. 3 is a flow diagram of an example process 300 for optimizing the parameters of a 3D model of the scene. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a mesh generation system, e.g., the system 100 in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0066] As described above, the system performs the optimization using a trained image diffusion neural network that is conditioned on the text prompt.

[0067] The trained image diffusion neural network can generally be any trained text-to-image diffusion neural network and can perform diffusion in the latent space or in the pixel space. In particular, it is possible that the trained image diffusion neural network has been trained using only 2D images and need not have been trained on any data sets that are specific to 3D scene generation. Examples of text-to-image diffusion neural networks that can be used include Imagen and StableDiffusion.

[0068] In particular, in the example of FIG. 3, the system optimizes the parameters 0 of the model using score distillation sampling (SDS) gradients.

[0069] The SDS gradients are generated using denoising outputs generated by the trained image diffusion neural network for noisy images generated from images of the scene generated from the 3D model of the scene.

[0070] In particular, the SDS gradient is a gradient that pushes the rendered images, i.e., the images of the scene generated from the 3D model, to a high probability density region for the text prompt as measured by the denoising output generated by the trained image diffusion neural network.

[0071] In particular, to compute the SDS gradient for a given image I of the scene that is rendered using an operator n that makes use of the 3D model given a camera pose (. the system samples a time step t and noise e and generates a noisy image Itfrom the given image and the sampled noise in accordance with the sampled timestep.

[0072] To generate a noisy (or noised image) itfrom given timestep t and noise e, the system can perform the following: where atis a noise level that depends on the sampled time step t. In particular, the noise level is a decreasing function of the sampled time step t, so that the larger t is the noisier Itwill be. The system can the compute the SDS gradient with respect to the parameters 9 of the 3D model as: where w(t) is a weight corresponding to the time step t)1Sthe initial denoising output generated by processing t, and the text prompt y by the denoising neural network and U70 is the gradient of the image I of the scene from the camera pose w ith respect to the parameters 9 of the 3D model.

[0073] Optionally, the system can incorporate classifier-free guidance to control the strength of the text conditioning on the prompt y. In this case, the SDS gradient can be: where a> is the guidance weight and (lc, t) is an unconditioned denoising output that is not conditioned on the text prompt.

[0074] Thus, when using classifier-free guidance, the system processes a second diffusion input that includes the noisy image and does not include the text prompt using the trained diffusion neural network to generate an initial unconditioned denoising output for the noisy image and then combines the initial denoising output and the initial unconditioned denoising output in accordance with the guidance weight to generate the denoising output.

[0075] Accordingly, as can be seen from the equation above, to determine the SDS gradient for the sampled camera pose, the system determines a gradient of the image of the scene from the camera pose with respect to the parameters 9 of the 3D model and determines the SDS gradient using the respective denoising outputs for each of the noisy images and the target image and the gradient of the image of the scene from the camera pose.

[0076] To use the SDS gradients to optimize the parameters of the 3D model, the system can repeatedly perform iterations of the process 300, e.g., until convergence criteria have been satisfied, until a threshold number of iterations have been performed, or some other termination criterion is satisfied.

[0077] The system samples a set of one or more camera poses (step 302).

[0078] The system then performs steps 304-310 for each of the one or more camera poses. The system generates an image of the scene from the camera pose using the 3D model of the scene and in accordance with the parameters (step 304). For example, as described above, the system can use volume rendering using signed distances and colors generated by the 3D model.

[0079] The system generates, from the image of the scene, one or more noisy images (step 306). The system can generate each of the one or more noisy images by sampling noise and a timestep and then generating the noisy image as described above.

[0080] The system generates a respective denoising output for each noisy image using the trained image diffusion neural network (step 308), e.g., as described above.

[0081] The system then determines a SDS gradient for the camera pose from the respective denoising outputs for each of the noisy images (step 310) as described above. For example, when there is a single noisy image for a given camera pose, the system can determine the SDS gradient as described above. When there are multiple noisy images for a given camera pose, the system can determine a respective SDS gradient for each noisy image as described above and then average the respective SDS gradients to generate the SDS gradient for the given camera pose.

[0082] The system then updates the parameters of the 3D model using the SDS gradients for the one or more camera poses in the set (step 312).

[0083] For example, the system can aggregate, e.g., average, the SDS gradients and then apply an optimizer, e.g., SGD. Adam, or AdamW, to the aggregated gradients to update the parameters.

[0084] Thus, the system updates the parameters of the 3D model by leveraging the trained diffusion neural network without needing to update the trained diffusion neural network.

[0085] As described above, optimizing the 3D model using SDS gradients requires repeatedly rendering images by submitting a significant number of queries to the neural- network-based 3D model and performing inference using a pixel-level diffusion model. As a result, this optimization may be compute intensive. To stay within the memory constraints of hardware on which the system is deployed, the system may need to use relative low- resolution images for performing the optimization. When this occurs, the first 3D mesh can miss high resolution details, given that the optimization is performed at low resolution. Additionally, when classifier-free guidance is employed with a relatively high guidance weight, the colors of the mesh can be over-saturated.

[0086] To account for one or both of these issues, the system can further refine the first 3D mesh. FIG. 4 is a flow diagram of an example process 400 for refining the first 3D mesh to generate the second 3D mesh. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a mesh generation system, e.g., the system 100 in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0087] For each view in a first set of one or more views, the system renders, using a differentiable Tenderer and from the first 3D mesh, a respective image of the scene from the view and a depth map of the scene from the view (step 402). The depth map of the scene assigns, to each pixel of the corresponding image of the scene, a respective depth value that represents a distance from the camera to the scene at the pixel.

[0088] The system can use any appropriate differentiable Tenderer to render an image and depth map for a given view using the mesh. One example of such a Tenderer is the NVdiffrast Tenderer described in Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (TOG), 39(6): 1-14, 2020.

[0089] For example, the first set of views (also referred to as the “canonical views”) can include multiple views that each correspond to a different canonical viewpoint of the object, e.g., the front of the object, the back of the object, and both sides of the object.

[0090] The system then updates, using the respective image and depth map for each view in the first set of one or more views, the first 3D mesh to generate the second 3D mesh (step 404).

[0091] As a particular example, the system can generate, using the respective image and depth map for each view in the first set of one or more view s, one or more pseudo-ground truth images for each view in the first set (step 406).

[0092] The pseudo-ground truth images generally represent photorealistic images of the object that are consistent with the 3D geometry of the generated mesh.

[0093] The system can then update the first 3D mesh by optimizing an objective that measures, for each view in the first set and for each pseudo-ground truth image for the view, an error between (i) the pseudo-ground truth image and (ii) an image rendered from the view by the differentiable Tenderer from the second 3D mesh (step 408). As described above, in some implementations, the system holds the geometry' of the mesh fixed and only updates, e.g., the texture of the mesh, by optimizing this objective.

[0094] That is. the system modifies the properties of the first 3D mesh to improve the photorealism of images generated from the 3D mesh. As one example, the obj ective can measure, for each view in the first set and for each pseudo-ground truth image for the view, a pixel-wise mean squared error between (i) the pseudo-ground truth image and (ii) an image rendered from the view by the differentiable Tenderer from the second 3D mesh.

[0095] Because the Tenderer is differentiable, the system can backpropagate through the Tenderer to update the 3D mesh using gradients of the objective.

[0096] The system can generate the pseudo-ground truth images using any of a variety of techniques.

[0097] As one example, for each view, the system can generate the one or more pseudoground images for the view using a generative neural network that is conditioned on at least the respective image and depth map for the view.

[0098] One example of a generative neural network that can be used to generate the pseudoground truth image(s) for a given view is a pre-trained depth-and-image-conditioned diffusion neural network, e.g., a latent space depth-and-image-conditioned diffusion neural network, e.g., StableDiffusion. or a pixel space depth-and-image conditioned diffusion neural network.

[0099] In some cases, the system generates the images for each of the views independently using the generative neural network. While this can generate highly realistic images of the object, which can serve to guide the re-texturing of the mesh, in some cases, when processed independently, the resulting images can exhibit 3D inconsistencies, which would give the object a different identity depending on the viewpoint.

[0100] In some implementations, to account for this, the system generates a tiled input that includes (i) a tiled image that has the respective images for the plurality of views in the first set and (ii) a tiled depth map that has the respective depth maps for the plurality of views in the first set.

[0101] For example, the system can assign each view to a respective region of a grid having the spatial dimensions of the tiled image. The system can then generate a tiled image that includes, for each view, the image for the view in the corresponding assigned region of grid and a tiled depth map that includes, for each view, the depth map for the view in the corresponding assigned region of the grid.

[0102] For example, when there are 4 views, e.g., front, back, and both sides, the system can generate a 2 x 2 grid and assign each view to one of the 4 regions of the 2 x 2 grid. The system then generates a tiled pseudo ground truth image using the generative neural network conditioned on the tiled input and then, for each view, extracts a ground truth image for the view from the tiled pseudo ground truth image. That is, the system extracts the corresponding tile of the tiled pseudo ground truth image as the pseudo ground truth image for the view.

[0103] Generating the pseudo-ground truth image as tiles from a larger, tiled image can assist in improving the 3D consistency between pseudo-ground truth images from different views. In other words, processing the larger, tiled image causes the generative neural network to incorporate context from the other tiles of the image when generating a given tile of the output image.

[0104] FIG. 5 is a flow diagram of an example process 500 for refining the second 3D mesh to generate the third 3D mesh. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a mesh generation system, e.g., the system 100 in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0105] In particular, after generating the second 3D mesh, in some cases, views generated from the mesh can still exhibit minor misalignment at their intersection as well as on unobserved object parts.

[0106] To account for this, the system can further refine the second 3D mesh.

[0107] As part of refining the second 3D mesh, the system can identify a second set of a plurality of views (step 502). For example, the second set of views can include randomly sampled views or a combination of randomly sampled views and the canonical views.

[0108] The system can then update the second 3D mesh to generate the third 3D mesh using gradients of a refinement loss computed at the second set of views (step 504).

[0109] Generally, the gradients of the refinement loss include a SDS gradient that, for each view in a second set of a plurality of views, is generated from denoising outputs generated by a trained diffusion neural network for a noisy image that is generated from an image of the scene from the view rendered using the differentiable Tenderer and from the third 3D mesh. The trained diffusion neural network can be the same as the one described above with reference to FIG. 3 or a different diffusion model, e g., a trained image-to-image diffusion neural network, either in the latent space or in the pixel space.

[0110] That is, the system computes the SDS gradient as described above with reference to FIG. 3. but then backpropagates the gradients through the differentiable Tenderer instead of the 3D model to update the mesh rather than updating the 3D model as described above. Optionally, during this updating, the system can penalize the third mesh for deviating from the second mesh.

[0111] For example, for each view in the second set of views, the system can generate a pseudo ground truth image of the scene from the view by rendering, using the differentiable Tenderer and from the second 3D mesh, a respective image of the scene from the view.

[0112] In this example, the gradients of the refinement loss can include a gradient of a photometric loss between, for each view in the second set of views, the pseudo ground truth image of the scene from the view and the image of the scene from the view rendered using the differentiable Tenderer and from the third 3D mesh. For example, this loss can be a pixelwise mean squared error between corresponding images.

[0113] For example, the gradient of the refinement loss can be a sum or a weighted sum of the SDS gradient and the photometric loss gradient.

[0114] Thus, when the gradients of the refinement loss include both the SDS gradients and the photometric loss gradient, this refinement enforces the resulting texture to not deviate too much from the original reflected in the second mesh, encouraging only regions with high SDS gradients to change.

[0115] FIG. 6 shows an example 600 of the operation of the system when receiving a text prompt “an animal with the head of a rabbit, the body of a squirrel, the antlers of a deer, and legs of a pheasant.'’

[0116] As shown in the example 600, the system first performs Neural Field Optimization via SDS gradients 610 and using randomly-sampled views to optimize the 3D model, e.g., as described above with reference to FIG. 3.

[0117] The system then performs mesh extraction 620 to extract the first mesh of the scene and then performs canonical view rendering 630 to render images of the scene from a set of canonical views, e.g., using the differentiable Tenderer as described above.

[0118] The system then performs conditional diffusion 640 to obtain a respective pseudoground truth image for each canonical view in the set, e.g., as described above with reference to FIG. 4.

[0119] The system then performs photorealistic texture optimization 650 to adjust the first mesh.

[0120] For example, the system can first perform the process 400 to update the mesh using the pseudo-ground truth images from the canonical views, i.e., using a mean-squared error loss. The system can then perform the process 500 to further update the mesh using randomly sampled views and the SDS gradient loss (optionally augmented with the photometric loss described above).

[0121] As described above, the 3D model of the scene maps a point in the scene and a viewing direction from the point to a signed distance for the point and a color along the point - viewing direction pair.

[0122] That is. given a point p, and a viewing direction d, the 3D model uses a learned function fghaving parameters 9 to map the point p(and the viewing direction d to a signed distance s(for the point and a color c(along the point - viewing direction pair. That is, fg(pj, d) = (Sj, Ci). The color can include one or more color values, e.g., RGB color values or CMYK color values or greyscale color values.

[0123] As one example, the learned function can include a single neural network, e.g., an MLP, that maps the point p;and the viewing direction d to a signed distance sffor the point and a color ct.

[0124] As another example, the learned function can include a first neural network that is configured to receive a first input that includes data representing coordinates of the point p(and to process the first input to generate an output that includes (i) the signed distance for the point and (ii) a feature vector. For example, the first neural network can be a multi-layer perceptron (MLP) that processes the coordinates to generate the output.

[0125] In this example, the learned function can also include a second neural network (c) that is configured to receive an input that includes the feature vector (generated by the first neural network) and data representing the viewing direction and process the second input to generate as output the color. For example, the second neural network can also be an MLP that processes the feature vector and the viewing direction to generate as output the color.

[0126] In order to allow the learned function to be used to perform volume rendering, the 3D model can also include an SDF to density transformation t that maps a signed distance to a volume density:

[0127] SDF to density7transformation.

[0128] When a request is received to generate an image from a given camera view, the system can then use the 3D model to render the image using volume rendering. One example of rendering an image using volume rendering will now be described.

[0129] As a particular example, to render a pixel of the image, a ray with direction d is cast from the camera center and M points are sampled along the ray.

[0130] For a given camera pose f . the operator n maps a pixel u to its color IMusing volume rendering: the color output by the learned function for point pmand direction d. crmis the output of the SDF to density transformation t for a signed distance smoutput by the learned function for point pmand direction d, and 8t= ||p£— p71|2is the distance between adjacent points.

[0131] More details about an example technique for using volume rendering to render the image, including shading and background modeling, are described in Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv: 2209. 14988, 2022.

[0132] This specification uses the term ■’configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0133] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g.. a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0134] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be. or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0135] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0136] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0137] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0138] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory- or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry'. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0139] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory' devices, including by way' of example semiconductor memory' devices, e.g., EPROM, EEPROM, and flash memory' devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0140] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory' feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0141] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0142] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g.. the Internet.

[0143] The computing system can include clients and servers. A client and server are generally remote from each other and ty pically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.

[0144] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0145] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0146] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: obtaining a text prompt; and generating a final three-dimensional (3D) mesh of a scene that is described by the text prompt, the generating comprising: optimizing parameters of a 3D model of the scene using a trained image diffusion neural network that is conditioned on the text prompt, wherein the 3D model of the scene maps a point in the scene and a viewing direction from the point to a signed distance for the point and a color along the point - viewing direction pair; and generating, in accordance with the optimized parameters of the 3d model of the scene, a first 3D mesh.

2. The method of claim 1, wherein generating the final 3D mesh further comprises: for each view in a first set of one or more views, rendering, using a differentiable renderer and from the first 3D mesh, a respective image of the scene from the view and a depth map of the scene from the view; and updating, using the respective image and depth map for each view in the first set of one or more views, the first 3D mesh to generate a second 3D mesh.

3. The method of claim 2, wherein updating, using the respective images and depth maps in the first set of one or more views, the first 3D mesh to generate a second 3D mesh comprises: generating, using the respective image and depth map for each view in the first set of one or more views, one or more pseudo-ground truth images for each view in the first set; and updating the first 3D mesh by optimizing an objective that measures, for each view in the first set and for each pseudo-ground truth image for the view, an error between the pseudo-ground truth image and an image rendered from the view by the differentiable renderer from the second 3D mesh.

4. The method of claim 3, wherein generating, using the respective image and depth map for each view in the first set of one or more views, one or more pseudo-ground truth images for each view in the first set comprises: for each view, generating the one or more pseudo-ground images for the view' using agenerative neural network that is conditioned on at least the respective image and depth map for the view.

5. The method of claim 4, wherein the generative neural network is a depth-conditioned diffusion neural network.

6. The method of claim 4 or claim 5, wherein the first set comprises a plurality of views and wherein, for each view, generating the one or more pseudo-ground images for the view using a generative neural network that is conditioned on at least the respective image and depth map for the view comprises: generating a tiled input that comprises (i) a tiled image that comprises the respective images for the plurality of views in the first set and (ii) a tiled depth map that comprises the respective depth maps for the plurality of views in the first set; generating a tiled pseudo ground truth image using the generative neural network conditioned on the tiled input; and for each view, extracting a ground truth image for the view from the tiled pseudo ground truth image.

7. The method of any one of claims 2-6, wherein generating the final 3D mesh further comprises: updating the second 3D mesh to generate a third 3D mesh using gradients of a refinement loss, the gradients of the refinement loss comprising a score distillation sampling (SDS) gradient that, for each view in a second set of a plurality of views, is generated from denoising outputs generated by a trained diffusion neural network for a noisy image that is generated from an image of the scene from the view rendered using the differentiable renderer and from the third 3D mesh.

8. The method of claim 7, wherein updating the second 3D mesh comprises: for each view in the second set of views, generating a pseudo ground truth image of the scene from the view by rendering, using the differentiable renderer and from the second 3D mesh, a respective image of the scene from the view, wherein the gradients of the refinement loss comprise a gradient of a photometric loss between, for each view in the second set of views, the pseudo ground truth image of the scene from the view and the image of the scene from the view rendered using the differentiablerenderer and from the third 3D mesh.

9. The method of claim 8, wherein the photometric loss is a mean-squared error loss between the pseudo ground truth image of the scene from the view and the image of the scene from the view rendered using the differentiable renderer and from the third 3D mesh.

10. The method of any preceding claim, wherein generating, in accordance with the optimized parameters of the 3D model of the scene, a first 3D mesh comprises: extracting the first 3D mesh as a surface at a zero-level set of signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene.

11. The method of claim 10, wherein extracting the first 3D mesh as a surface at a zerolevel set of signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene comprises: extracting the first 3D mesh using marching cubes.

12. The method of claim 10 or claim 11, wherein extracting the first 3D mesh as a surface al a zero-level set of signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene comprises: selecting a largest mesh component closer to a center of a volume defined by the signed distances generated by the 3D model of the scene in accordance with the optimized parameters of the 3D model of the scene.

13. The method of any preceding claim, wherein optimizing parameters of a 3D model of the scene using a trained image diffusion neural network that is conditioned on the text prompt comprises: optimizing the parameters using score distillation sampling (SDS) gradients that are generated using denoising outputs generated by the trained image diffusion neural network for noisy images generated from images of the scene generated from the 3D model of the scene.

14. The method of claim 13, wherein optimizing the parameters comprises repeatedly performing operations comprising:sampling a set of one or more camera poses; for each camera pose: generating an image of the scene from the camera pose using the 3D model of the scene and in accordance with the parameters; generating, from the image of the scene, one or more noisy images; generating a respective denoising output for each noisy image using the trained image diffusion neural network; determining a SDS gradient for the camera pose from the respective denoising outputs for each of the noisy images; and updating the parameters using the SDS gradients for the one or more camera poses in the set.

15. The method of claim 14, wherein generating, from the image of the scene, one or more noisy images comprises, for each noisy image: sampling random normal noise and a time step; and generating the noisy image by combining the random normal noise and the image of the scene in accordance with the time step.

16. The method of claim 14 or 15, wherein generating a respective denoising output for each noisy image using the trained image diffusion neural network comprises: processing a first diffusion input that comprises the noisy image and the text prompt using the trained diffusion neural network to generate an initial denoising output for the noisy image.

17. The method of claim 16, wherein generating a respective denoising output for each noisy image using the trained image diffusion neural network further comprises: processing a second diffusion input that comprises the noisy image and does not include the text prompt using the trained diffusion neural network to generate an initial unconditioned denoising output for the noisy image; and combining the initial denoising output and the initial unconditioned denoising output in accordance with a guidance weight to generate the denoising output.

18. The method of any one of claims 14-17, wherein determining a SDS gradient for the camera pose from the respective denoising outputs for each of the noisy images comprises:determining a gradient of the image of the scene from the camera pose with respect to the parameters of the 3D model; and determining the SDS gradient using the respective denoising outputs for each of the noisy images and the target image and the gradient of the image of the scene from the camera pose.

19. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1-18.

20. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-18.

Images