3D mesh generation using signed distance function

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for generating a mesh of a scene from a text description of the scene using a signed distance function.

Description

[Technical Field]

[0001] Cross-reference of related applications This application is a non-provisional patent application to Greek Patent Application No. 20230100331, filed on 19 April 2023, claiming priority thereto, the entirety of which is incorporated herein by reference. [Background technology]

[0002] This specification relates to generating a scene representation using machine learning models.

[0003] As an example, a neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to other layers in the network, for example, the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of its own set of weights. [Overview of the project]

[0004] This specification describes a system for generating a three-dimensional (3D) mesh of a scene from text prompts, which is implemented as a computer program on one or more computers.

[0005] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages:

[0006] The ability to generate highly photorealistic 2D images from simple text prompts has made significant progress in terms of speed and quality recently, thanks to the emergence of image diffusion models.

[0007] However, attempting to use a diffusion model trained on 2D images to monitor 3D model generation using view-dependent prompts has a major drawback.

[0008] For example, these methods generate a neural radiance field (NeRF) instead of a commonly used 3D mesh, making them impractical for most real-world applications.

[0009] As another example, these methods tend to produce supersaturated models, resulting in a cartoonish appearance in the output.

[0010] To address these issues, this specification describes techniques for generating highly photorealistic-looking 3D meshes from text prompts.

[0011] To this end, the technique described here extends NeRF to employ a signed distance function (SDF) backbone, resulting in improved 3D mesh extraction. Specifically, the system uses a 3D model of the scene to predict the signed distances of input points, and once the 3D model of the scene is optimized for a given input text prompt, the quality of the extracted mesh can be significantly improved.

[0012] Furthermore, this specification describes techniques for fine-tuning the extracted mesh texture to remove the effects of high saturation and improve the detail of the output 3D mesh.

[0013] Specifically, in some cases, one or more computers performing mesh generation may have a limited memory space, for example, due to other processes running on the computer or due to the underlying computer hardware. However, as described below, for example, to optimize a 3D model using an SDS gradient, it may be necessary to repeatedly render an image by submitting a significant number of queries to a neural network-based 3D model and performing inference using a pixel-level diffusion model. As a result, this optimization can result in a high memory load. Thus, the system needs to use relatively low-resolution images so as to stay within the memory constraints of the hardware on which the system is deployed, and in some cases, it may be necessary to use a relatively high guidance weight for optimization. When optimization is performed at a low resolution, the first 3D mesh may lose high-resolution details. In view of this, the techniques described in this specification can, as described herein, for example, fine-tune the extracted mesh texture to remove the effects of high saturation and improve the details of the output 3D mesh, while staying within the memory constraints and generating a high-quality final mesh.

[0014] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0015] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0016] [Figure 1] FIG. is a block diagram of an exemplary mesh generation system. [Figure 2] FIG. is a flowchart of an exemplary process for generating a final mesh. [Figure 3] FIG. is a flowchart of an exemplary process for optimizing the parameters of a 3D model. [Figure 4]It is a flowchart of an exemplary process for generating a second mesh from a first mesh. [Figure 5] It is a flowchart of an exemplary process for generating a third mesh from a second mesh. [Figure 6] An example of the operation of the system is shown. **DETAILED DESCRIPTION OF THE INVENTION**

[0017] Like reference numbers and names in the various drawings indicate like elements.

[0018] FIG. 1 is a block diagram of an exemplary mesh generation system 100. The mesh generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, where the systems, components, and techniques described below are implemented.

[0019] The system 100 receives a text prompt 102 and processes the text prompt 102 to generate a three-dimensional (3D) mesh 150 of a scene.

[0020] For example, the text prompt 102 may describe an object in the scene, and the 3D mesh 150 generated by the system may define the shape of the object, e.g., the geometry.

[0021] In other words, a 3D mesh of a 3D object is a collection of vertices and polygonal faces that define the shape of the 3D object.

[0022] Optionally, the mesh 150 can also specify some attributes at one or more of its vertices, faces, or edges. These can include, for example, a texture, e.g., texture coordinates or other texture data, vertex normal, reflection vector, or material parameters.

[0023] Specifically, system 100 can receive a text prompt 102 describing a scene, and then generate a 3D mesh 150 of the scene from the text prompt 102. In the embodiment shown in Figure 1, the text prompt is "Red-eyed tree frog".

[0024] The text prompt 102 can be any suitable natural language text sequence, for example, it may be received from a user of system 150. That is, system 100 can generate mesh 150 from any text prompt 102 during operation without imposing any requirements on the content of the text prompt 102.

[0025] To generate a 3D mesh 150 from the text prompt 102, system 100 optimizes the parameters 114 of the 3D model 110 of the scene using a pre-trained image diffusion neural network 120 conditioned on the text prompt 102.

[0026] For example, as will be explained in more detail below, system 100 may optimize the parameters 114 of the 3D model of the scene 110 using the 3D model of the scene 110 and a score distillation sampling (SDS) gradient calculated from the image of the scene generated using a trained spread-screen neural network 120.

[0027] Specifically, the 3D model 110 of the scene maps points within the scene and the line of sight directions from those points to the signed distance of the points and the color along the point-line of sight pairs.

[0028] The signed distance of a point can be, for example, the signed distance from the point to the surface of an object in the scene described by a text prompt. That is, the signed distance can be positive if the point is outside the surface of the object, and negative if the point is inside the object.

[0029] The parameters of the 3D model 110 may include parameters of one or more neural networks used as part of the mapping, such as a multilayer perceptron (MLP).

[0030] For example, these signed distances and colors may be used to synthesize images of the scene from the line of sight by employing SDF for density transformation, which can be learned as part of the 3D model 110, and using a volume rendering operator to render the image.

[0031] An example of compositing images of a scene from the direction of the viewer's gaze is explained below.

[0032] Therefore, the system 100 learns a signed distance function for the scene that maps points in the scene to signed distances by optimizing the 3D model 110 of the scene.

[0033] Next, the system 100 generates a first 3D mesh 112 according to the optimized parameters 114 of the 3D model 110 of the scene.

[0034] In other words, system 100 uses a learned signed distance function of the scene to generate data that identifies the vertices and faces of the first 3D mesh 112. As mentioned above, system 100 may optionally generate values ​​for one or more attributes for one or more of the vertices, faces, or edges of the mesh, for example, by generating a texture of the mesh from color values ​​mapped by the 3D model.

[0035] In some cases, system 100 uses the first 3D mesh 112 as the final 3D mesh 150.

[0036] In some other cases, the refinement engine 140 within system 100 further refines the first 3D mesh 112 to generate the final 3D mesh 150.

[0037] For example, the refinement engine 140 may generate a second 3D mesh from a first 3D mesh, and optionally generate a third 3D mesh from the second 3D mesh, and then provide a further refined mesh as the final 3D mesh 150.

[0038] In some of these cases, during refinement, system 100 may freeze the geometry of mesh 112 and update only one or more other attributes of mesh 112, such as the texture of mesh 112.

[0039] In other words, when generating a second mesh from a first mesh, when generating a third mesh from a second mesh, or both, the refinement engine 120 may update the texture of the mesh while simultaneously freezing the geometry of the mesh.

[0040] System 100 can use the final 3D mesh 150 for any of its various purposes.

[0041] For example, system 100 may use the final 3D mesh 150 to render an image of the scene from a new viewpoint.

[0042] For example, as shown in Figure 1, system 100 may render an image 160 of a red-eyed tree frog characterized by prompt 102 from a specific viewpoint. That is, the final 3D mesh may be rendered as a 2D image. The 2D image may be rendered on a display associated with the computing device.

[0043] As another example, system 100 could be part of a computer graphics pipeline, and the final 3D mesh 150 could be provided to other components of the pipeline, for example, to be included in an augmented reality (AR) or virtual reality (VR) environment, or to be included in other computer graphics applications, such as a video game or video incorporating computer graphics.

[0044] Figure 2 is a flowchart of an exemplary process 200 for generating a mesh for a scene. For convenience, process 200 is described as being performed by one or more computer systems located in one or more locations. For example, a mesh generation system preferably programmed according to this specification, such as system 100 in Figure 1, may perform process 200.

[0045] The system receives a text prompt (step 202).

[0046] Next, the system generates the final three-dimensional (3D) mesh of the scene described by the text prompts.

[0047] As part of generating the final mesh, the system generates the first 3D mesh of the scene (step 204).

[0048] Specifically, the system uses a pre-trained image-based diffusion neural network conditioned on text prompts to optimize the parameters of the 3D model of the scene (step 206).

[0049] As mentioned above, the 3D model of the scene maps points within the scene and the direction of view from those points to the signed distance of the points and the color along the point-to-view-direction pair.

[0050] Optimizing a 3D model is explained in more detail below, with reference to Figure 3.

[0051] After optimizing the 3D model, the system generates a first 3D mesh according to the optimized parameters of the 3D model in the scene (step 206).

[0052] Specifically, to generate a 3D mesh after optimizing the model, the system may extract a first 3D mesh as the surface in the zero-level set of signed distances generated by the 3D model of the scene, according to the optimized parameters of the 3D model of the scene. The system may perform this extraction using any suitable mesh generation technique. As an example, the system may perform this extraction using marching cubes.

[0053] In addition to extracting the geometry of a mesh, the system may also use mesh generation techniques to extract the texture of the mesh using the colors mapped by the 3D model. For example, the system may use the colors assigned to points on the mesh faces by the 3D model to determine the colors of the image corresponding to the faces of the mesh.

[0054] Since floaters (i.e., regions where the signed distance value from the expected object surface is approximately 0) can sometimes remain within the volume, the system may select the largest mesh component closer to the center of the volume for mesh generation. In other words, the system may select the largest mesh component closer to its center within a volume defined by the signed distance generated by the 3D model of the scene, according to the optimized parameters of the 3D model of the scene.

[0055] Optionally, the system can further refine the first 3D mesh.

[0056] Specifically, the system can generate a second 3D mesh from a first 3D mesh (step 208). At a higher level, when refining the first 3D mesh to generate the second 3D mesh, the system can refine one or more attributes of the mesh by fixing the mesh geometry and, for example, fine-tuning the mesh texture to account for suboptimal saturation and lack of detail in the initial mesh.

[0057] The process of generating a second mesh from a first mesh will be explained in more detail below with reference to Figure 4.

[0058] Furthermore, optionally, the system may generate a third 3D mesh from the second 3D mesh (step 210). At a higher level, when refining the second 3D mesh to generate the third 3D mesh, the system may refine the second mesh taking into account any deviations reflected in the second 3D mesh.

[0059] The generation of the third mesh is explained in more detail below with reference to Figure 5.

[0060] If a third mesh is generated, the system may use the third 3D mesh as the final mesh for the scene. If only a second mesh is generated, the system may use the second mesh as the final mesh for the scene. If only a first mesh is generated, the system may use the first mesh as the final mesh for the scene.

[0061] Figure 3 is a flowchart of an exemplary process 300 for optimizing the parameters of a 3D model of a scene. For convenience, the process 300 is described as being performed by one or more computer systems located in one or more locations. For example, a mesh generation system, preferably programmed according to this specification, such as system 100 in Figure 1, may perform the process 300.

[0062] As mentioned earlier, the system performs optimization using a pre-trained image diffusion neural network conditioned by text prompts.

[0063] A trained image diffusion neural network can generally be any trained text-image diffusion neural network, and it can perform diffusion in latent space or pixel space. Specifically, a trained image diffusion neural network may be trained using only 2D images and may not need to be trained on any dataset specifically designed for 3D scene generation. Examples of usable text-image diffusion neural networks include Imagen and StableDiffusion.

[0064] Specifically, in the embodiment shown in Figure 3, the system optimizes the model parameter θ using a score distillation sampling (SDS) gradient.

[0065] The SDS gradient is generated using the denoising output produced by a trained image diffusion neural network on a noise-containing image generated from a scene image derived from a 3D model of the scene.

[0066] Specifically, the SDS gradient is the gradient that pushes the rendered image, i.e., the image of the scene generated from the 3D model, into the high-probability-density region of the text prompts, as measured by the denoised output generated by the trained image diffusion neural network.

[0067] Specifically, a given image of a scene rendered using operator π, which utilizes a 3D model given camera pose ξ.

number

number

[0068] From a given time step t and noise ε, a noise-containing image (or noisy image)

number

number

number

number

[0069] The system can calculate the SDS gradient with respect to the parameter θ of the 3D model as follows:

number

number

number

number

number

[0070] Optionally, the system may incorporate classifier-less guidance to control the intensity of text conditioning for prompt y. In this case, the SDS gradient might be as follows:

number

number

[0071] Therefore, when using classifier-less guidance, the system uses a trained diffuse neural network to process a second diffuse input containing a noisy image but no text prompt, so as to generate an initial unconditionally denoised output for the noisy image, and then combines the initial denoised output and the initial unconditionally denoised output according to the guidance weights to generate a denoised output.

[0072] Therefore, as can be seen from the above equation, in order to determine the SDS gradient of the sampled camera pose, the system determines the gradient of the scene image from the camera pose with respect to the parameter θ of the 3D model, and uses the respective denoising output for each noise-containing image, the target image, and the gradient of the scene image from the camera pose to determine the SDS gradient.

[0073] To optimize the parameters of a 3D model using SDS gradients, the system may repeatedly execute iterations of process 300, for example, until a convergence criterion is met, until a threshold number of iterations have been performed, or until some other termination criterion is met.

[0074] The system samples one or more sets of camera poses (step 302).

[0075] Next, the system performs steps 304-310 for each of the one or more camera poses.

[0076] The system uses a 3D model of the scene to generate an image of the scene from the camera pose according to the parameters (step 304). For example, as mentioned above, the system may use volume rendering with signed distance and color generated by the 3D model.

[0077] The system generates one or more noise-containing images from the scene image (step 306). The system may generate one or more noise-containing images by sampling noise and time steps as described above, and then generating the noise-containing images.

[0078] The system generates a denoising output for each noise-containing image, for example, using a trained image diffusion neural network as described above (step 308).

[0079] Next, the system determines the SDS gradient of the camera pose from the denoising output of each noise-containing image, as described above (step 310).

[0080] For example, if there is a single noisy image for a given camera pose, the system may determine the SDS gradient as described above. If there are multiple noisy images for a given camera pose, the system may determine the SDS gradient for each noisy image as described above, and then average the respective SDS gradients to generate the SDS gradient for the given camera pose.

[0081] Next, the system updates the parameters of the 3D model using the SDS gradients of one or more camera poses in the set (step 312).

[0082] For example, the system may aggregate the SDS gradients, for instance by averaging them, in order to update the parameters, and then apply an optimizer such as SGD, Adam, or AdamW to the aggregated gradients.

[0083] Therefore, the system updates the parameters of the 3D model by leveraging the trained diffusion neural network, without needing to update the trained diffusion neural network itself.

[0084] As mentioned earlier, optimizing a 3D model using SDS gradients requires repeatedly rendering images by submitting a considerable number of queries to a neural network-based 3D model and performing inference using a pixel-level diffusion model. As a result, this optimization can be computationally intensive. To keep the system within the memory constraints of the hardware in which it is deployed, the system may need to use relatively low-resolution images for optimization. When optimization is performed, if the optimization is done at a low resolution, the first 3D mesh may lose high-resolution detail. Furthermore, if classifier-less guidance is employed with relatively high guidance weights, the mesh's color may become oversaturated.

[0085] Taking one or both of these problems into consideration, the system may further refine the first 3D mesh.

[0086] Figure 4 is a flowchart of an exemplary process 400 for refining a first 3D mesh to generate a second 3D mesh. For convenience, the process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a mesh generation system preferably programmed according to this specification, such as system 100 in Figure 1, may perform the process 400.

[0087] For each view in a first set of one or more views, the system renders an image of the scene from the view, and a depth map of the scene from the view, from a first 3D mesh using a differentiable renderer (step 402). The depth map of the scene assigns a depth value to each pixel of the corresponding image of the scene, representing the distance from the camera to the scene of the pixel.

[0088] The system may use any suitable differentiable renderer to render an image and depth map of a given view using a mesh. One example of such a renderer is the NVdiffrast renderer, described in "Modular primitives for high-performance differentiable rendering" by Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila, ACM Transactions on Graphics (TOG), 39(6):1-14, 2020.

[0089] For example, the first set of views (also called "canonical views") may include multiple views corresponding to different canonical viewpoints of an object, such as the front of the object, the back of the object, and both sides of the object.

[0090] Next, the system updates the first 3D mesh using the respective images and depth maps for each view within the first set of one or more views in order to generate a second 3D mesh (step 404).

[0091] As a specific example, the system may generate one or more pseudo-ground truth images for each view in a first set of one or more views, using the respective images and depth maps for each view in the first set (step 406).

[0092] Pseudo-ground truth images typically represent photorealistic images of objects, which correspond to the 3D geometry of the generated mesh.

[0093] Next, the system may update the first 3D mesh by optimizing an object that measures the error between (i) the pseudo-ground truth image and (ii) the image from the view rendered by the differentiable renderer from the second 3D mesh, for each view in the first set and for each pseudo-ground truth image of that view (step 408). As mentioned above, in some embodiments, the system fixes and holds the geometry of the mesh and updates only the texture of the mesh, for example, by optimizing this object.

[0094] In other words, the system modifies the properties of the first 3D mesh to improve the photorealism of the image generated from the 3D mesh.

[0095] As an example, the Objective may measure, for each view in the first set and for each pseudo-ground truth image of that view, the mean squared error in pixels between (i) the pseudo-ground truth image and (ii) the image from the view rendered by a differentiable renderer from a second 3D mesh.

[0096] Since the renderer is differentiable, the system can perform backpropagation through the renderer to update the 3D mesh using the gradient of the object.

[0097] The system can generate pseudo-ground truth images using one of various techniques.

[0098] For example, for each view, the system may generate one or more pseudo-ground images of the view using a generative neural network conditioned on at least each image and depth map of the view.

[0099] Examples of generative neural networks that can be used to generate pseudo-ground truth images (or more) of a given view include pre-trained depth and image-conditioned diffuse neural networks, such as latent spatial depth and image-conditioned diffuse neural networks, such as StableDiffusion, or pixel-space depth and image-conditioned diffuse neural networks.

[0100] In some cases, the system uses a generative neural network to independently generate images for each view. This can produce highly photorealistic images of objects, which can be used as a guide for retexturing meshes. However, in some cases, when processed independently, the resulting images may exhibit 3D inconsistencies, where objects have different identities depending on the viewpoint.

[0101] In some embodiments, taking this into consideration, the system generates a tiled input that includes (i) a tiled image having images of each of a plurality of views in a first set, and (ii) a tiled depth map having depth maps of each of the plurality of views in the first set.

[0102] For example, the system may assign each view to a region of a grid having spatial dimensions for the tiled image. The system may then generate, for each view, a tiled image containing the view's image in the corresponding assigned region of the grid, and for each view, a tiled depth map containing the view's depth map in the corresponding assigned region of the grid.

[0103] For example, if there are four views, such as the front, rear, and both sides, the system can generate a 2x2 grid and assign each view to one of the four regions of the 2x2 grid.

[0104] Next, the system generates a tiled pseudo-ground truth image using a generative neural network conditioned on the tiled input, and then, for each view, extracts the ground truth image of the view from the tiled pseudo-ground truth image. That is, the system extracts the corresponding tile from the tiled pseudo-ground truth image as the pseudo-ground truth image of the view.

[0105] Generating pseudo-ground truth images from tiles of a larger tiled image can help improve 3D consistency between pseudo-ground truth images from different views. In other words, by processing a larger tiled image, the generative neural network learns to incorporate context from other tiles in the image when generating a given tile in the output image.

[0106] Figure 5 is a flowchart of an exemplary process 500 for refining a second 3D mesh to generate a third 3D mesh. For convenience, the process 500 is described as being performed by one or more computer systems located in one or more locations. For example, a mesh generation system preferably programmed according to this specification, such as system 100 in Figure 1, may perform the process 500.

[0107] Specifically, after generating a second 3D mesh, in some cases the view generated from the mesh may still show slight discrepancies at the view intersections and in unobserved parts of the object.

[0108] Taking this into consideration, the system can further refine the second 3D mesh.

[0109] As part of refining the second 3D mesh, the system may identify a second set of multiple views (step 502). For example, the second set of views may include randomly sampled views, or a combination of randomly sampled views and canonical views.

[0110] Next, the system may update the second 3D mesh to generate a third 3D mesh using the refinement loss gradient calculated for the second set of views (step 504).

[0111] Typically, the refinement loss gradient includes the SDS gradient generated from the denoising output produced by a trained diffuse neural network with respect to a noise-containing image generated from a scene image from a view rendered from a third 3D mesh using a differentiable renderer, for each view in a second set of multiple views. The trained diffuse neural network may be the same as described above with reference to Figure 3, or it may be a different diffuse model in either latent space or pixel space, for example, a trained inter-image diffuse neural network.

[0112] In other words, the system calculates the SDS gradient as described above, with reference to Figure 3, but then backpropagates the gradient through a differentiable renderer instead of the 3D model to update the mesh, rather than updating the 3D model, as previously described.

[0113] Optionally, during this update, the system may impose a penalty if the third mesh deviates from the second mesh.

[0114] For example, the system could generate a pseudo-ground truth image of the scene from a view by rendering each image of the scene from the view from a second 3D mesh using a differentiable renderer for each view in a second set of views.

[0115] In this example, the refinement loss gradient may include the gradient of the photometric loss for each view in a second set of views, between the pseudo-ground truth image of the scene from the view and the image of the scene from the view rendered from a third 3D mesh using a differentiable renderer. For example, this loss could be the mean squared error in pixels between the corresponding images.

[0116] For example, the refinement loss gradient may be the sum or weighted sum of the SDS gradient and the photometric loss gradient.

[0117] Therefore, if the refinement loss gradient includes both the SDS gradient and the photometric loss gradient, this refinement forces the resulting texture not to deviate too much from the original texture reflected in the second mesh, and promotes that only areas with a high SDS gradient are changed.

[0118] Figure 6 shows an example 600 of the system's behavior when it receives the text prompt "An animal with the head of a rabbit, the body of a squirrel, the antlers of a deer, and the legs of a pheasant."

[0119] As shown in Example 600, the system first performs neural field optimization 610 via SDS gradients using randomly sampled views to optimize the 3D model, for example, as previously described with reference to Figure 3.

[0120] Next, the system performs a mesh extraction 620 to extract the first mesh of the scene, and then performs a canonical view rendering 630 to render an image of the scene from a set of canonical views, for example, using the differentiable renderer described above.

[0121] Next, the system performs a conditional diffusion 640 to acquire a pseudo-ground truth image for each canonical view in the set, as previously described with reference to Figure 4, for example.

[0122] Next, the system executes photorealistic texture optimization 650 to adjust the first mesh.

[0123] For example, the system may first execute process 400 to update the mesh using a pseudo ground truth image from the canonical view, i.e., using the mean squared error loss.

[0124] Next, the system may execute process 500 to further update the mesh using randomly sampled views and the SDS gradient loss (optionally enhanced with the photometric loss described above).

[0125] As described above, the 3D model of the scene maps points in the scene and the viewing directions from the points to the signed distance of the points and the color along the point-viewing direction pair.

[0126] That is, given a point p i and a viewing direction d, the 3D model maps the point p i and the viewing direction d to the signed distance s i of the point and the color c i along the point-viewing direction pair using a learned function f θ with parameters θ. That is, f θ (p i , d) = (s i , c i ). The color may include one or more color values, such as RGB color values or CMYK color values or grayscale color values.

[0127] As an example, the learned function may include a single neural network, such as an MLP, that maps the point p i and the viewing direction d to the signed distance s i of the point and the color c i .

[0128] As another example, the learned function receives a first input including data representing the coordinates of the point p i and, (i) the signed distance s of the pointi (ii) processing a first input to produce an output containing a feature vector, and a first neural network configured to perform the above. For example, the first neural network may be a multilayer perceptron (MLP) that processes coordinates to produce an output.

[0129] In this example, the trained function is also configured to receive an input containing a feature vector (generated by the first neural network) and data representing the direction of gaze, and to process the second input to generate a color as the output. c This could also include the following. For example, the second neural network could also be an MLP that processes feature vectors and gaze direction to generate color as output.

[0130] To enable the use of trained functions for volume rendering, the 3D model may also include an SDF-density transform t that maps signed distances to volume densities. t σ (s i )=αψ β ( -s i ) and

number

[0131] Upon receiving a request to generate an image from a given camera view, the system may use a 3D model to render the image using volume rendering.

[0132] Here, an example of rendering an image using volume rendering is explained.

[0133] As a concrete example, to render the pixels of an image, a ray of light in direction d is emitted from the center of the camera, and M points are sampled along the ray.

[0134] Given a camera pose ξ, operator π uses volume rendering to color the pixels u.

number

number

number

number

[0135] Further details on exemplary techniques for using volume rendering to render images, including shading and background modeling, are described in Dreamfusion: Text-to-3d using 2d diffusion by Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, arXiv preprint arXiv:2209.14988,2022.

[0136] This specification uses the term “configured” in relation to systems and computer program components. A system consisting of one or more computers being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform the operation or action when in operation. A computer program being configured to perform a particular operation or action means that the program, when executed by a data processing device, contains instructions that cause the device to perform the operation or action.

[0137] The subject matter and functional embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, which are executed by or control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in artificially generated propagating signals, such as machine-generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information to be transmitted to a suitable receiving device for execution by a data processing device.

[0138] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or multiple computers. The device may also be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include these. Optionally, in addition to hardware, the device may include code that constructs an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0139] Computer programs, which may also be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program in question, in a part of a file that holds one or more scripts stored in other programs or data, such as a markup language document, or in a series of collaborative files, such as a file that holds one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one place, or distributed across multiple locations and interconnected by a data communication network.

[0140] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and running on one or more identical computers.

[0141] The processes and logic flows described herein may be executed by one or more programmable computers running one or more computer programs to perform functions by performing operations on input data and generating outputs. Alternatively, the processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0142] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or integrated into dedicated logic circuits. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operationally connected to receive data from or transfer data to or both of these mass storage devices. However, a computer is not required to have such devices. Furthermore, a computer may be integrated into other devices, such as, to name a few, mobile phones, personal digital assistants (PDAs), mobile audio players or mobile video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0143] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0144] Embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device that allows the user to input into the computer, such as a mouse or trackball, in order to provide user interaction. Other types of devices may also be used to provide user interaction, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer may interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser on the user's device. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving a response message from the user.

[0145] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for handling the general and computationally intensive parts of machine learning training or production, i.e., inference workloads.

[0146] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0147] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface, a web browser, or an app on which a user can interact with the embodiments of the subject matter described herein, or one or more such combinations of backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet, for example.

[0148] A computing system may include a client and a server. The client and server are generally geographically separated and typically interact via a communication network. The relationship between the client and the server arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, may be received by the server from the device.

[0149] This specification includes details of many specific embodiments, which should not be construed as limitations on the scope of any invention or claimed scope, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein as separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may be implemented separately or in any suitable secondary combination in multiple embodiments. Furthermore, even if features are described as functioning in a particular combination and initially claimed as such, one or more features in the claimed combination may be removed from the combination in some cases, and the claimed combination may cover secondary combinations or variations of secondary combinations.

[0150] Similarly, while operations are depicted in a specific order in the drawings and enumerated in the claims, this should not be understood as meaning that such operations must be performed in a specific or sequential order shown, or that all exemplified operations must be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as meaning that such separation is necessary in all embodiments, and the program components and systems described can usually be combined into a single software product or packaged into multiple software products.

[0151] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the claims below. For example, the actions enumerated in the claims can still achieve the desired results even if they are performed in a different order. As an example, the process depicted in the accompanying drawings does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, To obtain a text prompt, This includes generating the final three-dimensional (3D) mesh of the scene described by the text prompt, and the generation includes, This includes optimizing the parameters of a 3D model of the scene using a pre-trained image diffusion neural network conditioned on the text prompt, wherein the 3D model of the scene maps points in the scene and lines of sight from those points to the signed distance of the points and the color along the point-line of sight pairs, and further generates A method comprising generating a first 3D mesh according to the optimized parameters of the 3D model of the scene.

2. The process of generating the aforementioned final 3D mesh further involves, For each view in a first set of one or more views, the respective image of the scene from the view and the depth map of the scene from the view are rendered from the first 3D mesh using a differentiable renderer. To generate a second 3D mesh, the first 3D mesh is updated using the respective images and depth maps for each view within the first set of one or more views. The method according to claim 1, including the method described in claim 1.

3. Updating the first 3D mesh using the respective images and depth maps of the first set of one or more views in order to generate a second 3D mesh is: Using the respective images and depth maps for each view in the first set of one or more views, one or more pseudo-ground truth images are generated for each view in the first set. The first 3D mesh is updated by optimizing an object that measures the error between the pseudo-ground truth image and the image from the view rendered by the differentiable renderer from the second 3D mesh for each view in the first set and for each pseudo-ground truth image of the view. The method according to claim 2, including the method described in claim 2.

4. Using the respective images and depth maps for each view in the first set of one or more views, one or more pseudo-ground truth images are generated for each view in the first set. For each view, generate one or more pseudo-ground truth images of the view using a generative neural network conditioned on at least the respective images and depth maps of the view. The method according to claim 3, including the method described in claim 3.

5. The method according to claim 4, wherein the generative neural network is a depth-conditional diffuse neural network.

6. The first set includes a plurality of views, and for each view, one or more pseudo-ground truth images of the view are generated using a generative neural network conditioned on at least the respective images and depth maps of the view. (i) generating a tiled input which includes a tiled image having the respective images of the plurality of views in the first set, and (ii) a tiled depth map having the respective depth maps of the plurality of views in the first set. Using the generative neural network conditioned by the tiled input, a tiled pseudo-ground truth image is generated. For each view, the ground truth image of that view is extracted from the tiled pseudo-ground truth image, The method according to claim 4 or claim 5, including the method according to claim 4 or 5.

7. The process of generating the aforementioned final 3D mesh further involves, The method according to any one of claims 2 to 6, comprising updating the second 3D mesh to generate a third 3D mesh using a refinement loss gradient, wherein the refinement loss gradient includes, for each view in a second set of a plurality of views, a score distillation sampling (SDS) gradient generated from denoising output generated by a trained diffuse neural network with respect to a noise-containing image generated from an image of the scene from the view rendered from the third 3D mesh using the differentiable renderer.

8. Updating the second 3D mesh described above means For each view in the second set of views, the process includes generating a pseudo-ground truth image of the scene from the view by rendering each image of the scene from the view from the second 3D mesh using the differentiable renderer, The gradient of the refinement loss includes, for each view in the second set of views, the gradient of the photometric loss between the pseudo-ground truth image of the scene from the view and the image of the scene from the view rendered from the third 3D mesh using the differentiable renderer. The method according to claim 7.

9. The method according to claim 8, wherein the photometric loss is the mean squared error loss between the pseudo-ground truth image of the scene from the view and the image of the scene from the view rendered from the third 3D mesh using the differentiable renderer.

10. To generate a first 3D mesh according to the optimized parameters of the 3D model of the scene, Extracting the first 3D mesh as the surface in the zero-level set of signed distances generated by the 3D model of the scene according to the optimized parameters of the 3D model of the scene, The method according to any one of claims 1 to 9, including the method described in any one of claims 1 to 9.

11. Extracting the first 3D mesh as a surface in the zero-level set of signed distances generated by the 3D model of the scene according to the optimized parameters of the 3D model of the scene is: Using a marching cube, extract the first 3D mesh. The method according to claim 10, including the method described in claim 10.

12. Extracting the first 3D mesh as a surface in the zero-level set of signed distances generated by the 3D model of the scene according to the optimized parameters of the 3D model of the scene is: Selecting the largest mesh component closest to the center of the volume defined by the signed distance generated by the 3D model of the scene, according to the optimized parameters of the 3D model of the scene. The method according to claim 10 or claim 11, including the method described in claim 11.

13. Using the pre-trained image diffusion neural network conditioned by the aforementioned text prompt to optimize the parameters of the 3D model of the scene is: The method according to any one of claims 1 to 12, comprising optimizing the parameters with respect to a noise-containing image generated from an image of the scene generated from the 3D model of the scene, using a score distillation sampling (SDS) gradient generated using the denoising output generated by the trained image diffusion neural network.

14. Optimizing the aforementioned parameters Sampling one or more sets of camera poses, For each camera pose, Using the 3D model of the scene, an image of the scene is generated from the camera pose according to the parameters. To generate one or more noise-containing images from the aforementioned images of the aforementioned scene, Using the aforementioned trained image diffusion neural network, a denoising output is generated for each noise-containing image. The SDS gradient of the camera pose is determined from the respective noise reduction outputs for each of the noise-containing images. The parameters are updated using the SDS gradient of one or more camera poses within the set. The method according to claim 13, which includes repeatedly performing an operation including the operation.

15. Generating one or more noise-containing images from the aforementioned image of the aforementioned scene is performed for each noise-containing image, Sampling random normal noise and time steps, The noise-containing image is generated by combining the random normal noise and the image of the scene according to the aforementioned time step, The method according to claim 14, including the method described in claim 14.

16. Using the aforementioned trained image diffusion neural network, generating a denoising output for each noise-containing image is: Using the trained diffuse neural network, process a first diffuse input, including the noisy image and the text prompt, to generate an initial denoising output of the noisy image. The method according to claim 14 or 15, including the method described in claim 14 or 15.

17. Using the aforementioned trained image diffusion neural network, generating a denoising output for each noise-containing image is further... Using the aforementioned trained spreading neural network, a second spreading input, which includes the noise-containing image but does not include the text prompt, is processed to generate an initial unconditionally denoised output of the noise-containing image. To generate the denoised output, the initial denoised output and the initial unconditional denoised output are combined according to guidance weights, The method according to claim 16, including the method described in claim 16.

18. Determining the SDS gradient of the camera pose from the respective noise reduction outputs for each of the noise-containing images is as follows: With respect to the parameters of the 3D model, the gradient of the image of the scene from the camera pose is determined, The SDS gradient is determined using the respective noise reduction outputs for each of the noise-containing images, the target image, and the gradient of the image of the scene from the camera pose. The method according to any one of claims 14 to 17, including the method described in any one of claims 14 to 17.

19. One or more computers, A system comprising one or more storage devices for storing instructions, wherein when an instruction is executed by the one or more computers, the system causes the one or more computers to perform the respective operations of the method according to any one of claims 1 to 18.

20. One or more computer-readable storage media for storing instructions, wherein, when executed by one or more computers, the instructions cause one or more computers to perform the respective operations of the method according to any one of claims 1 to 18.

Images