Autoregressive neural network for generating a polygon mesh of a 3D object

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for generating data for a three-dimensional mesh of a specified object using an autoregressive neural network.

Description

BACKGROUND

[0001] This specification relates to generating a three-dimensional mesh of an object using a neural network.

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0003] This specification describes a system of one or more computers in one or more physical locations that generates a three-dimensional (3D) mesh of an object using an autoregressive mesh generation neural network, i.e., generates an output sequence that specifies vertices and faces of a three-dimensional mesh.

[0004] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0005] Conventional methods of generating a representation of an object using a neural network have used alternative representations of the shape of the object, e.g., voxels, point clouds, occupancy functions, and surfaces, to generate 3D objects. In these conventional methods, mesh reconstruction is left as a post-processing step, and the resulting meshes have at best varying quality. On the other hand, the described techniques use an autoregressive neural network to directly generate 3D meshes, and effectively address the disordered elements and discrete face structure that exist in meshes. This results in mesh diversity and realistic geometry, and is directly usable for various graphics applications.

[0006] Furthermore, the generated meshes can be conditioned on any of a variety of context inputs, allowing the system to effectively generate meshes of a particular type of object that have realistic geometry but are also characterized by the context input.

[0007] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 An example mesh generation system is shown.

[0009] Figure 2 An example of the operation of a mesh generation neural network is shown.

[0010] Figure 3 is a flow diagram of an example process for generating an output sequence that specifies a mesh.

[0011] Figure 4 is a flowchart of an example process for generating vertices of a mesh.

[0012] Figure 5 is a flowchart of an example process for generating faces of a mesh.

[0013] Like reference numbers and designations in different drawings represent the same element.

[0014] DETAILED DESCRIPTION

[0015] This specification describes a system of one or more computers in one or more physical locations that generates three-dimensional meshes using an autoregressive mesh generation neural network, i.e., generates an output sequence of vertices and faces of a three-dimensional mesh that specifies an object.

[0016] For example, the mesh generation neural network can include a vertex generation neural network that generates a first sequence of vertex coordinates that includes vertices of a mesh, and a face generation neural network that generates a second sequence of mesh tokens that identify which vertices make up each face of the mesh.

[0017] Because the mesh generation neural network is autoregressive, the mesh generation neural network can be used to generate many different but high-quality meshes by sampling from the probability distribution generated by the neural network at each output position, given the same context input.

[0018] Once the output sequence is generated, the system can generate and render the mesh, or can provide the sequence to another system for mesh generation and rendering. For example, the system can generate one or more meshes from any given context input, and can provide the generated meshes for presentation to a user, e.g., as candidates for insertion into a video game, a virtual reality scene, a movie, or another virtual world. As another example, the generated meshes can be used as output of a computer vision system, or inserted into a virtual environment for interaction with a robot, e.g., as part of learning a control policy for the robot.

[0019] Accordingly, one or more other neural networks can be trained based on the output mesh. For example, a computer vision neural network can be trained to recognize shapes of the output mesh. Alternatively, an agent can be trained to perform a task in a virtual environment on an object represented by the output mesh. This can include positioning the output mesh in the virtual environment, and instructing a reinforcement learning agent to perform actions on the object represented by the output mesh to perform the task. The reinforcement learning agent can be trained based on rewards received in response to actions taken in the virtual environment.

[0020] The generated mesh can be used to render an image that includes the object (e.g., an image of the object or an image of the object within a scene such as a virtual environment). The rendering process can produce a two-dimensional (2D) image that represents a view of a three-dimensional (3D) scene that includes the object. The process can specify a position of the object in the environment. This can be part of a virtual reality or augmented reality rendering process. Thus, the position and / or orientation of the view (e.g., field of view) of the environment can depend on the position and / or orientation of the view (e.g., field of view) of the physical (real-world) environment (e.g., based on position and / or orientation measurements from one or more sensors such as one or more sensors within a virtual reality or augmented reality headset).

[0021] One or more of the generated meshes can be directly integrated into a computer graphics engine (e.g., a graphics processing unit, GPU). As described above, these can be used by the computer graphics engine to render images of the virtual environment. The computer graphics engine can store one or more of the generated meshes and access them when rendering of one or more objects represented by the one or more generated meshes is required (e.g., in the case that the one or more objects form part of the virtual environment). As part of the rendering process, deformations can be applied to the stored meshes (e.g., to reflect changes in the structure of the object being rendered). The meshes can be used for ray tracing. For example, ray tracing can render where virtual rays are modeled in the virtual environment, and where reflections of objects represented by particular meshes are modeled as part of the rendering process. This can include identifying intersections between the rays and the meshes, and shading the intersections based on properties of the objects.

[0022] As another example, the generated mesh can be output to one or more (additional) manufacturing devices for manufacturing of one or more objects having a shape described by the generated mesh. The output can include slicing the mesh and / or converting the mesh to a format suitable for manufacturing (e.g., via additive manufacturing). The format can be a set of instructions (e.g., G-code) that, when executed by the one or more manufacturing devices, cause the one or more manufacturing devices to manufacture the one or more objects. The set of instructions can be computer-readable instructions. Converting the mesh to a format for manufacturing can be based on user preferences, such as layer thickness in the case of additive manufacturing.

[0023] Figure 1 An example mesh generation system 100 is shown. The mesh generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0024] The mesh generation system 100 is a system that generates, using an autoregressive mesh generation neural network 110, an output sequence 150 of three-dimensional meshes 152 that specify an object.

[0025] A three-dimensional mesh of a three-dimensional object is a collection of vertices and polygonal faces that define the shape of the three-dimensional object.

[0026] In some cases, all polygonal faces have the same size, i.e., the same number of vertices. As one particular example, all polygonal faces can be triangles with three vertices.

[0027] In some other cases, faces can have variable sizes, i.e., variable numbers of vertices. Allowing faces to have different numbers of vertices can result in a more compact representation of an object, i.e., a representation with fewer faces. For example, any large flat surface of an object can be represented as a single face with many vertices, rather than a collection of multiple triangles.

[0028] In this specification, the system 100 is described as being configured to generate meshes composed of polygonal faces with variable numbers of vertices. However, it will be understood that the system 100 can alternatively be configured to limit the number of vertices in each face in a mesh, such that each face has the same number of vertices.

[0029] In some implementations, the system 100 performs unconditional mesh generation, i.e., generates the output sequence 150 without explicitly conditioning on the input. In these cases, the system 100 will generate a realistic mesh that represents a mesh that appears in the training data used to train the autoregressive mesh neural network 110.

[0030] In some other implementations, the system 100 performs conditional mesh generation. In these cases, the system 100 receives a context input 102 that specifies a context of a mesh to be generated, and uses the neural network 110 to generate a realistic mesh of an object that is characterized by the context input 102.

[0031] The system 100 can be configured to receive any of a variety of context inputs.

[0032] As one example, the context input can be an input that identifies an object class of the object, i.e., an input that specifies an object class to which the generated mesh representation should belong.

[0033] As another example, the context input can be an image of the object, i.e., an input that is an image of the object that the generated mesh should represent.

[0034] As another example, the context input can be a voxelized representation of the object, i.e., a collection of voxels that represent the object that the generated mesh should represent.

[0035] Conditional mesh generation will be described in more detail below with reference to Figures 3-5 Conditional mesh generation, i.e., how the neural network 110 is conditioned on the context input 102, is described in more detail below.

[0036] In general, the output sequence 150 generated by the system 100 represents the vertices and faces of a three-dimensional mesh 152 of an object. In particular, the output sequence can include a first sub-sequence including vertex coordinates of the vertices of the mesh, followed by a second sub-sequence including mesh tokens that identify which vertices make up each of the faces of the mesh.

[0037] Once the output sequence 150 is generated, the system 100 can generate and render the mesh 152, or can provide the sequence 150 to another system for mesh generation and rendering. For example, the system 100 can generate one or more meshes from any given context input 102, and can provide the generated meshes for presentation to a user, e.g., as candidates for insertion into a video game, a virtual reality scene, a movie, or another virtual world. As another example, the generated meshes can be used as output of a computer vision system, or inserted into a virtual environment for interaction with a robot, e.g., as part of learning a control policy for the robot.

[0038] Accordingly, one or more other neural networks can be trained based on the output mesh. For example, a computer vision neural network can be trained to recognize shapes of the output mesh. Alternatively, an agent can be trained to perform a task in a virtual environment on an object represented by the output mesh. This can include positioning the output mesh in the virtual environment, and instructing a reinforcement learning agent to perform actions on the object represented by the output mesh to perform the task. The reinforcement learning agent can be trained based on rewards received in response to actions taken in the virtual environment.

[0039] The generated mesh can be used to render an image that includes the object (e.g., an image of the object or an image of the object within a scene such as a virtual environment). The rendering process can produce a two-dimensional (2D) image that represents a view of a three-dimensional (3D) scene that includes the object. The process can specify a position of the object in the environment. This can be part of a virtual reality or augmented reality rendering process. Thus, a position and / or orientation of a view (e.g., field of view) of the environment can depend on a position and / or orientation of a view (e.g., field of view) of a physical (real-world) environment (e.g., based on position and / or orientation measurements from one or more sensors such as one or more sensors within a virtual reality or augmented reality headset).

[0040] One or more generated meshes can be integrated directly into a computer graphics engine (e.g., a graphics processing unit, GPU). As described above, these can be used by the computer graphics engine to render images of a virtual environment. The computer graphics engine can store one or more generated meshes and access them when it is necessary to render one or more objects represented by the one or more generated meshes (e.g., in the case that the one or more objects form part of a virtual environment). As part of the rendering process, deformations can be applied to the stored meshes (e.g., to reflect changes in the structure of the objects being rendered). The meshes can be used for ray tracing. For example, ray tracing can render where virtual rays are modeled in a virtual environment, and where reflections of objects represented by particular meshes are modeled as part of the rendering process. This can include identifying intersections between the rays and the meshes, and shading the intersections based on properties of the objects.

[0041] As another example, a generated mesh can be output to one or more (additional) manufacturing devices for manufacturing one or more objects having a shape described by the generated mesh. The output can include slicing the mesh and / or converting the mesh into a format suitable for manufacturing (e.g., via additive manufacturing). The format can be a set of instructions (e.g., G-code) that, when executed by the one or more manufacturing devices, cause the one or more manufacturing devices to manufacture the one or more objects. The set of instructions can be computer-readable instructions. Converting the mesh into a format for manufacturing can be based on user preferences, such as layer thickness in the case of additive manufacturing.

[0042] The output sequence 150 includes, at each of a plurality of output positions within the output sequence 150, a respective mesh token. The system 100 generates the mesh tokens in an autoregressive manner using the autoregressive mesh generation neural network 110, i.e., by generating the mesh tokens one-by-one, generating a mesh token at any given position conditioned on the mesh tokens that have already been generated at any earlier output positions in the output sequence 150. Depending on the output position, a mesh token can include any of the following: a coordinate value for a mesh vertex, an identifier for a mesh vertex, a stop token, or a new face token. Each of these will be described in more detail below.

[0043] In other words, at any given output position, the system 100 generates a mesh token for the given output position by processing an input for the given output position using the autoregressive mesh generation neural network 110, to generate a probability distribution over possible mesh tokens for the given output position, where the neural network 110 is conditioned on (at least) any mesh tokens that have already been selected at any earlier output positions in the output sequence.

[0044] System 100 then uses a probability distribution to select the grid token for the output position from the possible grid tokens for the output position.

[0045] How system 100 generates input for a given output position and which tokens are possible for a given output position depends on the position of the given output position in the output sequence 150.

[0046] Specifically, as will be described in more detail below, system 100 first generates a grid token representing the vertex coordinates of a polygonal face in the grid, and then generates a grid token identifying which vertices constitute each of the multiple faces in the grid.

[0047] Therefore, output sequence 150 includes a first set of output positions, followed by a second set of output positions. The mesh tokens at the first set of output positions represent the coordinates of the mesh vertices, and the mesh tokens at the second set of output positions identify the vertices constituting each of the plurality of faces.

[0048] In other words, System 100 decomposes the joint distribution of vertices and faces into a product of conditional distributions, one for possible vertex coordinates (each vertex coordinate is conditional on the already selected coordinates) and one for face vertices (each face vertex is conditional on the already selected face vertices).

[0049] In order to generate data that defines the mesh, i.e., to generate output sequence 150, the system first samples vertices from the vertex distribution, and then samples faces from the face distribution conditioned on the sampled vertices.

[0050] This will be referenced below. Figure 2 To describe in more detail.

[0051] Figure 2 An example of the operation of the autoregressive grid generating neural network 110 is shown.

[0052] exist Figure 2 In the example, neural network 110 includes vertex generation neural network 210 and face generation neural network 220.

[0053] like Figure 2 As shown, the system first uses a vertex generation neural network to generate mesh vertices, and then uses the generated vertices as conditions to generate mesh surfaces. Figure 2 In the example, the vertex generation neural network 210 is shown as generating the z, y, and x coordinates of vertices at three output locations.

[0054] To generate a given mesh vertex, the system generates the x, y, and z coordinates of the vertex in a three-dimensional coordinate system, where z is the vertical axis.

[0055] More specifically, the system generates the vertex coordinates as a flattened sequence of concatenated value tuples, each tuple representing the three coordinates of a different one of the mesh vertices. As a particular example, within each tuple, the three values can represent the (z, y, x) coordinates of the mesh in that order.

[0056] Accordingly, the output sequence 150 includes a first set of output positions that specify coordinates of the mesh vertices. That is, each output position in the first set corresponds to a z, y, or x coordinate of one of the mesh vertices.

[0057] In particular, to generate the coordinate at any given output position in the first subset, the system processes the sequence of coordinates that have already been generated at output positions preceding the given output position using a vertex generation neural network.

[0058] The vertex generation neural network includes a vertex encoder (referred to in the figures as a “vertex transformer”) that generates a respective embedding for each of the coordinates that have already been generated, and a vertex decoder that processes the sequence of embeddings to generate a probability distribution over possible values for the coordinate at the given output position. This can equivalently be referred to as generating an embedding using the vertex encoder and then processing the sequence of embeddings using the vertex decoder.

[0059] As used in this specification, an embedding is a numerical representation of a token. In particular, an embedding is a numerical representation in an embedding space, i.e., an ordered collection of a fixed number of numerical values, where the number of numerical values is equal to the dimensionality of the embedding space. For example, an embedding can be a vector of floating point values or other types of numerical values.

[0060] For example, the system can quantize the space of possible coordinate values, and for each output position in the first set, the vertex decoder can generate a probability distribution over quantized coordinate values in the resulting quantized set, i.e., a Categorical distribution. The system then selects a mesh token, i.e., a coordinate value, for the given output position using the probability distribution. This will be described in more detail below with reference to Figure 4

[0061] In some implementations, the system applies a mask to the logits output by the vertex generation neural network to mask out any mesh tokens that are not valid for the given output position. This will also be described below with reference to Figure 4

[0062] ​​The vertex encoder can generate the embedding for a given coordinate value in various ways. As a particular example, the embedding can be a combination, e.g., concatenated, summed, or averaged, of two or more of: (1) a coordinate embedding that indicates whether the input token is an x, y, or z coordinate; (2) a position embedding that indicates which vertex in the sequence the token belongs to; or (3) a value embedding that expresses the quantized coordinate value of the token. The embeddings (1), (2), and (3) can be predetermined or can be jointly learned with training of the vertex generation neural network.

[0063] In some implementations, rather than generating grid tokens that represent individual coordinates of a vertex, i.e., where each token represents only one of an x, y, or z coordinate, the system can generate grid tokens that each represent all three coordinates of a given vertex, i.e., each tuple can represent a complete coordinate tuple of a single vertex. For example, the system can model the probability distribution of the three coordinates of a vertex using a mixture of discrete logits.

[0064] Once the sequence of vertex coordinates has been generated, i.e., the grid tokens at the first subset of positions in the output sequence 150 have been generated, the system uses a face generation neural network 220 to generate data for faces in the three-dimensional mesh that specifies the object. Figure 2 An example of the face generation neural network 220 generating four grid tokens that identify four vertices that make up a rectangular face of the resulting mesh.

[0065] Each face is defined (“made up of”) by three or more vertices specified by the first subset of positions in the output sequence 150. In particular, the system uses the face generation neural network to generate grid tokens at a second subset of output positions in the output sequence 150. Each grid token at one of the positions in the second subset either (i) identifies one of the vertices generated by the vertex generation neural network or (ii) is a new face token that indicates that a previous face is complete and a new face will begin from the next grid token at the next output position.

[0066] Thus, once the grid tokens at the second subset of output positions have been generated, the faces in the mesh are: (i) a first face made up of the vertices identified by the grid tokens before the first new face token in the second subset; (ii) respective faces made up of the grid tokens at the positions between any two new face tokens in the second subset; and (iii) a last face made up of the vertices identified by the grid tokens after the last new face token in the output sequence.

[0067] To generate the grid tokens at the second subset, the face generation neural network includes a face encoder that generates a respective contextual embedding of an input set that consists of (i) each of the plurality of vertices specified by the output of the vertex generation neural network, (ii) the new face tokens, and (iii) the stop token.

[0068] The face generation neural network further includes a face decoder that autoregressively generates the mesh tokens at a second subset of the positions, the second subset of the positions defined conditioned on the respective context embedding of the input set.

[0069] Reference is made below to Figure 5 Generating context embeddings and mesh tokens in the second subset is described.

[0070] The system (or another system) trains the autoregressive mesh generation neural network on training data, the training data including training output sequences that specify meshes in the training data set, and, in cases where the system has the capability to perform conditional mesh generation, respective context inputs for each training output sequence.

[0071] In particular, when the autoregressive mesh generation neural network includes a vertex generation neural network and a face generation neural network, the system is capable of training the vertex generation neural network to optimize a target function that measures a probability assigned by the vertex generation neural network to the ground truth vertex coordinates in the training output sequences. The system is further capable of training the face generation neural network to optimize a second target function that measures a probability assigned by the face generation neural network to the faces specified by the ground truth output sequences, conditioned on the ground truth vertex coordinates from the ground truth training sequences.

[0072] As a particular example, the system is capable of training the vertex generation neural network by maximizing a log-likelihood objective that measures a log-likelihood of assigning to the ground truth vertex coordinates in a given ground truth output sequence by the probability distribution generated by the vertex generation neural network at the corresponding positions in the first subset of the output positions.

[0073] As another particular example, the system is capable of training the face generation neural network by maximizing a log-likelihood objective that measures a log-likelihood of assigning to the ground truth vertices in a given ground truth output sequence by the probability distribution at the corresponding positions in the second subset of the output positions.

[0074] Figure 3 is a flowchart of an example process 300 for generating an output sequence that defines a three-dimensional mesh. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, the system performing the process 300 can be the mesh generation system 100, e.g., as described above. Figure 1 An appropriately programmed mesh generation system of the mesh generation system 100 can perform the process 300.

[0075] Optionally, the system obtains context input (step 302). In particular, as described above, the system can perform unconditional mesh generation, in which the system generates a realistic mesh that is not conditioned on any input, or can perform conditional mesh generation, in which the system generates a realistic mesh of an object that is characterized by context input.

[0076] The system generates, using the autoregressive mesh generation neural network, an output sequence of vertices and faces of a three-dimensional mesh of the specified object (step 304).

[0077] The output sequence comprises a respective mesh token at each of a plurality of output positions, and the system generates the mesh tokens sequentially, i.e. one-by-one.

[0078] To generate a mesh token at a given output position, the system processes an input for a time step for the autoregressive mesh generation neural network, with the neural network conditioned on at least any mesh tokens that have been selected at any earlier output position, to generate a probability distribution over possible mesh tokens for the given output position. The possible mesh tokens will typically differ for different output positions in the output sequence. When context input has been obtained, the autoregressive mesh generation neural network is also conditioned on the context input for at least some of the output positions.

[0079] As a particular example, the output sequence can comprise a first set of output positions, followed by a second set of output positions. The mesh tokens at the first set of output positions can be respective coordinates for each of a plurality of vertices of the mesh, i.e. each mesh token in the first output position is a coordinate for one of the vertices of the mesh. The mesh tokens at the second set of output positions identify a vertex from a plurality of vertices of each of a plurality of faces that make up the mesh, i.e. each mesh token in the second output position identifies a respective vertex of one of the faces of the mesh.

[0080] Generating the probability distribution for a position in the first set of output positions is described below with reference to Figure 4 .

[0081] Generating the probability distribution for a position in the second set of output positions is described below with reference to Figure 5 .

[0082] The system then selects the mesh token at the output position from the possible mesh tokens for the given output position using the probability distribution. For example, the system can select the possible mesh token with the highest probability, or sample the mesh token from the probability distribution.

[0083] When the system is configured to sample from the probability distribution, due to the autoregressive nature of the computation, the system can generate a plurality of different realistic meshes for the same context input.

[0084] Figure 4 is a flowchart of an example process 400 for generating a mesh token for a given output position in the first output position subset. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a mesh generation system, such as the mesh generation system 100 of Figure 1 An appropriately programmed mesh generation system of the mesh generation system 100 is capable of performing the process 400.

[0085] The system is capable of repeatedly performing the process 400 to generate respective coordinates for all vertices of the mesh. Because the mesh generated by the system typically has a variable number of vertices, the system is capable of repeatedly performing the process 400 until a stopping criterion is met.

[0086] For example, the system is capable of repeatedly performing the process 400 to add coordinates to the first output position subset until the system selects a predetermined stopping token. Once the predetermined stopping token has been selected, the system considers the mesh tokens selected prior to the stopping token as tokens at the first output position subset.

[0087] The system generates a respective embedding for any vertex coordinates that have been selected at any earlier output position in the output sequence (step 402). In particular, the system is capable of generating the respective embedding by processing the embedding using a vertex encoder as described above with reference to Figure 2 The system generates a respective embedding for any vertex coordinates that have been selected at any earlier output position in the output sequence (step 402). In particular, the system is capable of generating the respective embedding by processing the embedding using a vertex encoder as described above with reference to

[0088] The system processes the respective embedding using a vertex decoder to generate a probability distribution for the given output position (step 404).

[0089] As a particular example, the vertex decoder can be a neural network having a Transformer architecture, i.e., can have the decoder architecture of a Transformer.

[0090] A Transformer is a type of neural network that repeatedly applies an attention mechanism to its input as part of generating an output. Depending on the configuration of the Transformer, the Transformer can include both an encoder and a decoder, only an encoder, or only a decoder. Examples of configurations of Transformers and details of components of Transformers, e.g., attention layers, feed-forward layers within attention network layers, and output layers of attention neural networks that produce network outputs, are described in more detail in Vaswani et al., Attention Is All You Need, arXiv: 1706.03762, Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, arXiv: 1910.10683, and Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv: 1810.04805, the entire contents of which are incorporated by reference herein in their entirety

[0091] In general, the output of the vertex decoder includes the respective logits, i.e., respective scores, for all possible coordinates and the stop token. In some implementations, the system then applies a softmax to the respective logits to generate a probability distribution.

[0092] In some other implementations, the system first masks the logits for any possible mesh tokens of the output positions that are invalid given the mesh tokens selected at earlier positions that are already in the first subset, and then, after the masking, applies a softmax to generate a probability distribution. Masking the logits for a mesh token means setting the logits to a value, e.g., negative infinity, such that the resulting probability for the mesh token is zero when processed by the softmax (and so the system does not select that mesh token when sampling from the distribution or when selecting the highest probability in the distribution).

[0093] For example, the system can mask the logits for the stop token unless the preceding coordinate is the last coordinate of the vertex, e.g., the x coordinate of the vertex when the tuple is ordered (z, y, x), i.e., such that the system does not stop generating coordinates, but the coordinates of the last vertex are only partially generated.

[0094] As another example, the system can impose additional requirements on the generation of a vertex, and then, at some or all of the output positions, mask the logits to ensure that those requirements are met.

[0095] As one particular example, the system can impose a requirement on the order in which the vertices are generated based on the position of a specified one of the three coordinates of the vertices. When generating the specified coordinate of each vertex, the system can mask the logits for those values of the specified coordinate that violate the requirement imposed on the order.

[0096] For example, the system can require that the z-coordinates of the vertices be monotonically increasing, i.e., that the vertex with the smallest z-coordinate value be generated first, and that the z-coordinates of the vertices monotonically increase from there. In this example, when generating the z-coordinate of any vertex other than the first vertex, the system can mask the logits for any possible coordinate value that is equal to or less than the z-coordinate value of a prior vertex (or, equivalently, that is not greater than all of the z-coordinate values of all prior vertices).

[0097] The system selects a grid token at the output position using a probability distribution, e.g., by sampling a grid token from the probability distribution (step 406). As described above, if the system samples or selects the stop token, the system identifies the vertices that have been generated as the only vertices in the grid.

[0098] Figure 5 is a flowchart of an example process 500 for generating a grid token at an output position in a second subset of output positions. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, the system can include a grid generation system, such as the grid generation system 100 of FIG. 1. Figure 1 An appropriately programmed grid generation system of the grid generation system 100 of FIG. 1 can perform the process 500.

[0099] The system generates respective contextual embeddings of the input set, which consist of (i) each of the plurality of vertices, (ii) the new face token, and (iii) the stop token (step 502). These embeddings are referred to as “contextual” embeddings because the embedding of any given input in the set depends on the values of the other inputs in the set.

[0100] To generate the respective contextual embeddings, the system processes (i) each of the plurality of vertices, (ii) the new face token, and (iii) the stop token using a face encoder. As described above, the new face token is a designated token that, when selected, indicates that the current face is complete, and that subsequent grid tokens will indicate the next vertex in the new face.

[0101] The face encoder is a neural network configured to process a sequence of inputs to generate a respective embedding of each input in the sequence. As a particular example, the face encoder can be a neural network with a Transformer architecture, i.e., can have the architecture of an encoder of a Transformer.

[0102] The system can then repeatedly perform steps 504-510 to generate the faces of the three-dimensional mesh, i.e., to identify the vertices that make up each face of the three-dimensional mesh. Because the mesh generated by the system typically has a variable number of faces, the system can repeatedly perform steps 504-510 until a stopping criterion is met.

[0103] For example, the system can repeatedly perform steps 504-510 to identify the face vertices until the system selects a stop token. Once the stop token is selected, the system considers the faces that have been defined by the mesh tokens selected prior to the stop token to be the faces of the mesh.

[0104] In other words, the system performs steps 504-510 for each output position in the second set of output positions to select a mesh token at the given output position.

[0105] In particular, to select a mesh token at a given output position in the second set of output positions, the system generates a decoder input for the given output position that includes a respective embedding of each mesh token generated at any earlier output position in the second set (step 504). In particular, the system generates a sequence of respective embeddings of mesh tokens.

[0106] The system can compute the embedding of a given mesh token in any of a variety of ways. As one particular example, the system can generate one or more embeddings that characterize the location of the vertex identified by the mesh token within the mesh, and then combine these embeddings with a contextual embedding for the mesh token, e.g., concatenate, average, or sum. The embeddings that characterize the location of the vertex can include an embedding of an identifier of the face to which the vertex belongs, an embedding of the location of the vertex within the face, or both. The embeddings can be predetermined or jointly learned with training of a neural network.

[0107] The system processes the decoder input using a face decoder to generate a pointer vector (step 506). The pointer vector is a vector having the same dimensionality as the contextual embedding.

[0108] The face decoder is a neural network configured to process a sequence of inputs and generate a vector as output. As a particular example, the face decoder can have a Transformer architecture, i.e., can have a Transformer decoder architecture with masking, and the pointer vector can be an output embedding of the last embedding in the decoder input generated by the Transformer decoder.

[0109] In some implementations, each attention layer within the masked Transformer decoder applies only self-attention to the decoder input. In some other implementations, to allow for additional conditioning on the vertices, the masked Transformer decoder also includes one or more attention layers that apply cross-attention to the context embeddings of the vertices.

[0110] From the pointer vector and the respective context embeddings of (i) each of the plurality of vertices, (ii) the new face token, and (iii) the stop token, the system generates probability distributions for (i) each of the plurality of vertices, (ii) the new face token, and (iii) the stop token (step 508).

[0111] In particular, for each context embedding, the system computes a dot product between the context embedding and the pointer vector to generate a logit.

[0112] In some implementations, the system then applies a softmax to the logits to generate the probability distributions.

[0113] In some implementations, the system masks the logits for any possible grid tokens of the output position that are invalid given the grid tokens that have been selected at earlier positions, and then applies a softmax to generate the probability distributions. For example, the system can mask the logits for the stop token until each vertex is included in at least one face. As another example, within a given face, the system can mask the logits for vertices that have already been included as part of the face.

[0114] The system uses the probability distributions to select the grid token at a given position, e.g., by sampling from the probability distributions (step 510).

[0115] As noted above, in some implementations, the system receives a context input, and the generated output sequence is conditioned on the received context input.

[0116] The system can implement this conditioning in any of a variety of ways, but generally, for at least some of the output positions in the output sequence, the system generates the grid token at the output sequence with the grid generation neural network conditioned on the context input.

[0117] Some specific examples of conditioning the grid generation neural network on the context input are as follows.

[0118] As noted above, in some implementations, the context input is a global feature that describes the entire mesh, e.g., an identifier of an object class of the object. In these implementations, the system can generate a learned embedding of the object class and, if necessary, project the embedding to the same dimensionality as the embeddings operated on by the vertex generation neural network and the face generation neural network. The system can then combine the embedding with the output of some or all of the hidden layers of the neural network. As a particular example, when the vertex generation neural network and the face generation neural network have a Transformer architecture, the system can combine the embedding with the intermediate representation after the self-attention layer in each block of the Transformer.

[0119] In some other implementations, the context input is a high-dimensional input, such as an image or a set of voxels. For high-dimensional context inputs, the system jointly trains, i.e., trains in conjunction with, the mesh generation neural network a domain-appropriate encoder of a sequence of context embeddings of the context input.

[0120] The vertex decoder, the face decoder, or both can condition on the sequence of context embeddings. In particular, the vertex decoder, the face decoder, or both can include one or more attention layers that apply cross-attention to the sequence of context embeddings.

[0121] In particular, when the context input is an image of the object, the system processes the image using an image encoder neural network to generate a sequence of image embeddings and conditions the mesh generation neural network on the sequence of image embeddings as described above. As a particular example, the image encoder neural network can be a two-dimensional convolutional neural network composed of a sequence of down-sampling residual blocks.

[0122] Similarly, when the context input is a set of voxels representing the object, the system processes the voxels using a voxel encoder neural network to generate a sequence of voxel embeddings and conditions the mesh generation neural network on the sequence of voxel embeddings as described above. As one particular example, the voxel encoder neural network can be a three-dimensional convolutional neural network composed of a sequence of down-sampling residual blocks.

[0123] This specification uses the term“configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions, the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. One or more computer programs are configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0124] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0125] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0126] A computer program can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, which can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and networks.

[0127] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or structured at all, and it can be stored on one or across multiple locations on storage devices. Thus, for example, an index database can include multiple collections of data, each of which can be organized differently and accessed in different ways.

[0128] Similarly, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components that are installed on one or more computers at one or multiple sites. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0129] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0130] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data files such as magnetic, magneto-optical, or optical disks, or a removable memory card such as a Universal Serial Bus (USB) flash drive, to name a few. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable memory device such as a USB flash drive, to name a few.

[0131] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and DVD-ROM disks.

[0132] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0133] Data processing apparatuses for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing general and compute-intensive portions of machine learning training or production, i.e., inference, workloads.

[0134] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0135] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0136] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at a server from the device.

[0137] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions, but rather as descriptions of particular implementations of certain embodiments of the inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0138] Similarly, while operations are depicted in the drawings in a particular order, and described within this specification as being performed in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order described or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems are generally integrable into a single software product or packaged into multiple software products.

[0139] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: An autoregressive mesh generation neural network is used to generate an output sequence of vertices and faces of a 3D mesh for a specified object. This output sequence contains a corresponding mesh token at each of a plurality of output locations, and the generation includes, at each of the plurality of output locations: An autoregressive grid generation neural network is used to process the input at the output position to generate a probability distribution of possible grid tokens for that output position, conditioned on any grid token that has been selected at least at any earlier output position. Use a probability distribution to select the grid token at the output position from the possible grid tokens at the output position. The probability distribution of possible grid tokens for generating the output position includes the logit of any possible grid token that masks the output position, wherein any possible grid token is invalid given that a grid token has already been selected at an earlier position.

2. The method according to claim 1, further comprising: Generate a 3D mesh of the object from the output sequence.

3. The method according to claim 1, wherein, The grid tokens used to select the output location using probability distributions include: Sample grid tokens from a probability distribution.

4. The method according to claim 1, wherein, The grid token at the first output location set is the corresponding coordinate of each of the multiple vertices of the grid.

5. The method according to claim 4, wherein, The autoregressive mesh generation neural network comprises a vertex generation neural network including a vertex decoder, and wherein the generation includes, for each output position in a first set of output positions: Use the vertex encoder to generate the corresponding embedding for any vertex coordinates that have already been selected at any earlier output position; and The corresponding embeddings are processed using a vertex decoder to generate a probability distribution.

6. The method according to claim 5, wherein, Vertex decoders are attention-based decoder neural networks.

7. The method according to claim 6, wherein, The vertex decoder has a Transformer architecture.

8. The method according to claim 1, wherein, The generation includes generating vertex coordinates until the selected mesh token is a predetermined stop token.

9. The method according to claim 1, wherein, The grid token at the second output location set identifies the vertex of each of the multiple faces that make up the grid.

10. The method according to claim 9, wherein, The mesh-generating neural network includes a surface decoder, and wherein the generation includes: Generate the corresponding contextual embeddings for (i) each of the multiple vertices, (ii) the new face token, and (iii) the stop token; and At each output position in the second set of output positions: Generate decoder inputs that include the corresponding embeddings of each grid token that have already been generated at any earlier output location in the second set; The decoder input is processed using a surface decoder to generate a pointer vector; and Generate the probability distributions of (i) each of the multiple vertices, (ii) the new face token, and (iii) the stop token from the pointer vector and the corresponding context embeddings of (i) each of the multiple vertices, (ii) the new face token, and (iii) the stop token.

11. The method according to claim 10, wherein, The mesh generative neural network includes a face encoder, wherein the corresponding contextual embedding for generating (i) each of a plurality of vertices, (ii) a new face token, and (iii) a stop token includes: The face encoder is used to process (i) each of the multiple vertices, (ii) the new face tokens, and (iii) the stop tokens.

12. The method according to claim 11, wherein, The surface encoder has a Transformer architecture.

13. The method according to any one of claims 10-12, wherein, The surface decoder has a Transformer architecture.

14. The method of claim 10, wherein, The corresponding embedding for each grid token, which has already been generated at any earlier output location in the second set, is generated based on the contextual embedding of the grid token.

15. The method according to claim 14, wherein, The surface decoder applies cross-attention to contextual embedding.

16. The method according to claim 1, further comprising: Receive context input for a specified 3D mesh; and The generation includes, at one or more of a plurality of output locations: An autoregressive grid generation neural network is used to process the input at a time step to generate a probability distribution of possible grid tokens for the output position, the autoregressive grid generation neural network being conditioned on (i) at least any grid token that has been selected at any earlier output position and (ii) the contextual input.

17. The method according to claim 16, wherein, The contextual input is an input identifying the object class, and wherein the autoregressive grid generative neural network, conditioned on the contextual input, includes generating an embedding of the object class, and the grid generative neural network is conditioned on the embedding.

18. The method according to claim 16, wherein, The contextual input is an image of an object, and wherein the autoregressive grid generative neural network conditioned on the contextual input includes: processing the image using an image encoder neural network to generate a sequence of image embeddings, and the grid generative neural network conditioned on the sequence of image embeddings.

19. The method of claim 16, wherein, The contextual input is a set of voxels representing an object, and wherein the autoregressive grid-generating neural network conditioned on the contextual input includes: using a voxel encoder neural network to process the set of voxels to generate a sequence of voxel embeddings, and the grid-generating neural network conditioned on the sequence of voxel embeddings.

20. The method according to any one of claims 18 or 19, wherein, The surface decoder applies cross-attention to either the sequence of voxel embeddings or the sequence of image embeddings.

21. The method according to any one of claim 18 or 19, wherein, The vertex decoder applies cross-attention to either the sequence of voxel embeddings or the sequence of image embeddings.

22. A computer-readable storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 21.

23. A system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform operations according to any one of claims 1 to 21.

Images