Simulation of the physical environment using fine and coarse resolution meshes

Abstract

A method, system, and apparatus comprising a computer program encoded on a computer storage medium for simulating the state of a physical environment. In one aspect, a method is provided that is executed by one or more computers for simulating the state of a physical environment. The method includes, for each of a plurality of time steps, obtaining data that defines a fine-resolution mesh and a coarse-resolution mesh, each characterizing the state of the physical environment at the current time step, wherein the fine-resolution mesh has a higher resolution than the coarse-resolution mesh; processing the data that defines the fine-resolution mesh and the coarse-resolution mesh using a graph neural network that includes (i) one or more fine-resolution update blocks, (ii) one or more coarse-resolution update blocks, and (iii) one or more upsampling update blocks; and determining the state of the physical environment at the next time step using the updated node embeddings for the nodes in the fine-resolution mesh.

Description

Technical Field

[0001] This specification relates to processing data using a machine learning model.

Background Art

[0002] A machine learning model receives an input and generates an output, for example, a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values of the model's parameters.

[0003] Some machine learning models are deep models that employ multiple layers of models to generate an output for the received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to the received input to generate an output.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] This specification generally describes a simulation system implemented as a computer program on one or more computers at one or more locations that can use a graph neural network to simulate the state of a physical environment over a series of time steps. In particular, this specification introduces a simulation system that can accurately predict (simulate) a wide range of physical environments in a high-resolution setting using a graph neural network.

[0006] Some implementations of the described techniques are adapted to specific computing hardware. For example, techniques are described that enable a mesh-based simulation to be partitioned into updates in a fine-resolution mesh and a coarse-resolution mesh that are used by the simulation system to simulate the state of a physical environment. This, in turn, enables the simulation system to utilize a computer system that includes a more powerful processor and a less powerful processor, for example, in terms of computing power such as FLOPS (floating point operations per second) or available working memory, to optimally allocate the computational resources for updates in the fine-resolution and coarse-resolution meshes.

Means for Solving the Problems

[0007] In one aspect, a method executed by one or more computers for simulating the state of a physical environment is provided. The method includes, for each of a plurality of time steps, obtaining data that defines a fine-resolution mesh and a coarse-resolution mesh, each characterizing the state of the physical environment at the current time step, where the fine-resolution mesh has a higher resolution than the coarse-resolution mesh; processing the data that defines the fine-resolution mesh and the coarse-resolution mesh using a graph neural network; and determining the state of the physical environment at the next time step using the updated node embeddings for the nodes in the fine-resolution mesh. The graph neural network includes (i) one or more fine-resolution update blocks, (ii) one or more coarse-resolution update blocks, and (iii) one or more upsampling update blocks. Each fine-resolution update block is configured to process the data that defines the fine-resolution mesh using a graph neural network layer to update the current node embedding of each node in the fine-resolution mesh. Each coarse-resolution update block is configured to process the data that defines the coarse-resolution mesh using a graph neural network layer to update the current node embedding of each node in the coarse-resolution mesh. Each upsampling update block is configured to generate data that defines an upsampling mesh comprising (i) each node from the fine-resolution mesh and each node from the coarse-resolution mesh, and (ii) a plurality of edges between the nodes of the fine-resolution mesh and the nodes of the coarse-resolution mesh, and to process the data that defines the upsampling mesh using a graph neural network layer to update the current node embedding of each node in the fine-resolution mesh.

[0008] In some implementations, the step of generating an upsampling mesh includes, for each node of the coarse-resolution mesh, identifying cells of the fine-resolution mesh that include the nodes of the coarse-resolution mesh, identifying one or more nodes in the fine-resolution mesh that are vertices of the cells that include the nodes of the coarse-resolution mesh, and instantiating respective edges in the upsampling mesh between each node of the coarse-resolution mesh and each of the identified nodes in the fine-resolution mesh.

[0009] In some implementations, the method further includes, for each edge in the upsampling mesh, generating an edge embedding for the edge based on the distance between the pair of nodes in the upsampling mesh connected by the edge.

[0010] In some implementations, to update the current node embedding of each node in the fine-resolution mesh, the step of processing data defining the upsampling mesh using a graph neural network layer includes updating the edge embedding for each edge in the upsampling mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of the first node in the coarse-resolution mesh and the second node in the fine-resolution mesh connected by the edge, and updating the node embedding for each node in the fine-resolution mesh based on (i) the node embedding for the node in the fine-resolution mesh and (ii) the respective edge embeddings of each edge connecting the node in the fine-resolution mesh to the corresponding node in the coarse-resolution mesh.

[0011] In some implementations, each upsampling block updates the current node embedding of the nodes in the fine-resolution mesh based at least in part on the current node embedding of the nodes in the coarse-resolution mesh.

[0012] In some implementations, the graph neural network further includes one or more downsampling update blocks. Each downsampling update block is configured to: (i) generate data defining a downsampling mesh that includes each node from a fine-resolution mesh and each node from a coarse-resolution mesh, and a plurality of edges between nodes of the fine-resolution mesh and nodes of the coarse-resolution mesh; and (ii) process the data defining the downsampling mesh using a graph neural network layer to update the current node embedding of each node in the coarse-resolution mesh.

[0013] In some implementations, generating the downsampling mesh includes: for each node of the fine-resolution mesh, identifying a cell of the coarse-resolution mesh that includes the node of the fine-resolution mesh; identifying one or more nodes of the coarse-resolution mesh that are vertices of the cell that includes the node of the fine-resolution mesh; and instantiating each edge in the downsampling mesh between the node of the fine-resolution mesh and each of the identified nodes of the coarse-resolution mesh.

[0014] In some implementations, the method further includes, for each edge in the downsampling mesh, generating an edge embedding for the edge based on the distance between the pair of nodes in the downsampling mesh connected by the edge.

[0015] In some implementations, to update the current node embedding of each node in a coarse-resolution mesh, the step of using a graph neural network layer to process data defining a downsampling mesh includes updating the edge embedding for each edge in the downsampling mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of a first node in the coarse-resolution mesh and a second node in the fine-resolution mesh connected by the edge, and updating the node embedding for each node in the coarse-resolution mesh based on (i) the node embedding for the node in the coarse-resolution mesh and (ii) the respective edge embeddings of each edge connecting the node in the coarse-resolution mesh to a corresponding node in the fine-resolution mesh.

[0016] In some implementations, each downsampling block updates the current node embedding of a node in a coarse-resolution mesh based at least in part on the current node embedding of a node in a fine-resolution mesh.

[0017] In some implementations, the graph neural network is trained on a set of training examples, one or more of which are generated by an operation that includes generating a target simulation of the state of a training physical environment over one or more time steps using a simulation engine, where the target simulation has a higher resolution than the fine-resolution mesh processed by the graph neural network, interpolating the target simulation to the same resolution as the fine-resolution mesh processed by the graph neural network to generate a lower-resolution version of the target simulation, and generating training examples using the lower-resolution version of the simulation mesh.

[0018] In some implementations, the step of obtaining data that defines the state of the physical environment at the current time step includes obtaining, for each node in a mesh with a fine resolution, one or more node features for the node, where the node corresponds to a position in the physical environment and the node features characterize the state of the corresponding position in the physical environment, and processing the node features using one or more neural network layers of a graph neural network to generate a current embedding for the node.

[0019] In some implementations, for each node in a mesh with a fine resolution, the node features for the node comprise one or more of a fluid density feature, a fluid viscosity feature, a pressure feature, or a tension feature.

[0020] In some implementations, the graph neural network further includes a decoder block, and determining the state of the physical environment at the next time step includes processing the updated node embeddings for each node in a mesh with a fine resolution to generate one or more respective dynamics features corresponding to each node in the mesh with a fine resolution, and determining the state of the physical environment at the next time step based on (i) the dynamics features for the nodes in the mesh with a fine resolution and (ii) the node features for the nodes in the mesh with a fine resolution at the current time step.

[0021] In some implementations, the mesh with a fine resolution and the mesh with a coarse resolution are each a three-dimensional mesh.

[0022] In some implementations, the mesh with a fine resolution and the mesh with a coarse resolution are each a triangular mesh.

[0023] In some implementations, the mesh with a fine resolution and the mesh with a coarse resolution each extend over the physical environment.

[0024] In some implementations, for each time step, the number of nodes in the fine-resolution mesh is greater than the number of nodes in the coarse-resolution mesh.

[0025] In some implementations, the method is executed on a computing system that includes a first processor and a second processor, where the second processor has a greater processing capacity or memory than the first processor. The method includes processing data that defines a fine-resolution mesh by performing one or more fine-resolution update blocks on the second processor, and processing data that defines a coarse-resolution mesh by performing one or more coarse-resolution update blocks on the first processor.

[0026] In some implementations, the method further includes processing data that defines a fine-resolution mesh by performing one or more fine-resolution update blocks on the second processor, then processing data that defines a downsampling mesh to update the current node embedding of each node in the coarse-resolution mesh, then processing data that defines a coarse-resolution mesh by performing one or more coarse-resolution update blocks on the first processor, and then processing data that defines an upsampling mesh to update the current node embedding of each node in the fine-resolution mesh.

[0027] In a second aspect, a method of controlling a robot using any of the methods described above is provided. The physical environment includes the real-world environment containing physical objects. The step of obtaining data defining a fine-resolution mesh and a coarse-resolution mesh each characterizing the state of the physical environment at the current time step includes determining a representation of the location, shape, or configuration of the physical objects at the current time step. The step of determining the state of the physical environment at the next time step includes determining a predicted representation of the location, shape, or configuration of the physical objects at the next time step. The method further includes, at each time step, controlling the robot using the predicted representation at the next time step to manipulate the physical objects.

[0028] In a third aspect, a system is provided. The system includes one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform any of the operations of the methods described above.

[0029] In a fourth aspect, a system is provided. The system includes one or more computers and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform any of the operations of the methods described above.

[0030] Graph neural networks use message passing between nodes to propagate information by exchanging information with adjacent nodes and iteratively update those node embeddings. However, since equally spaced points in space become more scattered in graph space, this structure is a limiting factor for high-resolution simulations. To address this, the simulation system can train the graph neural network to learn accurate dynamics as a proxy for a high-resolution physical environment on a mesh with lower resolution, removing the message passing bottleneck and improving performance. Additionally, the simulation system also introduces a hierarchical approach by passing messages on two meshes with different resolutions, i.e., a fine-resolution mesh and a coarse-resolution mesh, which significantly improves the accuracy of the graph neural network while requiring fewer computational resources.

[0031] The physical environment can be, for example, a continuous field or a deformable material. A continuous field can refer to a spatial region where each position in the spatial region is associated with one or more physical quantities, such as velocity, pressure, etc.

[0032] A "mesh" refers to a data structure that includes a set of nodes and a set of edges, where each edge connects a respective pair of nodes. The mesh can define an irregular (unstructured) grid that specifies a checkerboard-like arrangement of a geometric domain (such as a surface or space) into smaller elements (such as cells or zones) with a particular shape (e.g., triangular or tetrahedral shape). Each node can be associated with a respective spatial location in the physical environment.

[0033] The "resolution" of a mesh can refer to, for example, the number of nodes in the mesh and / or the node density in the mesh. The node density of a mesh can refer to the number of nodes per unit length when the mesh is one-dimensional, the number of nodes per unit area when the mesh is two-dimensional, the number of nodes per unit volume when the mesh is three-dimensional, and so on.

[0034] At each time step, the simulation system generates an initial node embedding for each node of the fine-resolution mesh and the coarse-resolution mesh, and then repeatedly updates the node embeddings of the nodes of the fine-resolution mesh and the coarse-resolution mesh using the update block of the graph neural network. Specifically, each update block of the graph neural network receives the fine-resolution mesh and / or the coarse-resolution mesh, updates the current node embedding for the nodes of the fine-resolution mesh or the coarse-resolution mesh, and then provides the fine-resolution mesh and / or the coarse-resolution mesh to the next update block in the graph neural network.

[0035] Throughout this specification, the "embedding" of an entity can refer to a numerically ordered set in a latent space (e.g., a space with fewer dimensions), e.g., a representation of the entity as a vector or matrix of numbers. The embedding of an entity can be generated, for example, as the output of a neural network that processes data characterizing that entity. Note that the embedding of an entity is often referred to as the latent representation of the entity, the encoded representation of the entity, or the feature vector representation of the entity, depending on the context.

[0036] The simulations generated by the simulation system described herein (e.g., characterizing the predicted state of a physical environment over a series of time steps) can be used for any of a variety of purposes. In some cases, a visual representation of the simulation may be generated, for example, as a video and provided to a user of the simulation system. In some cases, the representation of the simulation may be processed to determine that feasibility criteria are met, and a physical device or system may be constructed in response to the feasibility criteria being met. For example, the simulation system may generate an aerodynamic simulation of the airflow above an aircraft wing, and the feasibility criteria for physically constructing the aircraft wing may be that the forces or stresses on the aircraft wing do not exceed a threshold. In some cases, an agent that interacts with the physical environment (e.g., a reinforcement learning agent or a robotic agent) may use the simulation system to generate one or more simulations of the environment that simulate the impact of the agent performing various actions in the environment. In these instances, the agent may use the simulation of the physical environment as part of determining how many actions to perform in the environment.

[0037] The subject matter described herein may be implemented in certain embodiments so as to realize one or more of the following advantages.

[0038] Simulators of complex physics realities are very valuable for many scientific and engineering exercises. However, conventional simulators can be prohibitively expensive to create and use. Constructing a conventional simulator can require years of engineering effort and often involves a trade-off between generality and accuracy within a narrow range of settings. Additionally, high-quality simulators often require significant computational resources, making it difficult or impossible to scale up. The simulation system described herein can generate simulations of complex physical environments over multiple time steps with higher accuracy and using fewer computational resources (e.g., memory and computing power) than some conventional simulators. In some situations, the simulation system can generate simulations that are one or more orders of magnitude faster than conventional simulators. For example, the simulation system can predict the state of a physical environment at the next time step through a single pass through a graph neural network, whereas conventional simulators may be required to perform separate optimizations at each time step.

[0039] The simulation system generates simulations using a graph neural network that can learn to directly simulate complex physics from training data and can generalize implicitly learned physical principles to accurately simulate a wider range of physical environments under various conditions than are directly represented in the training data. This also enables the system to generalize to larger and more complex settings than those used during training. In contrast, some conventional simulators require physical principles to be explicitly programmed and must be manually adapted to the specific characteristics of each environment during simulation.

[0040] A simulation system can perform a mesh-based simulation, for example, where the state of the physical environment at each time step is represented by a mesh. Performing a mesh-based simulation can enable the simulation system to more accurately simulate a physical environment that includes deforming surfaces or volumes, which can be difficult for some physical environments, such as models of a collection of discrete particles, than would otherwise be possible.

[0041] However, generating an accurate mesh-based simulation of the state of a physical environment may require increasing the resolution of the mesh. As the mesh resolution increases, nodes in the mesh that are separated by the same distance in the reference frame of the physical environment are separated by a greater distance in the reference frame of the mesh. (The distance between two nodes in the reference frame of the mesh, for example, characterizes the minimum number of edges separating the two nodes in the mesh.) Thus, when a simulation is performed by processing the mesh using a graph neural network, increasing the mesh resolution requires the graph neural network to include more graph neural network layers to propagate information over the same physical distance. However, increasing the number of graph neural network layers increases the computational resource consumption by the graph neural network and increases the likelihood that the graph neural network will oversmooth the node embeddings associated with the nodes in the mesh, which can result in lower simulation accuracy.

[0042] The simulation system described in this specification addresses this problem by simulating the state of a physical environment using a fine-resolution mesh and a coarse-resolution mesh, where the fine-resolution mesh has a higher resolution than the coarse-resolution mesh. The higher resolution of the fine-resolution mesh enables a very accurate simulation of local effects in the physical environment. The lower resolution of the coarse-resolution mesh enables information sharing between distant nodes in the coarse-resolution mesh, for example when the coarse-resolution mesh is processed using a graph neural network layer. The simulation system exploits the complementary advantages of the fine-resolution mesh and the coarse-resolution mesh by enabling information sharing along edges that connect nodes in the fine-resolution mesh to nodes in the coarse-resolution mesh. Thus, by simulating the state of the physical environment using both a fine-resolution mesh and a coarse-resolution mesh, the simulation system can significantly improve the simulation accuracy while reducing the use of computational resources.

[0043] A simulation system can train a graph neural network used to perform mesh-based simulations on a set of training data. To generate the training data, the simulation system can use a simulation engine (e.g., a physics engine) to simulate the state of the physical environment at a resolution higher than the fine-resolution mesh processed by the graph neural network. The simulation system can then interpolate the simulation to a lower resolution of the fine-resolution mesh processed by the graph neural network to generate a lower-resolution version of the simulation and generate training data based on the lower-resolution version of the simulation. Generating training data in this way can increase the accuracy of the training data, thereby enabling the graph neural network trained on the training data to achieve higher simulation accuracy.

[0044] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0045]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6A

Figure 6B

[0046] Like reference numerals and designations in the various drawings indicate like elements.

[0047] Replacing a costly conventional numerical solver with a learning-based simulator can be advantageous, for example, because the learning-based simulator may be much faster than classical methods. Further, the learning-based simulator may be discriminable by a configuration that expands an interesting means for inverse design. A recent approach to learning simulations discretized on an unstructured mesh is MeshGraphNets (see, e.g., Pfaff, T. et al., "Learning mesh-based simulation with graph networks", 9th International Conference on Learning Representations, 2021), which encodes the simulation mesh at each time step into a graph and makes predictions on this graph using a message-passing graph neural network (see, e.g., Scarselli, F. et al., "The graph neural network model", IEEE Transactions on Neural Networks, 20(1):61-80, 2008). MeshGraphNets show strong generalization and accurate prediction in a wide range of physical systems.

[0048] The accuracy of conventional solvers is often limited by the resolution of the simulation mesh. This is particularly true for chaotic systems such as hydrodynamics, i.e., processes at extremely short length scales such as turbulent mixing, which affect the overall flow and generally need to be solved on a very fine mesh to accurately solve the underlying partial differential equation (PDE). This leads to a characteristic spatial convergence where simulation accuracy increases monotonically with mesh resolution. This is an important property for the actual use of numerical solvers as it allows trading off computations to obtain the desired solution accuracy.

[0049] However, for example, as the mesh becomes finer, the message passing graph neural network executes more update steps to propagate information along the same physical distance, so a similar phenomenon may not apply to learning simulation techniques, particularly graph neural network models such as MeshGraphNets. This results in a significantly large computational cost, reduced accuracy at high resolution, and may also cause oversmoothing.

[0050] To address these issues, this specification introduces a simulation system that implements a hierarchical framework for learning mesh-based simulations using a graph neural network that performs message passing at two different resolutions. That is, the simulation system performs message passing on a fine-resolution mesh and a coarse-resolution mesh that facilitate the propagation of information. In addition to being more accurate and computationally efficient than conventional methods (see, for example, FIG. 6B), the simulation system restores spatial convergence for the graph neural network model (see, for example, FIG. 6A). Moreover, the simulation system modifies the training distribution for using highly accurate predictions that better capture the dynamics of the physical environment being simulated (see, for example, FIG. 4). Instead of replicating the spatial convergence curve of a conventional solver, this enables the simulation system to make better predictions than a reference simulation engine (e.g., a physics engine) at a given resolution. Together, these techniques improve accuracy for high-resolution simulations with lower computational cost.

[0051] These features and other features are described in more detail below.

[0052] FIG. 1A shows an exemplary simulation system 100 that can simulate the state of a physical environment using a graph neural network 150. The simulation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described below are implemented.

[0053] The “physical environment” can refer to any type of physical system, including, for example, fluids, rigid solids, deformable materials, any other type of physical system, or combinations thereof. A “simulation” of the physical environment can include the simulated respective states of the physical environment at each time step in a series of time steps. The state of the physical environment at a particular time step can be represented as a mesh (or multiple meshes with different resolutions), as seen in FIG. 1B and described in more detail below. The state of the physical environment at the initial time step can be provided as an input to the simulation system 100, for example, by a user of the simulation system 100, through a user interface or application programming interface (API) made available by the simulation system 100. At each time step in the series of time steps, the simulation system 100 can process data defining the current state 102 of the physical environment and generate a prediction of the state 202 of the physical environment at the next time step.

[0054] Some physical environments, such as those containing fluids, can be effectively simulated as a set of individual particles, while other physical environments, such as those containing deformable materials and complex structures, can be more challenging to simulate in the same way. Specifically, simulating such physical environments through a particle representation can be computationally inefficient and error-prone, for example, causing inaccurate predictions. Instead, such physical environments can be more appropriately represented by a mesh that can span the entirety of the physical environment, a particular region of the physical environment, or each surface of one or more objects within the physical environment.

[0055] The simulation system 100 can be used to simulate the dynamics of various physical environments through a mesh-based representation. The exemplary physical environments described below are provided for illustration purposes only, and it should be understood that the simulation system 100 can be used to simulate the state of any type of physical environment, including any type of material or physical object.

[0056] At each time step, to simulate such a physical environment, simulation system 100 processes data that defines the current state 102 of the physical environment, where such data specifies respective current node features 104.f and 104.c for nodes in a fine-resolution and a coarse-resolution mesh, processes the data to encode it into respective current node embeddings 114.f and 114.c for the nodes, continuously updates the respective current node embeddings 114.f and 114.c to generate a final updated node embedding 134.f for nodes in the fine-resolution mesh, decodes the final updated node embedding 134.f to generate a dynamics feature 144.f for nodes in the fine-resolution mesh, and predicts a next state 202 of the physical environment based on the current node features 104.f (and, optionally, one or more previous node features) and the dynamics feature 144.f for nodes in the fine-resolution mesh. Various aspects of this process are described in more detail below.

[0057] For example, a physical environment that includes continuous fields, deformable materials, and / or complex structures can be represented by a mesh G=(V,E), for example, an undirected graph. The mesh represents the spatial domain of the physical environment

[0058]

Number

[0059] is defined over, where n is the dimension of the physical environment. The physical environment can be a one-dimensional physical environment (e.g., a spring, a linear polymer), a two-dimensional physical environment (e.g., a superfluid, a membrane), a three-dimensional physical environment (e.g., an aircraft wing, a trapped ion), or in some cases, a physical environment with more dimensions than three (e.g., 10-dimensional supergravity). A "continuous field" generally refers to a spatial region associated with a physical quantity (such as velocity, pressure, temperature, electromagnetic field, probability amplitude, etc.) that varies continuously over a region. For example, each spatial location in a velocity field can have a specific value of the associated velocity, e.g., direction and magnitude. As another example, each spatial location in an electromagnetic field can have specific values of the associated electric and magnetic fields, e.g., their respective directions and magnitudes. A continuous field can be a field of real numbers, imaginary numbers, or complex numbers, depending on the problem. For example, each spatial location in the probability amplitude of an electron can have an associated complex number value.

[0060] Generally, a "mesh" refers to a data structure that includes a set of nodes V and a set of edges E, where the edges connect pairs of nodes. A mesh can define an irregular (unstructured) grid that specifies a checkerboard-like arrangement of a geometric domain (such as a surface or space) into smaller elements (such as cells or zones) having a particular shape, e.g., triangular or tetrahedral shape. Each node can be associated with a respective spatial location in the physical environment. In some implementations, a mesh can represent each surface of one or more objects in the environment. In some implementations, for example, when the physical environment represents a continuous field, the mesh can span (e.g., cover) the physical environment.

[0061] For ease of explanation, the physical environment is assumed to evolve according to Eulerian dynamics over a fixed mesh, and thus the simulation system 100 need not consider world edges within the mesh. However, the simulation system 100 can also be adapted to physical environments that evolve according to Lagrangian dynamics, where, for example, the mesh represents moving and deforming surfaces or volumes. In these cases, the set of world edges E W enables the modeling of external dynamics, such as (self-) collisions and contacts, for a mesh G = (V, E, E W) may be included therein. For example, in an implementation where the mesh represents one or more objects in a physical environment, the simulation system 100 can identify each pair of nodes in the mesh having respective spatial positions that are separated by a distance shorter than a threshold distance in the world space W (e.g., in the reference frame of the physical environment), and can instantiate a world edge between each corresponding pair of nodes in the mesh. Specifically, the simulation system 100 can instantiate a world edge between pairs of nodes that are not already connected by an edge. Representing the current state 102 of the physical environment through both edges and world edges enables the simulation system 100 to simulate the interaction between pairs of nodes that are substantially removed from each other in the mesh space (e.g., separated by a plurality of other nodes and edges), but are substantially close to each other in the world space (e.g., having proximate spatial locations in the reference frame of the physical environment). Including world edges in the mesh can facilitate more efficient message passing between spatially proximate nodes. Thus, world edges can enable a more accurate simulation that uses fewer update iterations (i.e., message passing steps) in the update module 120, thereby reducing the consumption of computing resources during the simulation.

[0062] Each node i ∈ V in the mesh has a position x in the physical environment corresponding to the node i characterizing the current state 102 of the physical environment at the current time step t k by the current node feature f i (t k ). For example, in an implementation involving the simulation of a physical environment having a continuous field such as in a fluid dynamics or aerodynamics simulation, the node feature f of each node ican include fluid viscosity, fluid density, or any other suitable physical aspect at a location within the physical environment corresponding to the node. As another example, in an implementation involving the simulation of a physical environment having an object, such as a structural mechanics simulation, each node can represent a point on the object, and the node-specific feature f of the object characterizing the point on the object i , for example, can be related to the position of each point on the object, the pressure at that point, the tension at that point, and any other suitable physical aspect. Further, each node can additionally include, as part of the node-specific feature f, one or more of fluid density, fluid viscosity, pressure, or tension at a location within the physical environment corresponding to the node i to which it can be related. Generally, the mesh representation is not limited to the physical environments described above, and other types of physical environments can also be represented through the mesh and simulated using the simulation system 100

[0063] In some implementations, the node-specific feature associated with each node at the current time step can further include the respective state of the node at each of one or more previous time steps t k-1 , t k-2 ,..., t k-C . For example, the node-specific feature associated with each node at the current time step can be the respective node-specific feature f i (t k-1 ), f i (t k-2 ),..., f i (t k-C) can include. Such an implementation may be suitable in a physical environment (e.g., temporal variability) with memory effort where the current state 102 of the physical environment depends, for example, on the convolution with a previous state of the physical environment through a response function (e.g., a convolution kernel). For example, the polarization density of an electromagnetic medium at the current time step generally depends on the electric fields at a plurality of previous time steps through a dispersive permittivity. The state of a node at one or more previous time steps can also capture hidden states and / or non-inverting changes, such as plastic deformation, hysteresis. In the case of computational fluid dynamics (CFD) and other related systems (e.g., continuum mechanics systems), a longer history of the state of the physical environment enables the graph neural network 150 to learn correction terms (similar to higher-order integrators), for example, to enable more accurate prediction and / or longer time steps for simulating the state of the physical environment over a longer period of time using fewer time steps.

[0064] The simulation system 100 has two mesh-based representations of the physical environment over its spatial domain

[0065] [Number]

[0066] , namely, (i) a fine-resolution mesh G f =(V f , E f )(where V f and E f are, respectively, the set of nodes 11.f and the set of edges 13.f of the fine-resolution mesh 10.f) and, (ii) a coarse-resolution mesh G c =(V c , E c )(where V c and E coperates in, respectively, a set of nodes 11.c and a set of edges 13.c of a coarse-resolution mesh 10.c. The fine-resolution mesh 10.f has a higher resolution than the coarse-resolution mesh 10.c and, for example, has a greater number of nodes and / or a greater node density. Generally, the coarse-resolution mesh 10.c is introduced by the simulation system 100 for the purpose of facilitating more efficient message passing of the graph neural network 150, for example, to efficiently model fast-acting or non-local dynamics.

[0067] The simulation system 100 can generate meshes of fine resolution 10.f and coarse resolution 10.c using a mesh generation algorithm, for example, in particular, Delaunay triangulation, Rupert's algorithm, algebraic methods, differential equation methods, variational methods, unstructured grid methods. Alternatively, the simulation system 100 can first generate a fine-resolution mesh 10.f using a mesh generation algorithm and then average or interpolate the fine-resolution mesh 10.f to generate a coarse-resolution mesh 10.c.

[0068] FIG. 1B is a diagram of exemplary fine-resolution 10.f and coarse-resolution 10.c meshes characterizing the current state 102 of the physical environment. The fine-resolution 10.f and coarse-resolution 10.c meshes are shown in FIG. 1B as two-dimensional meshes with triangular cells (e.g., Delaunay triangulation), but it should be noted that the fine-resolution 10.f and coarse-resolution 10.c meshes can generally be of any dimension and can have cells of any shape.

[0069] Each node i ∈ V in the fine-resolution mesh 10.f f corresponds to a position x in the physical environment corresponding to the node 11.f i of the current state 102 of the physical environment at the current time step t kcharacterizes the current node feature f i f (t k ) related to 104.f. Pairs of nodes 11.f in the fine-resolution mesh 10.f are connected by edges 13.f that form cells 14.f. For illustration, internal nodes are identified as black circles and boundary nodes are identified as white circles with black outlines.

[0070] Similarly, each node i ∈ V in the coarse-resolution mesh c is the position x in the physical environment corresponding to node 11.c i characterizes the current state 102 of the physical environment at the current time step t k by the current node feature f i c (t k ) related to 104.c. Pairs of nodes 11.c in the coarse-resolution mesh 10.c are connected by edges 13.c that form cells 14.c. For illustration, internal nodes are identified as black circles and boundary nodes are identified as white circles with black outlines.

[0071] The respective nodes in the fine-resolution mesh 10.f and the coarse-resolution mesh 10.c need not coincide, and thus, the current state 102 of the physical environment at different positions in the physical environment can be characterized. Moreover, since the fine-resolution mesh 10.f has a higher resolution than the coarse-resolution mesh 10.c, the simulation system 100 can determine the current node feature 104.c for each node in the coarse-resolution mesh 10.c from the current node feature 104.f of the nodes in the fine-resolution mesh 10.f. For example, the simulation system 100 can average or interpolate the current node feature 104.f associated with a group of nodes in the fine-resolution mesh 10.f to generate the current node feature 104.c associated with the nodes in the coarse-resolution mesh 10.c. In some implementations, the current node feature 104.c of the coarse-resolution mesh 10.c includes only geometric (e.g., static) features that do not change with each time step. For example, the geometric features can include, for example, a node type that distinguishes between internal nodes and boundary nodes as a one-hot vector. As some further examples, the node type can indicate whether the node is part of a physical object, the boundary of an object, part of an actuator, part of a fluid containing the object, a wall, an inflow or outflow of the physical environment, a point of attachment of an object, or another feature of the physical environment.

[0072] In some implementations, the current node features 104.f and 104.c of the fine-resolution mesh 10.f and the coarse-resolution mesh 10.c can also include global features 108 of the physical environment, such as the representation of the forces applied to the physical environment, the gravitational constant of the physical environment, the magnetic field of the physical environment, or any other suitable feature, or a combination thereof. For example, at each time step, the simulation system 100 can concatenate the global feature 108 on top of the current node features 104.f and 104.c associated with each node in the fine-resolution mesh 10.f and each node in the coarse-resolution mesh 10.c before the graph neural network 150 processes the current state 102 of the physical environment.

[0073] The graph neural network 150 includes an encoder module 110, an updater module 120, and a decoder module 130.

[0074] The encoder 110 includes one or more neural network layers. Specifically, the encoder 110 can consist of any suitable number (e.g., 5 layers, 25 layers, or 100 layers) and be connected in any suitable configuration (e.g., as a linear series of layers or as a directed graph of layers) of any suitable type of neural network layer (e.g., fully connected layers, convolutional layers, attention layers, etc.). By way of example only, the encoder 110 can be implemented as a multi-layer perceptron (MLP) with residual connections.

[0075] At each time step, the encoder 110 processes the current node feature 104.f of each node i ∈ V in the fine-resolution mesh f to obtain the current node embedding for the node at that time step

[0076]

Number

[0077] Generate 114.f. Similarly, at each time step, the encoder 110 processes the current node feature 104.c of each node i ∈ V in the coarse-resolution mesh c to generate the current node embedding for the node at that time step

[0078]

Number

[0079] Generate 114.c. Generally, the node embedding for a node represents the individual characteristics of the node in the latent space.

[0080] At each time step, the encoder 110 also generates the current edge embedding for each edge in the fine-resolution mesh 10.f

[0081]

Number

[0082] and the current edge embedding for each edge in the coarse-resolution mesh 10.c at that time step

[0083]

Number

[0084] can be generated. Generally, edge embedding for an edge connecting a pair of nodes in a mesh represents the pairwise property of the corresponding pair of nodes in the latent space. For example, for each edge in a fine-resolution mesh 10.f or a coarse-resolution mesh 10.c, the encoder 110 can process the respective current node features and / or the respective positions of the pair of nodes i, j ∈ V connected by the edge, and can generate the respective current edge embedding for the edge. More specifically, the encoder 110 can generate the current edge embedding for each edge in the fine-resolution 10.f or coarse-resolution 10.c mesh based on the respective current node features of the nodes connected by the edge, the difference between the respective current node features of the nodes connected by the edge, the weighted sum of the differences between the respective current node features of the nodes connected by the edge, the respective positions of the nodes connected by the edge, the difference between the respective positions of the nodes connected by the edge, the magnitude of the difference between the respective positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof.

[0085] The update data 120 includes a series 122 of update blocks including (i) one or more fine-resolution update blocks 122.f, (ii) one or more coarse-resolution update blocks 122.c, (iii) one or more upsampling update blocks 122.u, and (iv) one or more downsampling update blocks 122.d.

[0086] At each time step, the update data 120 processes the current node embeddings 114.f and 114.c using the series 122 of update blocks to obtain the final updated node embeddings for the nodes in the fine-resolution mesh 10.f at that time step

[0087]

Number

[0088] Generate 134.f. Generally, the update data 120 updates the current node embeddings 114.f and 144.c multiple times in a time step to generate the final updated node embedding 134.f. The operation of each update block 122 will be described below with respect to FIGS. 2A-2D. The update block 122 can be arranged in various different topologies with various numbers of blocks, for example, to target the prediction accuracy of several levels for several resolutions in a mesh of fine resolution 10.f. Exemplary topologies are described below with respect to FIGS. 3A and 3B.

[0089] The decoder 130 includes one or more neural network layers. Specifically, the decoder 130 can include any suitable type of neural network layer (e.g., fully connected layer, convolutional layer, attention layer, etc.) having any suitable number (e.g., 5 layers, 25 layers, or 100 layers) and connected in any suitable configuration (e.g., as a linear series of layers or as a directed graph of layers). By way of example only, the decoder 130 can be implemented as a multi-layer perceptron (MLP) with residual connections.

[0090] At each time step, the decoder 130 processes the final updated node embedding 134.f associated with each node in the fine-resolution mesh 10.f to obtain one or more dynamics features for the nodes at that time step

[0091]

Number

[0092] Generate 144.f. The dynamics feature 144.f characterizes the rate of change of the current node feature 104.f associated with the node. The dynamics feature 144.f can represent the rate of change of any suitable current node feature 104.f for a node in the fine-resolution mesh 10.f, such as position, velocity, momentum, density, electromagnetic field, probability field, or any other suitable physical aspect.

[0093] At each time step, the prediction engine 160 can determine the node features for each node in the fine-resolution mesh 10.f at the next time step based on (i) the current node feature 104.f of the node at the current time step and (ii) the dynamics feature 144.f of the node, for example, by integrating the dynamics feature 144.f any suitable number of times. For example, in the case of a first-order system and assuming equally spaced time steps t k+1 -t k =Δt, the prediction engine 160 can determine the node features for the node at the next time step based on the current node feature 104.f at the current time step and the dynamics feature 144.f corresponding to the node,

[0094]

Equation

[0095] which can be determined as, because it is derived from the first-order finite difference method

[0096]

Equation

[0097] is derived. The prediction engine 160 can at least partially control the accuracy of such predictions by choosing appropriately spaced time steps Δt.

[0098] Similarly, in the case of the second-order system, the prediction engine 160 determines the node features for the nodes at the next time step based on the current node features 104.f at the current time step, the node features at the previous time step, and the dynamics features 144.f corresponding to the nodes,

[0099] [Number]

[0100] which can be determined as, because it is derived from the second-order central difference method

[0101] [Number]

[0102] is derived. Also in this case, the prediction engine 160 can at least partially control the accuracy of such a prediction by choosing appropriately spaced time steps Δt.

[0103] Therefore, by determining the node features for all the nodes in the fine-resolution mesh 10.f at the next time step, the simulation system 100 can determine the next state 202 of the physical environment. As described above, the simulation system 100 can determine the node features for all the nodes in the coarse-resolution mesh 10.c at the next time step by averaging or interpolating the node features related to the nodes in the fine-resolution mesh 10.f at the next time step. In an implementation where the node features of the coarse-resolution mesh 10.c contain only geometric features, the simulation system 100 does not need to update the node features of the coarse-resolution mesh 10.c because such features are static over time steps.

[0104] The simulation system 100 can train the graph neural network 150 using a supervised learning technique on a set of training data. The training data includes a set of training examples, and each training example specifies (i) respective training inputs that can be processed by the graph neural network 150, and (ii) corresponding target outputs that the graph neural network 150 is encouraged to generate by processing the training inputs. The training inputs are training node features for each node in the fine-resolution mesh 10.f

[0105]

Number

[0106] and training node features for each node in the coarse-resolution mesh 10.c at a particular time step t k include. In some implementations, the training node features associated with the nodes in the coarse-resolution mesh 10.c include only geometric features, such as node types that specify interior nodes or boundary nodes. The target outputs include one or more target dynamics features for each node in the fine-resolution mesh 10.f at the time step

[0107]

Number

[0108] include.

[0109]

Number

[0110] include.

[0111] The simulation system 100 can train the graph neural network 150 over multiple training iterations. In each training iteration, the simulation system 100 samples a batch of one or more training examples from the training data and provides them to the graph neural network 150 that can process the training inputs specified in the training examples to generate an estimated value of the target output, i.e., the predicted dynamics feature for the training input, corresponding outputs. The simulation system 100 can evaluate an objective function L, such as a cross-entropy or mean squared error objective function, that measures the similarity between (i) the target output specified by the training example and (ii) the output generated by the graph neural network 150. For example, the objective function L can be based on the error between the predicted dynamics feature for the nodes in the fine-resolution mesh 10.f

[0112]

Number

[0113] and the target dynamics feature for the nodes, as follows, i.e.,

[0114]

Number

[0115] can be based on like this, provided that d θis a function representing the graph neural network 150 model, and θ is the neural network parameter of the graph neural network 150. The simulation system 100 can use a per-node and per-time-step objective function as in Equation (3), or average the objective function over multiple nodes and / or multiple time steps. The simulation system 100 can determine the gradient of the objective function, for example, using an error backpropagation technique, and use that gradient to update the network parameter values of the graph neural network 150 to optimize the objective function, for example, using any suitable gradient descent optimization algorithm, such as Adam. The simulation system 100 can also determine the performance metric of the graph neural network 150 in a set of validation data not used during the training of the graph neural network 150.

[0116] To generate training data, simulation system 100 can use a simulation engine (e.g., a physics engine such as COMSOL Multiphysics from COMSOL Inc.) to simulate the state of the physical environment over one or more time steps. Specifically, simulation system 100 simulates the state of the physical environment on a mesh that has a higher resolution than the fine-resolution mesh 10.f processed by graph neural network 150. Simulation system 100 then generates a lower-resolution version of the simulation by interpolating the simulation (e.g., bilinearly or bicubically) to the resolutions of the fine-resolution mesh 10.f and the coarse-resolution mesh 10.c, and generates training data based on the lower-resolution version of the simulation. Specifically, simulation system 100 can determine a training input and a target output for each training example based on the lower-resolution version of the simulation. Generating training data in this way can increase the accuracy of the training data, thereby enabling the graph neural network 150 trained on the training data to achieve a higher simulation accuracy.

[0117] FIG. 4 is a diagram showing examples of a low-resolution simulation 410, a high-resolution simulation 420, and a lower-resolution version 430 of the high-resolution simulation 420 after interpolation. The simulation is of a Karman vortex street and was simulated using COMSOL. The grayscale in FIG. 4 shows the x-component of the velocity field. The low-resolution simulation 410 mesh is not fine enough to resolve all flow features, and characteristic vortex shedding is suppressed. The high-resolution simulation 420 with a finer mesh resolves the dynamics correctly. The high-accuracy prediction from the high-resolution simulation 420 is interpolated onto a lower-resolution version 430 of the high-resolution simulation 420 such that vortex shedding is still visible. The lower-resolution version 430 has the same resolution as the fine-resolution mesh 10.f and can be used by the simulation system 100 to generate training examples. In this way, the graph neural network 150 can implicitly learn the effects at a smaller scale without any changes to the model code and achieve predictions that are more accurate than those possible using a classical solver on a coarse scale at inference time.

[0118] After training the graph neural network 150, the simulation system 100 can be used to simulate the states of different types of physical environments. For example, the simulation system 100 can effectively generalize from single-time-step predictions using hundreds or thousands of nodes during training to different types of physical environments, different initial conditions, thousands of time steps, and at least an order of magnitude more nodes.

[0119] Figure 2A is a diagram showing the operation of an exemplary high-resolution update block 122.f used by update data 120 to perform node embedding updates on a high-resolution mesh 10.f. As can be seen in Figure 2A, each node 11.f.0 in the high-resolution mesh 10.f receives information from each adjacent node 11.f.1-6 connected to node 11.f.0 by an edge.

[0120] Each high-resolution update block 122.f includes one or more neural network layers and is configured to process data defining the high-resolution mesh 10.f to generate an updated node embedding for each node in the high-resolution mesh

[0121]

Number

[0122] Specifically, one or more first neural network layers of the high-resolution update block 122.f are configured to (i) edge embeddings of the edges in the high-resolution mesh 10.f

[0123]

Number

[0124] and (ii) respective node embeddings for each pair of nodes connected by an edge

[0125]

Number

[0126] and

[0127]

Number

[0128] Process an input including and generate an updated edge embedding for an edge

[0129]

Number

[0130] configured to be. Additionally, one or more second neural network layers of the high-resolution update block 122.f include (i) a node embedding of a node in the high-resolution mesh 10.f

[0131]

Number

[0132] and (ii) an updated edge embedding of each edge connected to the node

[0133]

Number

[0134] process an input including and generate an updated node embedding for the node

[0135]

Number

[0136] configured to be. For example, the high-resolution update block 122.f can generate the updated node embedding as

[0137]

Number

[0138] whereas F f and S fEach represents the operation of one or more first neural network layers and one or more second neural network layers of the fine-resolution update block 122.f. For example, each of the one or more first neural network layers and one or more second neural network layers of the fine-resolution update block 122.f can include respective multi-layer perceptrons (MLPs) with residual connections.

[0139] Each update block 122.f of the fine resolution can be a message passing block with a different set of network parameters. That is, each update block 122.f of the fine resolution can be identical to each other, that is, have the same neural network architecture, but have a separate set of neural network parameters. Alternatively, the update 120 can implement a single fine-resolution update block 122.f as a message passing block and can call the single fine-resolution update block 122.f one or more times when the block 112.f is implemented within the series of update blocks 122.

[0140] FIG. 2B is a diagram showing the operation of an exemplary coarse-resolution update block 122.c used by the update 120 to perform node embedding updates on the coarse-resolution mesh 10.c. As can be seen in FIG. 2B, each node 11.c.0 in the coarse-resolution mesh 10.c receives information from each adjacent node 11.c.1-5 connected to the node 11.c.0 by an edge.

[0141] Each update block 122.c of the coarse resolution includes one or more neural network layers, and the updated node embedding for each node in the coarse-resolution mesh 10.c

[0142]

Number

[0143] To generate, it is configured to process data defining a mesh 10.c with a coarse resolution. Specifically, one or more first neural network layers of the coarse-resolution update block 122.c perform (i) edge embedding of edges in the mesh 10.c with a coarse resolution

[0144]

Number

[0145] and (ii) node embedding for each pair of nodes connected by the edges

[0146]

Number

[0147] and

[0148]

Number

[0149] to process the input including them, and is configured to update the edge embedding for the edges

[0150]

Number

[0151] In addition, one or more second neural network layers of the coarse-resolution update block 122.c perform (i) node embedding of nodes in the mesh 10.c with a coarse resolution

[0152]

Number

[0153] and (ii) the updated edge embedding of each edge connected to the node

[0154]

Number

[0155] process an input including the above to generate an updated node embedding for the node

[0156]

Number

[0157] is configured to generate. For example, the low-resolution update block 122.c can generate the updated node embedding as

[0158]

Number

[0159] , provided that F c and S c represent the operations of one or more first neural network layers and one or more second neural network layers of the low-resolution update block 122.c, respectively. For example, one or more first neural network layers and one or more second neural network layers of the low-resolution update block 122.c can each include a respective multi-layer perceptron (MLP) having a residual connection.

[0160] Each update block 122.c with coarse resolution can be a message passing block having different sets of network parameters. That is, each update block 122.c with coarse resolution can be identical to each other, that is, having the same neural network architecture but having separate sets of neural network parameters. Alternatively, the update 120 can use a single update block 122.c with coarse resolution as a message passing block, and when the block 122.c is implemented in the series 122 of update blocks, the single update block 122.c with coarse resolution can be called one or more times.

[0161] FIG. 2C is a diagram showing the operation of an exemplary upsampling update block 122.u used by the update 120 to perform node embedding updates on the fine-resolution mesh 10.f using information about the coarse-resolution mesh 10.c. As seen in FIG. 2C, each node 11.f in the fine-resolution mesh 10.f receives information from each of the nodes 11.c.1-3 in the coarse-resolution mesh 10.c that are vertices of the cell 14.c surrounding the node 11.f.

[0162] Each upsampling update block 122.u is configured to generate data defining an upsampling mesh G u =(V u , E u ). The set of nodes V u =V f ∪V c of the upsampling mesh includes each node from the fine-resolution mesh 10.f and each node from the coarse-resolution mesh 10.c. The set of edges E u of the upsampling mesh includes the edges between the nodes of the fine-resolution mesh 10.f and the nodes of the coarse-resolution mesh 10.c. Generally, the upsampling update block 122.u transfers information from the nodes in the coarse-resolution mesh 10.c to the nodes in the fine-resolution mesh 10.f using the edges of the upsampling mesh.

[0163] The upsampling update block 122.u can generate the edges of the upsampling mesh as follows. For each node i ∈ V in the coarse-resolution mesh c the upsampling update block 122.u identifies the cells of the fine-resolution mesh 10.f that contain the nodes of the coarse-resolution mesh 10.c. The upsampling update block 122.u identifies one or more nodes j = j(i) ∈ V in the fine-resolution mesh 10.f that are the vertices of the cell f The upsampling update block 122.u then instantiates each edge k ij ∈ E u in the upsampling mesh between each node of the coarse-resolution mesh 10.c and each of the identified nodes in the fine-resolution mesh 10.f. The upsampling update block 122.u then performs edge embedding for each edge in the upsampling mesh based on, for example, the respective positions of the nodes connected by the edge, the difference between the respective positions of the nodes connected by the edge, the magnitude of the difference between the respective positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof

[0164]

Number

[0165] to generate.

[0166] Each upsampling update block 122.u includes one or more neural network layers and performs updated node embedding for each node in the fine-resolution mesh 10.f

[0167]

Number

[0168] It is configured to process data that defines an upsampling mesh in order to generate . Specifically, one or more first neural network layers of the upsampling update block 122.u process (i) the edge embedding of the edges in the upsampling mesh

[0169]

Number

[0170] and (ii) the node embedding of each of the first nodes in the coarse-resolution mesh 10.c and the second nodes in the fine-resolution mesh 10.f connected by the edges

[0171]

Number

[0172] and

[0173]

Number

[0174] to process an input including the above and update the edge embedding for the edges

[0175]

Number

[0176] In addition, one or more second neural network layers of the upsampling update block 122.u process (i) the node embedding of the nodes in the fine-resolution mesh 10.f

[0177]

Number

[0178] and (ii) the updated respective edge embeddings of each edge in the upsampling mesh connected to the node

[0179]

Number

[0180] process an input including the above to generate an updated node embedding for the node

[0181]

Number

[0182] For example, the upsampling update block 122.u is configured to generate the updated node embedding as

[0183]

Number

[0184] where F u and S u represent the operations of one or more first neural network layers and one or more second neural network layers of the upsampling update block 122.u respectively. For example, one or more first neural network layers and one or more second neural network layers of the upsampling update block 122.u can each include respective multi-layer perceptrons (MLPs) with residual connections.

[0185] Each upsampling update block 122.u can be a message passing block having a different set of network parameters. That is, each upsampling update block 122.u can be identical to each other, that is, having the same neural network architecture, but having a separate set of neural network parameters. Alternatively, the update data 120 can use a single upsampling update block 122.u as a message passing block, and when the block 122.u is implemented in the series 122 of update blocks, the single upsampling update block 122.u can be called one or more times.

[0186] FIG. 2D is a diagram showing the operation of an exemplary downsampling update block 122.d used by the update data 120 to perform node embedding updates on the coarse resolution mesh 10.c using information about the fine resolution mesh 10.f. As seen in FIG. 2D, each node 11.c in the coarse resolution mesh 10.c receives information from each of the nodes 11.f.1 to 3 in the fine resolution mesh 10.f that are vertices of the cells 14.f surrounding the node 11.c.

[0187] Each downsampling update block 122.d is configured to generate data defining a downsampling mesh G d =(V d ,E d ). The set V d =V f ∪V c of the nodes of the downsampling mesh includes each node from the fine resolution mesh 10.f and each node from the coarse resolution mesh 10.c. The set E dIt includes edges between the nodes of the fine-resolution mesh 10.f and the nodes of the coarse-resolution mesh 10.c. Generally, the downsampling update block 122.d transfers information from the nodes in the fine-resolution mesh 10.f to the nodes in the coarse-resolution mesh 10.c using the edges of the downsampling mesh.

[0188] The downsampling update block 122.d can generate the edges of the downsampling mesh as follows. For each node i ∈ V in the fine-resolution mesh f the downsampling update block 122.d identifies the cells of the coarse-resolution mesh 10.c that contain the nodes of the fine-resolution mesh 10.f. The downsampling update block 122.d identifies one or more nodes j = j(i) ∈ V in the coarse-resolution mesh 10.c that are the vertices of the cell. c The downsampling update block 122.d then instantiates each edge k ij ∈ E d in the downsampling mesh between the node of the fine-resolution mesh 10.f and each of the identified nodes in the coarse-resolution mesh 10.c. The downsampling update block 122.d then performs edge embedding for each edge in the downsampling mesh based on, for example, the respective positions of the nodes connected by the edge, the difference between the respective positions of the nodes connected by the edge, the magnitude of the difference between the respective positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof

[0189]

Number

[0190] to generate.

[0191] Each downsampling update block 122.d includes one or more neural network layers and is configured to process data that defines a downsampling mesh to generate an updated node embedding for each node in the coarse-resolution mesh 10.c. Specifically, one or more first neural network layers of the downsampling update block 122.d process an input that includes (i) an edge embedding of an edge in the downsampling mesh, and (ii) respective node embeddings of a first node in the fine-resolution mesh 10.f and a second node in the coarse-resolution mesh 10.c that are connected by the edge, to generate an updated edge embedding for the edge.

[0192]

Number

[0193] to generate an updated node embedding for each node in the coarse-resolution mesh 10.c. Specifically, one or more first neural network layers of the downsampling update block 122.d process an input that includes (i) an edge embedding of an edge in the downsampling mesh,

[0194]

Number

[0195] and (ii) respective node embeddings of a first node in the fine-resolution mesh 10.f and a second node in the coarse-resolution mesh 10.c that are connected by the edge,

[0196]

Number

[0197] and

[0198]

Number

[0199] to generate an updated edge embedding for the edge.

[0200]

Number

[0201] configured to generate. Additionally, one or more second neural network layers of the downsampling update block 122.d process an input including (i) the node embeddings of the nodes in the coarse-resolution mesh 10.c

[0202]

Number

[0203] and (ii) the respective updated edge embeddings of each edge in the downsampling mesh connected to the nodes

[0204]

Number

[0205] to generate an updated node embedding for the nodes. For example, the downsampling update block 122.d can generate the updated node embedding as

[0206]

Number

[0207] whereas F

[0208]

Number

[0209] and S d and drespectively represent the operations of one or more first neural network layers and one or more second neural network layers of the downsampling update block 122.d. For example, one or more first neural network layers and one or more second neural network layers of the downsampling update block 122.d can each include a respective multi-layer perceptron (MLP) with residual connections.

[0210] Each downsampling update block 122.d can be a message passing block with a different set of network parameters. That is, each downsampling update block 122.d can be identical to each other, that is, have the same neural network architecture, but have a separate set of neural network parameters. Alternatively, the update 120 can use a single downsampling update block 122.d as the message passing block, and when the block 122.d is implemented in the series 122 of update blocks, the single downsampling update block 122.d can be called one or more times.

[0211] Figures 3A and 3B are block diagrams of an exemplary update data module 120 topology that uses various series 122 of update blocks to update node embeddings for nodes in the fine-resolution mesh 10.f and the coarse-resolution mesh 10.c. Updates in the fine-resolution mesh 10.f are shown using solid arrows, and updates in the coarse-resolution mesh 10.c are shown using dashed arrows. This topology enables the update 120 to perform efficient message passing. Specifically, the coarse-resolution update block 122.c is significantly faster than the fine-resolution update block 122.f due to the smaller number of nodes and edges in the coarse-resolution mesh 10.c compared to the fine-resolution mesh 10.f. The coarse-resolution update block 122.c can also propagate more information in the coarse-resolution mesh 10.c.

[0212] Therefore, the update data 120 can implement an efficient update method by performing updates several times (e.g., 1 to 4 times) in the high-resolution mesh 10.f using several (e.g., 1 to 4) high-resolution update blocks 122.f to aggregate local features, can be downsampled to the low-resolution mesh 10.f using the downsampling update block 122.d, can perform many (e.g., 10 to 100) updates in the low-resolution mesh 10.c using many (e.g., 10 to 100) low-resolution update blocks 122.c, can be upsampled to the high-resolution mesh 10.f using the upsampling update block 122.u, and can calculate small-scale dynamics by performing updates several times (e.g., 1 to 4 times) in the high-resolution mesh 10.f using several (e.g., 1 to 4) high-resolution update blocks 122.f. Such an update method in which the update data 120 performs updates in the high-resolution mesh 10.f, downsampling, updates in the low-resolution mesh 10.c, and then returns to the high-resolution mesh 10.f for upsampling is called a "block cycle". The update data 120 can perform any number of these block cycles, as described below.

[0213] In FIG. 3A, the update data 120 uses a series 122 of N + 4 update blocks that perform a single block cycle. In this case, the first high-resolution update block 122.f.1 is followed by a downsampling update block 122.d, a series of a plurality (N) of low-resolution update blocks 122.f.1 to N, an upsampling update block 122.u, and a second high-resolution update block 122.f.2. Collectively, the series 122 of update blocks may be denoted as "f-d-Nc-u-f", where "f" represents the high-resolution update block 122.f, "c" represents the low-resolution update block 122.c, "u" represents the upsampling update block 122.u, and "d" represents the downsampling update block 122.d.

[0214] In FIG. 3B, the update data 120 uses a series 122 of 11 update blocks that perform two block cycles. In this case, the series 122 of update blocks may be denoted as "f-d-2c-u-f-d-2c-u-f".

[0215] FIG. 5 is a flowchart of an exemplary process for simulating the state of a physical environment using a graph neural network. For convenience, the process 500 will be described as being executed by a system of one or more computers located at one or more locations. For example, a simulation system appropriately programmed in accordance with this specification, such as the simulation system 100 of FIG. 1A, can execute the process 500.

[0216] For each of a plurality of time steps, the simulation system performs the following operations.

[0217] The simulation system obtains (502) data that defines a fine-resolution mesh and a coarse-resolution mesh, each characterizing the state of the physical environment at the current time step. The fine-resolution mesh and the coarse-resolution mesh each have a respective set of nodes and edges that can represent the physical environment, an area extending across the physical environment, or one or more objects within the physical environment. The fine-resolution mesh has a higher resolution than the coarse-resolution mesh; for example, the fine-resolution mesh has a greater number of nodes than the coarse-resolution mesh. The mesh can be a one-dimensional mesh, a two-dimensional mesh, a three-dimensional mesh, or a mesh with more than three dimensions. In some implementations, the mesh is a triangular mesh, i.e., has cells in the shape of triangles. The data that defines the fine-resolution mesh and the coarse-resolution mesh at the current time step includes the current node embedding for the nodes in the fine-resolution mesh and the current node embedding for the nodes in the coarse-resolution mesh. The data can also include the current edge embedding for the edges in the fine-resolution mesh and the current edge embedding for the edges in the coarse-resolution mesh.

[0218] The simulation system can obtain data defining a fine-resolution mesh by obtaining, for each node in the fine-resolution mesh, one or more current node features for the node that characterize the state of the physical environment at the location in the physical environment corresponding to the node. For example, the node features at an initial time step can be provided by a user, e.g., through an API, and then the simulation system can execute process 500 to obtain node features for each subsequent time step. In some implementations, the node features include one or more of fluid density, fluid viscosity, pressure, or tension at the location in the physical environment corresponding to the node at the current time step. The simulation system can then process the one or more node features for each node in the fine-resolution mesh using an encoder module of a graph neural network to generate a current node embedding for the node. The simulation system can also generate a current edge embedding for each edge in the fine-resolution mesh using the encoder module based on the current node features and / or respective positions for the nodes connected by the edge.

[0219] The simulation system can obtain data defining a coarse-resolution mesh in a similar manner. In some implementations, the current node features for nodes in the coarse-resolution mesh are averaged and / or interpolated from the current node features for nodes in the fine-resolution mesh. In some implementations, the current node features for nodes in the coarse-resolution mesh include only geometric (e.g., static) features that do not change with each time step. For example, the geometric features can include a node type that designates an interior node or a boundary node. In these cases, the simulation system can reuse the node features for nodes in the coarse-resolution mesh from a previous time step.

[0220] The simulation system processes data defining a fine-resolution mesh and a coarse-resolution mesh (504) using an update data module of a graph neural network to update the current node embedding for nodes in the fine-resolution mesh.

[0221] The update data module includes (i) one or more fine-resolution update blocks, (ii) one or more coarse-resolution update blocks, (iii) one or more upsampling update blocks, and (iv) one or more downsampling update blocks. The update data module can perform various different sequences of update blocks, for example, in the form of one or more block cycles. For example, to perform a block cycle, the update data module can include a sequence of one or more fine-resolution update blocks, downsampling update blocks, one or more coarse-resolution update blocks, and upsampling update blocks.

[0222] Each fine-resolution update block is configured to process data defining the fine-resolution mesh using a graph neural network layer to update the current node embedding for each node in the fine-resolution mesh. For example, a fine-resolution update block can update the edge embedding for each edge in the fine-resolution mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of the nodes in the fine-resolution mesh connected by the edge. The fine-resolution update block can then update the node embedding for each node in the fine-resolution mesh based on (i) the node embedding for the node in the fine-resolution mesh and (ii) the respective edge embeddings of each edge connected to the node.

[0223] Each low-resolution update block is configured to process data defining a low-resolution mesh using a graph neural network layer to update the current node embedding of each node in the low-resolution mesh. For example, a low-resolution update block can update the edge embedding for each edge in the low-resolution mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of the nodes in the low-resolution mesh connected by the edge. The low-resolution update block can then update the node embedding for each node in the low-resolution mesh based on (i) the node embedding for the node in the low-resolution mesh and (ii) the respective edge embeddings of each edge connected to the node.

[0224] Each upsampling update block is configured to generate data defining an upsampling mesh. The upsampling mesh includes (i) each node from the fine-resolution mesh and each node from the low-resolution mesh, and (ii) a plurality of edges between the nodes of the fine-resolution mesh and the nodes of the low-resolution mesh. For example, for each node in the low-resolution mesh, the upsampling update block can identify the cells of the fine-resolution mesh that include the node of the low-resolution mesh. The upsampling update block can then identify one or more nodes in the fine-resolution mesh that are vertices of the cell that includes the node of the low-resolution mesh. The upsampling update block can then instantiate each edge in the upsampling mesh between the node of the low-resolution mesh and each of the identified nodes in the fine-resolution mesh. The upsampling update block can then generate an edge embedding for each edge in the upsampling mesh based on the respective positions between pairs of nodes in the upsampling mesh connected by the edge, e.g., the distance between pairs of nodes in the upsampling mesh connected by the edge.

[0225] Each upsampling update block is further configured to process data defining an upsampling mesh using a graph neural network layer to update the current node embedding of each node in a fine-resolution mesh. For example, an upsampling update block can update the edge embedding for each edge in the upsampling mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of the first node in the coarse-resolution mesh and the second node in the fine-resolution mesh connected by the edge. The upsampling update block can then update the node embedding for each node in the fine-resolution mesh based on (i) the node embedding for the node in the fine-resolution mesh and (ii) the respective edge embeddings of each edge connecting the node in the fine-resolution mesh to the corresponding node in the coarse-resolution mesh.

[0226] Each downsampling update block is configured to generate data that defines a downsampling mesh. The downsampling mesh includes (i) each node from a fine-resolution mesh and each node from a coarse-resolution mesh, and (ii) a plurality of edges between the nodes of the fine-resolution mesh and the nodes of the coarse-resolution mesh. For example, for each node of the fine-resolution mesh, the downsampling update block can identify a cell of the coarse-resolution mesh that includes the node of the fine-resolution mesh. The downsampling update block can then identify one or more nodes of the coarse-resolution mesh that are vertices of the cell that includes the node of the fine-resolution mesh. The downsampling update block can then instantiate each edge in the downsampling mesh between the node of the fine-resolution mesh and each of the identified nodes of the coarse-resolution mesh. The downsampling update block then generates an edge embedding for each edge in the downsampling mesh based on the respective position between pairs of nodes in the downsampling mesh that are connected by the edge, e.g., the distance between pairs of nodes in the downsampling mesh that are connected by the edge.

[0227] Each downsampling update block is further configured to process data defining a downsampled mesh using a graph neural network layer to update the current node embedding of each node in a coarse-resolution mesh. For example, a downsampling update block can update the edge embedding for each edge in the downsampled mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of a first node in the coarse-resolution mesh and a second node in the fine-resolution mesh connected by the edge. The downsampling block can then update the node embedding for each node in the coarse-resolution mesh based on (i) the node embedding for the node in the coarse-resolution mesh and (ii) the respective edge embeddings of each edge connecting the node in the coarse-resolution mesh to the corresponding node in the fine-resolution mesh.

[0228] The simulation system determines the state of the physical environment at the next time step (506) using the updated node embeddings for the nodes in the fine-resolution mesh. For example, the simulation system can process the updated node embeddings for the nodes in the fine-resolution mesh using a decoder module to generate one or more respective dynamics features corresponding to each node in the fine-resolution mesh. The simulation system can then use a prediction engine to determine the state of the physical environment at the next time step based on (i) the dynamics features for the nodes in the fine-resolution mesh and (ii) the node features for the nodes in the fine-resolution mesh at the current time step.

[0229] Generally, a graph neural network is trained on a set of training examples to generate an accurate prediction of the physical environment it is modeling. For example, for high-accuracy predictions, a simulation system can use a simulation engine (e.g., a physics engine) to generate a target simulation of the state of a training physical environment over one or more time steps, where the target simulation has a higher resolution than the fine-grained mesh processed by the graph neural network. The simulation system can then interpolate the target simulation to the same resolution as the fine-grained mesh processed by the graph neural network to generate a lower-resolution version of the target simulation. The simulation system can then use the lower-resolution version of the simulation mesh to generate one or more of the training examples.

[0230] The above-described systems and methods may be adapted for implementation on a computing system that includes first and second processors (or processor blocks) that communicate with each other and have different relative capabilities. Specifically, here, the second processor has a relatively greater processing capacity or memory than the first processor. As an example, such a computing system may include a first general-purpose processor and a second processor having one or more neural network accelerators. A neural network accelerator is special hardware used to accelerate neural network computations, such as a GPU (Graphics Processing Unit) or TPU (Tensor Processing Unit). Generally, a neural network accelerator is configured to perform hardware matrix multiplication, for example, using parallel computing. A neural network accelerator can include a set of one or more multiply-accumulate units (MACs) for performing such operations. As another example, the first processor may include, for example, a first general-purpose processor having a first computing capacity defined in terms of FLOPS (floating-point operations per second) and / or a certain amount of memory available for computing. The second processor may include a second general-purpose processor having a greater second computing capacity, for example, a greater number of FLOPS, and / or a greater amount of memory available for computing. As a further example, the first processor may include a processor having a first number of neural network accelerators, and the second processor may include a processor having a greater second number of neural network accelerators.

[0231] In such a computing system, the second processor can be used for the update of the fine-resolution mesh 10.f, and the first processor can be used for the update of the coarse-resolution mesh 10.c. That is, the graph neural network 150 can be distributed between the first processor and the second processor to optimally allocate the computing resources for the mesh updates of the fine resolution 10.f and the coarse resolution 10.c. For example, one or more fine-resolution update blocks 122.u can be implemented on the second processor, and one or more coarse-resolution update blocks 122.c can be implemented on the first processor. Since the fine-resolution update block 122.f is generally computationally more expensive than the coarse-resolution update block 122.c, this enables the simulation system 100 to more efficiently simulate the state of the physical environment.

[0232] Thus, in some implementations, the simulation system 100 processes the data defining the fine-resolution mesh 10.f by implementing one or more fine-resolution update blocks 122.f on the second processor, and processes the data defining the coarse-resolution mesh 10.c by implementing one or more coarse-resolution update blocks 122.c on the first processor. One or more upsampling update blocks 122.u can be implemented on the first processor and / or the second processor. Similarly, one or more downsampling update blocks 122.d can be implemented on the first processor and / or the second processor.

[0233] The processors (processor blocks) can operate in parallel, but this can be inefficient, for example, when inputs and outputs are defined only on a fine-resolution mesh 10.f and the first and last updates on other meshes are wasted. Thus, in some implementations, the simulation system 100 first processes the data defining the fine-resolution mesh 10.f by performing one or more fine-resolution update blocks 122.f on a second processor, then processes the data defining the downsampling mesh (using any processor) to update the current node embedding of each node in the coarse-resolution mesh 10.c, then processes the data defining the coarse-resolution mesh 10.c by performing one or more coarse-resolution update blocks 10.c on the first processor, and then processes the data defining the upsampling mesh (using any processor) to update the current node embedding of each node in the fine-resolution mesh 10.f. The step of processing the data defining the coarse-resolution mesh 10.c by performing one or more coarse-resolution update blocks on the first processor can include performing multiple updates of the data defining the coarse-resolution mesh 10.c on the first processor.

[0234] Some implementations of the systems and methods described above can be used for real-world control, such as controlling a mechanical agent, e.g., a robot, in a real-world environment to perform tasks, for example, using the simulation system 100 for model-based predictive control or as part of an optimal control system for controlling an agent. As an example, the simulation system 100 may be used in this way to assist a robot in manipulating deformable objects.

[0235] More specifically, the physical environment can be the real-world environment that includes physical objects, for example, objects to be picked up and / or manipulated by a robot. The simulation system 100 can be used to control the robot. Specifically, obtaining data characterizing the state of the physical environment at the current time step can include determining a representation of the location, shape, or configuration of the physical object, for example, by capturing an image of the object. For example, the simulation system 100 can determine node features for nodes in a mesh of fine resolution 10.f and coarse resolution 10.c from the representation of the physical object by determining node embeddings for the nodes and then generating node embeddings for the nodes. The simulation system 100 can determine the state of the physical environment at the next time step, for example, by determining a predicted representation of the location, shape, or configuration of the physical object when receiving, for example, a force or deformation from the robot's actuator.

[0236] The simulation system 100 can use, for example, the actuator to control the robot using the predicted representation at the next time step to manipulate the physical object. For example, the simulation system 100 can control the robot to optimize an objective function that depends on the difference between the predicted representation and the target location, target shape, or target configuration of the physical object, so as to manipulate the physical object towards the target location, target shape, or target configuration of the physical object using the predicted representation. Controlling the robot can include the simulation system 100 providing a control signal to the robot based on the predicted representation to cause the robot to perform an action to manipulate the physical object to execute a task, for example, using the actuator.

[0237] Some examples of simulation system 100 use a reinforcement learning process with a reward that is at least partially based on the value of an objective function to learn to perform tasks involving manipulating physical objects, such as controlling the actuators of a robot. Alternatively or in addition, this may involve the simulation system 100 controlling the robot using a model predictive control (MPC) process or an optimal control process.

[0238] Figures 6A and 6B are plots of experimental data showing mean squared error (MSE) versus minimum edge length (edge min) for (i) a reference simulator (COMSOL), (ii) two variant forms of a MeshGraphNets (MGN) learning solver having 15 message passing steps (mps) and 25 mps, respectively, and (iii) a fine-resolution mesh of exemplary simulation systems 100-1 and 100-2 using two different update module topologies having 15 mps and 25 mps, respectively. 10 -2 For each of simulation systems 100-1 and 100-2 having a fixed resolution corresponding to the minimum edge length, the same coarse-resolution mesh is used. The update module of the first simulation system 100-1 includes a series of 15 blocks that perform a single block cycle "f-d-11c-u-f", thereby resulting in a total of 15 mps. The update module of the second simulation system 100-2 includes a series of 25 blocks that perform two block cycles "3f-d-6c-u-3f-d-6c-u-3f", thereby resulting in a total of 25 mps.

[0239] The set of training data for the MGN model and exemplary simulation system includes 1000 trajectories of incompressible flow over a long cylinder in a channel, simulated using COMSOL. Each trajectory includes 200 time steps. Parameters such as the radius and position of the obstacle, the inflow initial velocity, and the mesh resolution vary between trajectories. In particular, the mesh resolution covers a wide range from 100 to 10000 nodes.

[0240] The influence of mesh resolution in each prediction of the simulation system is evaluated against a set of validation data that includes 500 trajectories with various mesh resolutions but otherwise constant initial conditions. The minimum edge length (edge min) of these meshes ranges from 10 -2 to 10 -3 . Since analytical solutions are generally not available for non-trivial simulation setups, high-resolution simulations are typically used as a proxy for the "ground truth" solution of the underlying partial differential equation (PDE). Here, by running COMSOL at the maximum resolution in this validation data set, the ground truth reference trajectory (u ref ) was generated (edge min = 10 -2 ). The error is measured by performing next-step prediction on the validation data set using a learning model or classical solver at a given mesh resolution, linearly interpolating the ground truth trajectory onto the simulation mesh, and calculating the MSE.

[0241] The results in Fig. 6A show a significant reduction in MSE for simulation systems 100-1 and 100-2 compared to the MGN baseline, with the overall number of mps fixed. The second simulation system 100-2 with 25 mps manages to closely track the spatial convergence curve of the reference simulator. Thus, the simulation system 100 is effective in solving the message passing bottleneck for the root problem and can achieve higher accuracy using the same number of mps as other graph neural network models. The message passing speed is a bottleneck for MGN performance for high-resolution meshes, but this bottleneck is removed using the simulation system 100 with the multiscale mesh method.

[0242] In Fig. 6B, both the MGN models (15 mps and 25 mps) and simulation systems 100-1 and 100-2 are trained on a training dataset with a mixed mesh resolution, but with high-accuracy predictions as described above (see, for example, Fig. 4). This indicates that the learning solver can learn an effective model of subgrid dynamics and make accurate predictions even at very coarse mesh resolutions. The effect extends to an edge length of 10 -2 corresponding to a very coarse mesh with only about 100 nodes. However, this method does not alleviate the message propagation bottleneck for the MGN model, and the error increases above the convergence curve for edge lengths less than 0.0016. Thus, if a highly resolved output mesh is desired, the accuracy remains limited using MGN. The label "simulation system 100 with high accuracy" can be used for a method that performs well in both low-resolution and very high-resolution meshes. In the case of the second simulation system 100-2 with 25 mps, the error remains below the reference solver curve at all resolutions, with all the performance advantages of the simulation system 100.

[0243] This specification uses the term "configured" with respect to systems and computer program components. For one or more computer systems to be configured to perform a particular operation or action means that software, firmware, hardware, or combinations thereof that cause the system to perform that operation or action in operation are installed on the system. For one or more computer programs to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing apparatus, cause the apparatus to perform that operation or action.

[0244] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, and structural equivalents thereof, including the structures disclosed in this specification, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. A computer storage medium can be, or include, a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or one or more combinations of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus.

[0245] The term "data processing apparatus" refers to data processing hardware and includes, by way of example, all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus can also be, or further include, dedicated logic circuit configurations, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The apparatus can optionally include, in addition to the hardware, code for creating an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code constituting one or more combinations thereof.

[0246] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The program may or may not correspond to a file in a file system. The program can be stored in a part of a file that holds one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, such as files that store one or more modules, subprograms, or portions of code. The computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0247] In this specification, the term "engine" is widely used to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or software components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and operating on one or more of the same computers.

[0248] The processes and logical flows described in this specification can be executed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be executed by a dedicated logic circuit configuration, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit configuration and one or more programmed computers.

[0249] A computer suitable for the execution of a computer program can be based on a general purpose or special purpose microprocessor or both, or any other kind of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory or both. Essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and the memory can be augmented by, or incorporated in, dedicated logic circuitry. Generally, a computer also includes, or is operatively coupled to receive from and transfer data to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, it is not essential for a computer to have such devices. Moreover, a computer can be incorporated in another device, such as a cellular phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0250] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0251] To interact with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Also for interacting with a user, other types of devices may be used, for example, the feedback provided to the user may be any form of perceptual feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, the computer can interact with the user by sending documents to the devices used by the user and receiving documents from such devices, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser, and the computer can also interact with the user by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0252] A data processing apparatus for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing common and computationally intensive parts of machine learning training or manufacturing, i.e., inference, workload.

[0253] The machine learning model may be implemented and deployed using a machine learning framework, such as the TensorFlow framework.

[0254] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes, for example, a back-end component as a data server, or includes a middleware component, such as an application server, or a front-end component, such as a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as by a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0255] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, such as an HTML page, to a user device for the purpose of, for example, displaying data to a user interacting with a device acting as a client and receiving user input from such users. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0256] This specification includes many specific implementation details, but these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination within a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately, or in any suitable sub-combination, in multiple embodiments. Moreover, features may be described above as acting in certain combinations and even initially claimed as such, but one or more features from the claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to a sub-combination or variant of a sub-combination.

[0257] Similarly, operations are shown in the drawings and recited in the claims in a particular order, but this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all illustrated operations be performed, to achieve desirable results. In some environments, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together within a single software product or packaged into multiple software products.

[0258] Certain embodiments of the present subject matter are described. Other embodiments fall within the scope of the following claims. For example, the actions recited in the claims can be performed in various orders and still achieve desirable results. As an example, the processes shown in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Explanation of Signs

[0259] 10.c Coarse-resolution mesh 10.f Fine-resolution mesh 11.c Node 11.f Node 13.c Edge 13.f Edge 14.f Cell 14.c Cell 100 Simulation system 102 Current state of the physical environment 104.c Current node feature 104.f Current node feature 108 Global feature 110 Encoder module 114.c Current node embedding 114.f Current node embedding 120 Update module 122 Series of update blocks 122.c Coarse-resolution update block 122.d Downsampling update block 122.f Fine-resolution update block 122.u Upsampling update block 130 Decoder module 134.f Final updated node embedding 144.f Dynamics feature 150 Graph neural network 160 Prediction engine Next state of the 202 physical environment 410 Low-resolution simulation 420 High-resolution simulation 430 Lower-resolution version

Claims

1. A method executed by one or more computers for simulating the state of a physical environment, for each of a plurality of time steps, obtaining data defining a fine-resolution mesh and a coarse-resolution mesh, each characterizing the state of the physical environment at the current time step, wherein the fine-resolution mesh has a higher resolution than the coarse-resolution mesh; processing the data defining the fine-resolution mesh and the coarse-resolution mesh using a graph neural network, the graph neural network comprising (i) one or more fine-resolution update blocks, (ii) one or more coarse-resolution update blocks, and (iii) one or more upsampling update blocks; each fine-resolution update block being configured to process the data defining the fine-resolution mesh using a graph neural network layer to update the current node embedding of each node in the fine-resolution mesh; each coarse-resolution update block being configured to process the data defining the coarse-resolution mesh using a graph neural network layer to update the current node embedding of each node in the coarse-resolution mesh; each upsampling update block (i) generating data defining an upsampling mesh comprising each node from the fine-resolution mesh and each node from the coarse-resolution mesh, and (ii) a plurality of edges between the nodes of the fine-resolution mesh and the nodes of the coarse-resolution mesh; processing the data defining the upsampling mesh using a graph neural network layer to update the current node embedding of each node in the fine-resolution mesh; configured to perform; determining the state of the physical environment at the next time step using the updated node embeddings for the nodes in the fine-resolution mesh; and a method.

2. The step of generating the upsampling mesh, for each node of the coarse-resolution mesh, Identifying cells of the fine-resolution mesh that contain the nodes of the coarse-resolution mesh; Identifying one or more nodes in the fine-resolution mesh that are vertices of the cells that contain the nodes of the coarse-resolution mesh; Instantiating each edge in the upsampling mesh between the nodes of the coarse-resolution mesh and each of the identified nodes in the fine-resolution mesh; The method according to claim 1, comprising the steps of.

3. For each edge in the upsampling mesh, Generating an edge embedding for the edge based on the distance between the pair of nodes in the upsampling mesh connected by the edge; The method according to claim 2, further comprising the steps of.

4. Using a graph neural network layer to process data defining the upsampling mesh to update the current node embedding of each node in the fine-resolution mesh, (i) updating the edge embedding for each edge in the upsampling mesh based on the edge embedding for the edge and (ii) the respective node embeddings of the first node in the coarse-resolution mesh and the second node in the fine-resolution mesh connected by the edge; (i) updating the node embedding for each node in the fine-resolution mesh based on the node embedding for the node in the fine-resolution mesh and (ii) the respective edge embeddings of each edge connecting the node in the fine-resolution mesh to the corresponding node in the coarse-resolution mesh; The method according to any one of claims 1 to 3, comprising the steps of.

5. The method according to any one of claims 1 to 4, wherein each upsampling block updates the current node embedding of the nodes in the fine-resolution mesh based at least in part on the current node embedding of the nodes in the coarse-resolution mesh.

6. The graph neural network further comprises one or more downsampling update blocks, and each downsampling update block, generating data that defines a downsampling mesh comprising (i) each node from the fine-resolution mesh and each node from the coarse-resolution mesh, and (ii) a plurality of edges between the nodes of the fine-resolution mesh and the nodes of the coarse-resolution mesh; processing the data that defines the downsampling mesh using a graph neural network layer to update the current node embedding of each node in the coarse-resolution mesh; The method according to any one of claims 1 to 5, which is configured to perform the above. **Claim 7** The step of generating the downsampling mesh, for each node of the fine-resolution mesh, identifying a cell of the coarse-resolution mesh that includes the node of the fine-resolution mesh; identifying one or more nodes of the coarse-resolution mesh that are vertices of the cell that includes the node of the fine-resolution mesh; instantiating each edge in the downsampling mesh between the node of the fine-resolution mesh and each of the identified nodes of the coarse-resolution mesh. The method according to claim 6, comprising the above. **Claim 8** For each edge in the downsampling mesh, generating an edge embedding for the edge based on the distance between the pair of nodes in the downsampling mesh connected by the edge. The method according to claim 7, further comprising the above. **Claim 9** The step of processing the data that defines the downsampling mesh using a graph neural network layer to update the current node embedding of each node in the coarse-resolution mesh comprises: (i) updating the edge embedding for each edge in the downsampling mesh based on (i) the edge embedding for the edge and (ii) the respective node embeddings of a first node in the coarse-resolution mesh and a second node in the fine-resolution mesh connected by the edge. (i) the node embedding for the nodes in the mesh of the coarse resolution, and (ii) based on each edge embedding of each edge connecting the nodes in the mesh of the coarse resolution to corresponding nodes in the mesh of the fine resolution, updating the node embedding for each node in the mesh of the coarse resolution The method according to any one of claims 6 to 8, comprising the above steps **Claim 10** The method according to any one of claims 6 to 9, wherein each downsampling block updates the current node embedding of the nodes in the coarse-resolution mesh based at least in part on the current node embedding of the nodes in the fine-resolution mesh **Claim 11** The graph neural network is trained on a set of training examples, one or more of the training examples being generated by operations, the operations being generating a target simulation of the state of a training physical environment over one or more time steps using a simulation engine, the target simulation having a higher resolution than the fine-resolution mesh processed by the graph neural network generating a lower-resolution version of the target simulation by interpolating the target simulation to the same resolution as the fine-resolution mesh processed by the graph neural network generating the training example using the lower-resolution version of the simulation mesh The method according to any one of claims 1 to 10, comprising the above steps **Claim 12** The step of obtaining data defining the state of the physical environment at the current time step, for each node in the fine-resolution mesh obtaining one or more node features for the node, the node corresponding to a position in the physical environment, the node features characterizing the state of the corresponding position in the physical environment processing the node features using one or more neural network layers of the graph neural network to generate the current embedding for the node The method according to any one of claims 1 to 11, comprising

13. The method according to claim 12, wherein for each node in the mesh of the fine resolution, the node feature for the node comprises one or more of a fluid density feature, a fluid viscosity feature, a pressure feature, or a tension feature.

14. The graph neural network further comprises a decoder block, and the step of determining the state of the physical environment at the next time step comprises processing the updated node embedding for each node in the mesh of the fine resolution to generate one or more respective dynamics features corresponding to each node in the mesh of the fine resolution; and (i) determining the state of the physical environment at the next time step based on the dynamics feature for the node in the mesh of the fine resolution and (ii) the node feature for the node in the mesh of the fine resolution at the current time step. The method according to any one of claims 12 to 13, comprising

15. The method according to any one of claims 1 to 14, wherein the mesh of the fine resolution and the mesh of the coarse resolution are each a three-dimensional mesh.

16. The method according to any one of claims 1 to 15, wherein the mesh of the fine resolution and the mesh of the coarse resolution are each a triangular mesh.

17. The method according to any one of claims 1 to 16, wherein the mesh of the fine resolution and the mesh of the coarse resolution each extend over the physical environment.

18. The method according to any one of claims 1 to 17, wherein for each time step, the number of nodes in the mesh of the fine resolution is greater than the number of nodes in the mesh of the coarse resolution.

19. Executed on a computing system comprising a first processor and a second processor, the second processor having a greater processing capacity or memory than the first processor, the method comprising processing data defining the mesh of the fine resolution by implementing the one or more update blocks of the fine resolution on the second processor. Processing data defining the coarse-resolution mesh by implementing the one or more coarse-resolution update blocks on the first processor The method according to any one of claims 1 to 18, comprising: Claim 20 Processing data defining the fine-resolution mesh by implementing the one or more fine-resolution update blocks on the second processor, and then Processing data defining the downsampling mesh to update the current node embedding of each node in the coarse-resolution mesh, and then Processing data defining the coarse-resolution mesh by implementing the one or more coarse-resolution update blocks on the first processor, and then Processing data defining the upsampling mesh to update the current node embedding of each node in the fine-resolution mesh The method according to claim 19 when dependent on claim 6, further comprising: Claim 21 A method of controlling a robot using the method according to any one of claims 1 to 20, comprising: The physical environment comprises a real-world environment including physical objects, The step of obtaining data defining the fine-resolution mesh and the coarse-resolution mesh each characterizing the state of the physical environment at the current time step comprises determining a representation of the location, shape, or configuration of the physical object at the current time step, The step of determining the state of the physical environment at the next time step comprises determining a predicted representation of the location, shape, or configuration of the physical object at the next time step, The method at each time step Controlling the robot using the predicted representation at the next time step to manipulate the physical object The method further comprising: Claim 22 One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of each method according to any one of claims 1 to 21. Claim 23 one or more computers, and one or more storage devices communicatively coupled to the one or more computers comprising, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of each of the methods of any one of claims 1 to 21 a system

Images