Simulation of physical environments using fine resolution and coarse resolution grids

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for simulating a state of a physical environment. In one aspect, a method performed by one or more computers for simulating a state of a physical environment is provided. The method comprises, for each of a plurality of time steps: obtaining data defining a fine resolution grid and a coarse resolution grid each characterizing a state of the physical environment at a current time step, wherein the fine resolution grid has a higher resolution than the coarse resolution grid; processing data defining a fine resolution grid and a coarse resolution grid using a graph neural network, the graph neural network comprising: (i) one or more fine resolution update blocks, (ii) one or more coarse resolution update blocks, and (iii) one or more upsampling update blocks; and determining a state of the physical environment at the next time step using the updated node embedding for the nodes in the fine resolution grid.

Description

Background Art

[0001] This specification relates to using machine learning models to process data.

[0002] A machine learning model receives an input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model parameters.

[0003] Some machine learning models are deep models that use multiple layers of the model to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention

[0004] This specification generally describes a simulation system implemented as a computer program on one or more computers in one or more locations that is capable of simulating the state of a physical environment over a series of time steps using a graph neural network. Specifically, this specification introduces a simulation system that is capable of accurately predicting (simulating) a wide range of physical environments in a high-resolution setting using a graph neural network.

[0005] Some embodiments of the techniques are adapted to specific computing hardware. For example, techniques are described that enable a mesh-based simulation to be partitioned into updates to a fine-resolution mesh and a coarse-resolution mesh that are used by the simulation system to simulate the state of the physical environment. This in turn enables the simulation system to utilize computer systems including higher and lower-capability processors, e.g., in terms of computing power such as FLOPS (floating point operations per second) or available working memory, to optimally allocate computing resources for updates to the fine-resolution and coarse-resolution meshes.

[0006] In one aspect, a method for simulating a state of a physical environment performed by one or more computers is provided. The method includes: for each time step of a plurality of time steps: obtaining data defining a fine resolution grid and a coarse resolution grid, wherein the fine resolution grid and the coarse resolution grid each characterize the state of the physical environment at the current time step, wherein the fine resolution grid has a higher resolution than the coarse resolution grid; using a graph neural network to process the data defining the fine resolution grid and the coarse resolution grid; and using updated node embeddings of nodes in the fine resolution grid to determine the state of the physical environment at the next time step. The graph neural network includes: (i) one or more fine resolution update blocks, (ii) one or more coarse resolution update blocks, and (iii) one or more upsampling update blocks. Each fine resolution update block is configured to process the data defining the fine resolution grid using a graph neural network layer to update the current node embedding of each node in the fine resolution grid. Each coarse resolution update block is configured to process the data defining the coarse resolution grid using a graph neural network layer to update the current node embedding of each node in the coarse resolution grid. Each upsampling update block is configured to: generate data defining an upsampling grid, the upsampling grid comprising: (i) each node from the fine-resolution grid and each node from the coarse-resolution grid, and (ii) a plurality of edges between the nodes of the fine-resolution grid and the nodes of the coarse-resolution grid; and process the data defining the upsampling grid using a graph neural network layer to update a current node embedding of each node in the fine-resolution grid.

[0007] In some embodiments, for each node of the coarse-resolution grid, generating an upsampled grid includes: identifying a cell of the fine-resolution grid that includes the coarse-resolution grid node; identifying one or more nodes in the fine-resolution grid, wherein the one or more nodes are vertices of the cell that includes the coarse-resolution grid node; and in the upsampled grid, instantiating corresponding edges between the nodes of the coarse-resolution grid and each identified node in the fine-resolution grid.

[0008] In some implementations, the method further includes, for each edge in the upsampling grid: generating an edge embedding for the edge based on a distance between a pair of nodes in the upsampling grid connected by the edge.

[0009] In some embodiments, using a graph neural network layer to process data defining an upsampled grid to update a current node embedding for each node in the fine resolution grid includes: updating an edge embedding for each edge in the upsampled grid based on: (i) the edge embedding of the edge, and (ii) corresponding node embeddings of a first node in the coarse resolution grid and a second node in the fine resolution grid connected by the edge; and updating a node embedding for each node in the fine resolution grid based on: (i) the node embedding of the node in the fine resolution grid, and (ii) the corresponding edge embedding of each edge connecting the node in the fine resolution grid to the corresponding node in the coarse resolution grid.

[0010] In some implementations, each upsampling block updates current node embeddings for nodes in the fine-resolution grid based at least in part on the current node embeddings for nodes in the coarse-resolution grid.

[0011] In some embodiments, the graph neural network further includes one or more downsampling update blocks. Each downsampling update block is configured to: generate data defining a downsampling grid, the downsampling grid including: (i) each node from the fine resolution grid and each node from the coarse resolution grid, and (ii) a plurality of edges between the nodes of the fine resolution grid and the nodes of the coarse resolution grid; and use the graph neural network layer to process the data defining the downsampling grid to update the current node embedding of each node in the coarse resolution grid.

[0012] In some embodiments, generating a downsampled mesh includes, for each node of the fine resolution mesh: identifying a unit cell of the coarse resolution mesh that includes the node of the fine resolution mesh; identifying one or more nodes in the coarse resolution mesh that are vertices of the unit cell that includes the node of the fine resolution mesh; and, in the downsampled mesh, instantiating corresponding edges between the node of the fine resolution mesh and each identified node of the coarse resolution mesh.

[0013] In some implementations, the method further includes, for each edge in the down-sampling grid: generating an edge embedding for the edge based on a distance between a pair of nodes connected by the edge in the down-sampling grid.

[0014] In some embodiments, using a graph neural network layer to process data defining a downsampled grid to update a current node embedding for each node in the coarse resolution grid includes: updating an edge embedding for each edge in the downsampled grid based on: (i) the edge embedding of the edge, and (ii) corresponding node embeddings of a first node in the coarse resolution grid and a second node in the fine resolution grid connected by the edge; and updating the node embedding for each node in the coarse resolution grid based on: (i) the node embedding of the node in the coarse resolution grid, and (ii) the corresponding edge embedding of each edge connecting the node in the coarse resolution grid to the corresponding node in the fine resolution grid.

[0015] In some implementations, each downsampling block updates current node embeddings of nodes in the coarse-resolution grid based at least in part on current node embeddings of nodes in the fine-resolution grid.

[0016] In some embodiments, the graph neural network has been trained on a set of training examples, wherein one or more of the training examples are generated by operations including: generating a target simulation of a state of a training physical environment at one or more time steps using a simulation engine, wherein the target simulation has a higher resolution than a fine resolution grid processed by the graph neural network; generating a lower resolution version of the target simulation by interpolating the target simulation to the same resolution as the fine resolution grid processed by the graph neural network; and generating training examples using the lower resolution version of the simulation grid.

[0017] In some embodiments, obtaining data defining a state of the physical environment at a current time step includes, for each node in the fine resolution grid: obtaining one or more node features of the node, wherein the node corresponds to a location in the physical environment, and wherein the node features characterize the state of the corresponding location in the physical environment; and processing the node features using one or more neural network layers of a graph neural network to generate a current embedding of the node.

[0018] In some embodiments, for each node in the fine resolution grid, the node characteristic of the node includes one or more of the following: a fluid density characteristic, a fluid viscosity characteristic, a pressure characteristic, or a tension characteristic.

[0019] In some embodiments, the graph neural network further includes a decoder block, wherein determining the state of the physical environment at the next time step includes: processing the updated node embedding of each node in the fine resolution grid to generate one or more corresponding dynamic features corresponding to each node in the fine resolution grid; and determining the state of the physical environment at the next time step based on: (i) the dynamic features of the nodes in the fine resolution grid, and (ii) the node features of the nodes in the fine resolution grid at the current time step.

[0020] In some embodiments, the fine resolution grid and the coarse resolution grid are each three-dimensional grids.

[0021] In some implementations, the fine-resolution mesh and the coarse-resolution mesh are each triangular meshes.

[0022] In some implementations, the fine resolution grid and the coarse resolution grid each span the physical environment.

[0023] In some implementations, for each time step, the number of nodes in the fine resolution grid is greater than the number of nodes in the coarse resolution grid.

[0024] In some embodiments, the method is performed on a computing system including a first processor and a second processor, wherein the second processor has a higher processing power or memory than the first processor. The method includes: processing data defining a fine resolution grid by implementing one or more fine resolution update blocks on the second processor; and processing data defining a coarse resolution grid by implementing one or more coarse resolution update blocks on the first processor.

[0025] In some embodiments, the method further includes: processing data defining a fine resolution grid by implementing one or more fine resolution update blocks on a second processor; then processing data defining a downsampled grid to update a current node embedding of each node in a coarse resolution grid; then processing data defining a coarse resolution grid by implementing one or more coarse resolution update blocks on a first processor; then processing data defining an upsampled grid to update a current node embedding of each node in the fine resolution grid.

[0026] In a second aspect, a method for controlling a robot using any of the above-mentioned methods is provided. The physical environment includes a real-world environment, which includes physical objects. Data defining a fine resolution grid and a coarse resolution grid are obtained, and the fine resolution grid and the coarse resolution grid each characterize the state of the physical environment at a current time step, including: determining a representation of the position, shape or configuration of the physical object at the current time step. Determining the state of the physical environment at the next time step includes: determining a predicted representation of the position, shape or configuration of the physical object at the next time step. The method further includes at each time step: using the predicted representation at the next time step to control the robot to manipulate the physical object.

[0027] In a third aspect, a system is provided. The system includes one or more non-transitory computer storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of any one of the above-mentioned methods.

[0028] In a fourth aspect, a system is provided. The system includes: one or more computers; and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions, which, when executed by the one or more computers, cause the one or more computers to perform any of the operations in the above-mentioned methods.

[0029] Graph neural networks use message passing between nodes to propagate information and iteratively update their node embeddings by exchanging information with neighboring nodes. However, this structure becomes a limiting factor for high-resolution simulations because points that are equally distant in space become farther apart in graph space. To address this issue, the simulation system is able to train graph neural networks to learn accurate alternative dynamics of high-resolution physical environments on lower-resolution grids, eliminating the message passing bottleneck and improving performance. In addition, the simulation system also introduces a hierarchical approach by passing messages on two grids with different resolutions - a fine resolution grid and a coarse resolution grid - which significantly improves the accuracy of the graph neural network while requiring fewer computational resources.

[0030] The physical environment can be, for example, a continuous field or a deformable material. A continuous field can refer, for example, to a region of space, wherein each position in the region of space is associated with one or more physical quantities, such as velocity, pressure, and the like.

[0031] A "mesh" refers to a data structure comprising a set of nodes and a set of edges, wherein each edge connects a corresponding pair of nodes. A mesh can define an irregular (unstructured) grid that specifies the tessellation of a geometric domain (e.g., a surface or space) into smaller elements (e.g., unit cells or partitions) of a particular shape (e.g., a triangle or tetrahedron shape). Each node can be associated with a corresponding spatial location in a physical environment.

[0032] The "resolution" of a mesh can refer to, for example, the number of nodes in the mesh and / or the density of nodes in the mesh. The node density of a mesh can refer to the number of nodes per unit length if the mesh is one-dimensional, the number of nodes per unit area if the mesh is two-dimensional, the number of nodes per unit volume if the mesh is three-dimensional, etc.

[0033] At each time step, the simulation system generates an initial node embedding for each node of the fine-resolution grid and the coarse-resolution grid, and then uses the update block of the graph neural network to repeatedly update the node embeddings of the nodes of the fine-resolution grid and the coarse-resolution grid. Specifically, each update block of the graph neural network receives a fine-resolution grid and / or a coarse-resolution grid, updates the current node embeddings of the nodes of the fine-resolution grid or the coarse-resolution grid, and then provides the fine-resolution grid and / or the coarse-resolution grid to the next update block in the graph neural network.

[0034] Throughout this specification, an "embedding" of an entity can refer to a representation of the entity in a latent space (e.g., a lower dimensional space) as an ordered set of numerical values, e.g., a numerical vector or matrix. An embedding of an entity can be generated, for example, as the output of a neural network processing data representing the entity. Note that, depending on the context, an embedding of an entity is often referred to as a latent representation of the entity, an encoded representation of the entity, or a feature vector representation of the entity.

[0035] The simulations generated by the simulation system described in this specification (e.g., which characterize the predicted state of the physical environment over a series of time steps) can be used for various purposes. In some cases, a visual representation of the simulation can be generated, such as a video, and provided to a user of the simulation system. In some cases, the representation of the simulation can be processed to determine that the feasibility criteria are met, and a physical device or system can be constructed in response to meeting the feasibility criteria. For example, a simulation system can generate an aerodynamic simulation of the airflow on an aircraft wing, and the feasibility criteria for physically constructing the aircraft wing can be that the force or stress on the aircraft wing does not exceed a threshold. In some cases, an agent (e.g., a reinforcement learning agent or a robotic agent) that interacts with a physical environment can use a simulation system to generate one or more simulations of the environment, which simulate the effects of the agent performing various actions in the environment. In these cases, the agent can use a simulation of the physical environment as part of determining whether to perform certain actions in the environment.

[0036] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0037] Realistic simulators of complex physics are invaluable to many scientific and engineering disciplines. However, the creation and use costs of conventional simulators can be prohibitively high. Building a conventional simulator may require years of engineering work, and versatility and accuracy must usually be weighed within a narrow range of settings. In addition, high-quality simulators typically require a large amount of computing resources, which makes expansion difficult or infeasible. Compared to some conventional simulators, the simulation system described in this specification can generate simulations of complex physical environments with higher accuracy over a large number of time steps, and use less computing resources (e.g., memory and computing power). In some cases, the simulation system can generate simulations one or more orders of magnitude faster than conventional simulators. For example, the simulation system can predict the state of the physical environment at the next time step through a graph neural network in a single pass, while a conventional simulator may need to perform separate optimizations at each time step.

[0038] The simulation system generates simulations using a graph neural network that is able to learn to simulate complex physics directly from training data, and is able to generalize the implicitly learned physical principles to accurately simulate a wider range of physical environments under different conditions than those directly represented in the training data. This also allows the system to generalize to settings that are larger and more complex than those used in training. In contrast, some conventional simulators require explicitly programmed physical principles and must be manually tuned to the specific characteristics of each environment being simulated.

[0039] The simulation system can perform a grid-based simulation, e.g., where the state of the physical environment at each time step is represented by a grid. Performing a grid-based simulation can enable the simulation system to simulate certain physical environments more accurately than would otherwise be possible, e.g., physical environments including deforming surfaces or volumes that are difficult to model as clouds of incoherent particles.

[0040] However, generating accurate grid-based simulations of physical environment states may require increasing the resolution of the grid. As the grid resolution increases, nodes in the grid that are the same distance apart in the physical environment reference system are separated by a larger distance in the grid reference system. (The distance between two nodes in the grid reference system characterizes, for example, the minimum number of edges that separate the two nodes in the grid). Therefore, if a simulation is performed by processing the grid using a graph neural network, increasing the resolution of the grid requires the graph neural network to include a greater number of graph neural network layers in order to propagate information over the same physical distance. However, increasing the number of graph neural network layers increases the consumption of graph neural network computing resources and increases the likelihood that the graph neural network will over-smooth the node embeddings associated with nodes in the grid (which may result in lower simulation accuracy).

[0041] The simulation system described in this specification solves this problem by simulating the state of the physical environment using a fine resolution grid and a coarse resolution grid, i.e., where the fine resolution grid has a higher resolution than the coarse resolution grid. The higher resolution of the fine resolution grid can simulate local effects in the physical environment with high accuracy. For example, when the coarse resolution grid is processed using a graph neural network layer, the lower resolution of the coarse resolution grid enables information to be shared between distant nodes in the coarse resolution grid. The simulation system takes advantage of the complementary advantages of the fine resolution grid and the coarse resolution grid by enabling information sharing along the edges connecting nodes in the fine resolution grid to nodes in the coarse resolution grid. Therefore, by using both the fine resolution grid and the coarse resolution grid to simulate the state of the physical environment, the simulation system can significantly improve the accuracy of the simulation while reducing the use of computing resources.

[0042] The simulation system can train a graph neural network for performing grid-based simulations on a set of training data. To generate the training data, the simulation system can use a simulation engine (e.g., a physics engine) to simulate the state of a physical environment at a higher resolution than the fine-resolution grid processed by the graph neural network. The simulation system can then generate a lower resolution version of the simulation by interpolating the simulation to the resolution of the fine-resolution network processed by the graph neural network, and generate training data based on the lower resolution version of the simulation. Generating training data in this manner can improve the accuracy of the training data, thereby enabling the graph neural network trained on the training data to achieve higher simulation accuracy.

[0043] The details of one or more embodiments of the subject matter of the present specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A is a block diagram of an example simulation system that can use a graph neural network to simulate the state of a physical environment.

[0045] Figure 1B is a schematic diagram of example fine-resolution and coarse-resolution grids representing the state of a physical environment.

[0046] Figure 2A is a schematic diagram illustrating the operation of an example fine resolution update block.

[0047] Figure 2B is a schematic diagram illustrating the operation of an example coarse resolution update block.

[0048] Figure 2C is a schematic diagram illustrating the operation of an example upsampling update block.

[0049] Figure 2D is a schematic diagram illustrating the operation of an example downsampling update block.

[0050] Figure 3A and 3B is a block diagram of an example updater module topology using different update block sequences.

[0051] Figure 4 is a schematic diagram showing examples of a low-resolution simulation, a high-resolution simulation, and a low-resolution version of the high-resolution simulation.

[0052] Figure 5 is a flowchart of an example process of using a graph neural network to simulate the state of a physical environment.

[0053] Fig. 6A and 6Bis a plot of experimental data showing the relationship between the mean square error and the minimum edge length for two simulated systems using different updater module topologies.

[0054] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0055] Using learned simulators to replace expensive traditional numerical solvers can be advantageous, for example, because learned simulators have the potential to be faster than classical methods. In addition, learned simulators can be differentiable by construction, which opens up interesting avenues for inverse engineering. A recent approach to learning simulation discretized on unstructured grids is MeshGraphNets (e.g., see Pfaff, T., et al. “Learning mesh-based simulation with graph networks,” 9th International Conference on Learning Representations, 2021), which encodes the simulation mesh at each time step as a graph and makes predictions on this graph using a message passing graph neural network (e.g., see Scarselli, F., et al. “The graph neural network model,” IEEE Transactions on Neural Networks, 20(1):61–80, 2008). MeshGraphNets show strong generalization and accurate predictions on a wide range of physical systems.

[0056] The accuracy of conventional solvers is often limited by the resolution of the simulation grid. This is especially true for chaotic systems such as fluid dynamics: processes with very small length scales such as turbulent mixing affect the overall flow and often need to be resolved on very fine grids to accurately solve the underlying partial differential equations (PDEs). This leads to a characteristic spatial convergence; where the simulation accuracy increases monotonically with the grid resolution. This is an important property of numerical solvers used in practice, as it allows the desired solution accuracy to be obtained at the expense of computational effort.

[0057] However, similar phenomena may not apply to learned simulation methods, especially graph neural network models like MeshGraphNets, for example, because as the mesh becomes finer, the message passing graph neural network performs more update steps to propagate information along the same physical distance. This leads to significantly higher computational cost, reduced accuracy at high resolutions, and may also cause over-smoothing.

[0058] To address these issues, this specification introduces a simulation system that implements a hierarchical framework for learning grid-based simulations using graph neural networks, which runs message passing at two different resolutions. That is, the simulation system implements message passing on a coarse-resolution grid that promotes information propagation and a fine-resolution grid. The simulation system recovers the spatial convergence of the graph neural network model (see, for example, Fig. 6A ) and is more accurate and computationally efficient than traditional methods (see, for example, Figure 6B ). In addition, the simulation system modifies the training distribution to use high-accuracy predictions that better capture the dynamics of the physical environment being simulated (see, for example, Figure 4 ). This allows the simulation system to make better predictions at a given resolution than a reference simulation engine (e.g., a physics engine), as opposed to replicating the spatial convergence curves of a traditional solver. Together, these approaches improve the accuracy of high-resolution simulations at a lower computational cost.

[0059] These and other features are described in greater detail below.

[0060] Figure 1A An example simulation system 100 is shown that is capable of simulating states of a physical environment using a graph neural network 150. The simulation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0061] A "physical environment" can refer to any type of physical system, including, for example, fluids, rigid solids, deformable materials, any other type of physical system, or a combination thereof. A "simulation" of a physical environment can include a corresponding simulated state of the physical environment at each time step in a sequence of time steps. The state of the physical environment at a time step can be represented as a grid (or as multiple grids at different resolutions), such as Figure 1B 1 and described in more detail below. The state of the physical environment at an initial time step can be provided as input to the simulation system 100, for example, by a user of the simulation system 100, for example, through a user interface or application programming interface (API) available to the simulation system 100. At each time step in the sequence of time steps, the simulation system 100 can process the data defining the current state 102 of the physical environment and generate a prediction of the state 202 of the physical environment at the next time step.

[0062] While some physical environments, e.g., those including fluids, can be effectively simulated as a set of individual particles, other physical environments, e.g., those including deformable materials and complex structures, may be more difficult to simulate in the same manner. In particular, simulating such physical environments through particle representations may be computationally inefficient and prone to failure, such as producing inaccurate predictions. Instead, such physical environments can be more appropriately represented by a mesh that can span the entire physical environment, a specific region of the physical environment, or represent the corresponding surface of one or more objects in the physical environment.

[0063] The simulation system 100 can be used to simulate the dynamics of different physical environments through a grid-based representation. It should be understood that the example physical environment described below is for illustration purposes only, and the simulation system 100 can be used to simulate the state of any type of physical environment including any type of material or physical object.

[0064] To simulate such a physical environment, at each time step, the simulation system 100: processes data defining a current state 102 of the physical environment, wherein such data specifies respective current node features 104.f and 104.c of nodes in the fine-resolution and coarse-resolution grids; encodes the data into respective current node embeddings 114.f and 114.c for the nodes; sequentially updates the respective current node embeddings 114.f and 114.c to generate final updated node embeddings 134.f for the nodes in the fine-resolution grid; decodes the final updated node embeddings 134.f to generate dynamic features 144.f for the nodes in the fine-resolution grid; and predicts the next state 202 of the physical environment based on the current node features 104.f (and in some cases based on one or more previous node features) and the dynamic features 144.f for the nodes in the fine-resolution grid. Various aspects of this process will be described in more detail below.

[0065] A physical environment including, for example, continuous fields, deformable materials and / or complex structures can be represented by a grid G ​​= (V, E) - for example, an undirected graph. The grid is represented in the spatial domain of the physical environment. As defined above, where n is the dimensionality of the physical environment. The physical environment can be a one-dimensional physical environment (e.g., a spring, a linear polymer), a two-dimensional physical environment (e.g., a superfluid, a membrane), a three-dimensional physical environment (e.g., an airplane wing, a trapped ion), or in some cases a higher dimensional physical environment of more than three dimensions (e.g., ten-dimensional supergravity). A "continuous field" generally refers to a region of space associated with a physical quantity (e.g., velocity, pressure, temperature, electromagnetic field, probability amplitude, etc.) that varies continuously across the region. For example, each spatial position in a velocity field can have a specific velocity value associated with it, e.g., a direction and amplitude. As another example, each spatial position in an electromagnetic field can have specific electric and magnetic field values ​​associated with it, e.g., corresponding directions and amplitudes. A continuous field may be a real field, an imaginary field, or a complex field, depending on the problem. For example, each spatial position in the probability amplitude of an electron can have a complex value associated with it.

[0066] In general, a "grid" refers to a data structure comprising a set of nodes V and a set of edges E, wherein each edge connects a pair of nodes. A grid can define an irregular (unstructured) grid that specifies a subdivision surface of a geometric domain (e.g., a surface or space) as smaller elements (e.g., unit cells or partitions) of a particular shape, e.g., a triangle or tetrahedron shape. Each node can be associated with a corresponding spatial location in a physical environment. In some embodiments, a grid can represent a corresponding surface of one or more objects in an environment. In some embodiments, a grid can span (e.g., cover) a physical environment, e.g., if the physical environment represents a continuous field.

[0067] For ease of description, it is assumed that the physical environment evolves on a fixed grid according to Eulerian dynamics, so the simulation system 100 does not need to consider the world edges in the grid. However, the simulation system 100 can also be applied to physical environments that evolve according to Lagrangian dynamics, where, for example, the grid represents a moving and deforming surface or volume. In these cases, it is possible to W ) includes a set of world edges E WIn order to be able to model external dynamics, such as (self) collisions and contacts. For example, in an embodiment in which a grid represents one or more objects in a physical environment, the simulation system 100 can identify each pair of nodes in the grid having corresponding spatial positions, each pair of nodes being separated by a distance less than a threshold distance in world space W (e.g., in the reference frame of the physical environment), and instantiate a world edge between each corresponding pair of nodes in the grid. Specifically, the simulation system 100 can instantiate world edges between pairs of nodes that are not yet connected by edges. Representing the current state 102 of the physical environment by both edges and world edges allows the simulation system 100 to simulate interactions between a pair of nodes that are generally far apart from each other in grid space (e.g., separated by multiple other nodes and edges), but are generally close to each other in world space (e.g., have similar spatial positions in the reference frame of the physical environment). Including world edges in the grid can facilitate more efficient message passing between spatially proximal nodes. Therefore, world edges can allow for a more accurate simulation to be achieved using fewer update iterations (ie, message passing steps) in the updater module 120, thereby reducing computational resource consumption during simulation.

[0068] Each node in the grid i∈V can be associated with the current node feature f i (t k ) is associated with the current node feature f i (t k ) represents the current time step t k The location x in the physical environment corresponding to the node i The current state of the physical environment at 102. For example, in embodiments involving simulations of a physical environment with continuous fields, such as in fluid dynamics or aerodynamics simulations, the node feature f of each node i The node may include fluid viscosity, fluid density, or any other suitable physical aspect at a location in the physical environment corresponding to the node. As another example, in an embodiment involving a simulation of a physical environment with an object, such as a structural mechanics simulation, each node may represent a point on the object and may be associated with an object-specific node feature f that characterizes the point on the object. i associated with, for example, the position of a corresponding point on the object, the pressure at that point, the tension at that point, and any other appropriate physical aspect. In addition, each node can additionally be associated with a node feature f including one or more of the following: i Associated with: fluid density, fluid viscosity, pressure or tension at the location in the physical environment corresponding to the node. In general, the mesh representation is not limited to the aforementioned physical environments, and other types of physical environments can also be represented by meshes and simulated using the simulation system 100 .

[0069] In some embodiments, the node features associated with each node at the current time step can further include the node at one or more previous time steps t k-1 ,y k-2 , ..., t k-C For example, the node features associated with each node at the current time step can include the corresponding node features f characterizing the state of the node at each time step in one or more previous time steps. i (t k-1 ), f i (t k-2 ), ..., f i (t k-C ). Such an embodiment can be applicable to physical environments with memory effects (e.g., time dispersion), where the current state 102 of the physical environment depends on the convolution with the previous state of the physical environment, for example, through a response function (e.g., a convolution kernel). For example, the polarization density of an electromagnetic medium at the current time step typically depends on the electric field at multiple previous time steps through the diffuse dielectric constant. The state of a node at one or more previous time steps can also capture hidden states and / or non-reversible changes, such as plastic deformation, hysteresis. For computational fluid dynamics (CFD) and other related systems (e.g., continuous mechanical systems), a longer history of the state of the physical environment allows the graph neural network 150 to learn correction terms (similar to high-order integrators), thereby achieving more accurate predictions and / or longer time steps, for example, simulating the state of the physical environment over a longer period of time with fewer time steps.

[0070] The simulation system 100 is in its spatial domain We operate on two grid-based representations of the physical environment: (i) a fine-resolution grid G f =(V f , E f ), where V f and E f are respectively a set of nodes 11.f and a set of edges 13.f of the fine resolution grid 10.f, and (ii) a coarse resolution grid G c =(V c , E c ), where V c and E care a set of nodes 11.c and a set of edges 13.c of the coarse resolution grid 10.c, respectively. The fine resolution grid 10.f has a higher resolution than the coarse resolution grid 10.c, for example, a larger number of nodes and / or a higher node density. Typically, the coarse resolution grid 10.c is introduced by the simulation system 100 for the purpose of facilitating more efficient message passing of the graph neural network 150, for example, to efficiently model fast-acting or non-local dynamics.

[0071] The system 100 can generate the fine resolution mesh 10.f and the coarse resolution mesh 10.c using a mesh generation algorithm, such as Delaunay triangulation, Rupert's algorithm, algebraic methods, differential equation methods, variational methods, unstructured grid methods, etc. Alternatively, the simulation system 100 can first generate the fine resolution mesh 10.f using a mesh generation algorithm, and then average or interpolate the fine resolution mesh 10.f to generate the coarse resolution mesh 10.c.

[0072] Figure 1B is an illustration of an example fine resolution 10.f and coarse resolution 10.c grid representing the current state 102 of the physical environment. Note that while the fine resolution 10.f and coarse resolution 10.c grids are Figure 1B , as a two-dimensional mesh with triangular cells (eg, a Delaunay triangulation), but the fine resolution 10.f and coarse resolution 10.c meshes can generally be of any size and can have cells of any shape.

[0073] Each node i∈V in the fine-resolution grid 10.f f Is related to the current node feature 104.f associated with the current node feature 104.f represents the current time step t k The location x in the physical environment corresponding to node 11.f i The current state of the physical environment at 102. Pairs of nodes 11.f in the fine resolution grid 10.f are connected by edges 13.f forming unit cells 14.f. For illustration purposes, internal nodes are identified as black circles and boundary nodes are identified as white circles with a black outline.

[0074] Similarly, the coarse resolution grid i∈V c Each node in the 104.c is associated with the current node feature 104.c represents the current time step t k The location x in the physical environment corresponding to node 11.c iThe current state of the physical environment at the coarse resolution grid 10.c 102. Node pairs 11.c in the coarse resolution grid 10.c are connected by edges 13.c forming unit cells 14.c. For ease of illustration, internal nodes are identified as black circles and boundary nodes are identified as white circles with a black outline.

[0075] The corresponding nodes in the fine resolution grid 10.f and the coarse resolution grid 10.c do not need to overlap, so the current state 102 of the physical environment at different locations in the physical environment can be represented. In addition, since the resolution of the fine resolution grid 10.f is higher than that of the coarse resolution grid 10.c, the simulation system 100 can determine the current node features 104.c of each node in the coarse resolution grid 10.c based on the current node features 104.f of the nodes in the fine resolution grid 10.f. For example, the simulation system 100 can average or interpolate the current node features 104.f associated with the node group in the fine resolution grid 10.f to generate the current node features 104.c associated with the node in the coarse resolution grid 10.c. In some embodiments, the current node features 104.c of the coarse resolution grid 10.c only include geometric (e.g., static) features that do not change with each time step. For example, the geometric features can include node types that distinguish internal nodes from boundary nodes, for example, as a one-hot vector. As some further examples, the node type can indicate whether the node is part of a physical object, a boundary of an object, part of an actuator, part of a fluid containing the object, a wall, an inflow or outflow of a physical environment, an attachment point of an object, or another feature of the physical environment.

[0076] In some embodiments, the current node features 104.f and 104.c of the fine resolution 10.f and coarse resolution 10.c grids can also include global features 108 of the physical environment, such as representations of forces applied to the physical environment, gravitational constants of the physical environment, magnetic fields of the physical environment, or any other suitable features or combinations thereof. For example, at each time step, the simulation system 100 can link the global features 108 to the current node features 104.f and 104.c associated with each node in the fine resolution grid 10.f and each node in the coarse resolution grid 10.c before the graph neural network 150 processes the current state 102 of the physical environment.

[0077] The graph neural network 150 includes an encoder module 110 , an updater module 120 , and a decoder module 130 .

[0078] The encoder 110 includes one or more neural network layers. Specifically, the encoder 110 can include any suitable number (e.g., 5 layers, 25 layers, or 100 layers) of any suitable type (e.g., fully connected layers, convolutional layers, attention layers, etc.) of neural network layers, and connected in any suitable configuration (e.g., as a linear sequence of layers or as a directed graph of layers). By way of example only, the encoder 110 can be implemented as a multilayer perceptron (MLP) with residual connections.

[0079] At each time step, the encoder 110 processes a fine resolution grid i∈V f The current node features 104.f of each node in to generate the current node embedding for the node at time step 114.f. Similarly, at each time step, the encoder 110 processes the coarse resolution grid i∈V c The current node features 104.c of each node in the to generate the current node embedding for the node at that time step 114.c. Typically, node embeddings of nodes represent individual attributes of nodes in the latent space.

[0080] At each time step, the encoder 110 can also generate the current edge embedding for each edge in the fine-resolution grid 10.f at that time step and the current edge embedding for each edge in the coarse resolution grid 10.c Typically, an edge embedding of an edge connecting a pair of nodes in a grid represents paired attributes of the corresponding node pair in the latent space. For example, for each edge in the fine resolution grid 10.f or the coarse resolution grid 10.c, the encoder 110 can process the corresponding current node features and / or corresponding positions associated with the node pair i, j∈V connected by the edge, and generate a corresponding current edge embedding for the edge. More specifically, the encoder 110 can generate a current edge embedding for each edge in the fine resolution 10.f or the coarse resolution 10.c grid based on the following: the corresponding current node features of the nodes connected by the edge, the difference between the corresponding current node features of the nodes connected by the edge, the weighted sum of the differences between the corresponding current node features of the nodes connected by the edge, the corresponding positions of the nodes connected by the edge, the difference between the corresponding positions of the nodes connected by the edge, the magnitude of the difference between the corresponding positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof.

[0081] Updater 120 includes a series of update blocks 122, which include: (i) one or more fine resolution update blocks 122.f, (ii) one or more coarse resolution update blocks 122.c, (iii) one or more upsampling update blocks 122.u, and (iv) one or more downsampling update blocks 122.d.

[0082] At each time step, updater 120 processes the current node embeddings 114.f and 114.c using a sequence of update blocks 122 to generate the final updated node embeddings for the nodes in the fine resolution grid 10.f at that time step 134.f. Typically, the updater 120 updates the current node embeddings 114.f and 144.c multiple times at this time step to generate the final updated node embedding 134.f. Figures 2A-2D The operation of each update block 122 is described below. The update blocks 122 can be arranged in various different topologies with various numbers of blocks, for example, to achieve a specific level of prediction accuracy for a specific resolution in the fine resolution 10.f grid. Figure 3A and 3B Describe the example topology.

[0083] Decoder 130 includes one or more neural network layers. Specifically, decoder 130 can include any suitable number (e.g., 5 layers, 25 layers, or 100 layers) of any suitable type (e.g., fully connected layers, convolutional layers, attention layers, etc.) of neural network layers, and connected in any suitable configuration (e.g., as a linear sequence of layers or a directed graph of layers). By way of example only, decoder 130 can be implemented as a multilayer perceptron (MLP) with residual connections.

[0084] At each time step, the decoder 130 processes the final updated node embedding 134.f associated with each node in the fine resolution grid 10.f to generate one or more dynamic features for that node at that time step 144.f. The dynamic feature 144.f characterizes the rate of change of the current node feature 104.f associated with the node. The dynamic feature 144.f can represent the rate of change of any suitable current node feature 104.f for a node in the fine resolution grid 10.f, such as position, velocity, momentum, density, electromagnetic field, probability field, or any other suitable physical aspect.

[0085] At each time step, the prediction engine 160 can determine the node characteristics for each node in the fine resolution grid 10.f at the next time step based on: (i) the current node characteristics 104.f of the node at the current time step, and (ii) the dynamic characteristics 144.f of the node, for example, by integrating the dynamic characteristics 144.f any suitable number of times. For example, for a first order system and assuming equally spaced time steps t k+1 -t k =Δt, the prediction engine 160 can determine the node feature for the node at the next time step based on the current node feature 104.f at the current time step and the dynamic feature 144.f corresponding to the node as:

[0086]

[0087] because A first order finite difference method is followed. Prediction engine 160 can control the accuracy of such predictions at least in part by selecting appropriately spaced time steps Δt.

[0088] Similarly, for a second order system, the prediction engine 160 can determine the node features for the node at the next time step based on the current node features 104.f at the current time step, the node features at the previous time step, and the dynamic features 144.f corresponding to the node as:

[0089]

[0090] because A second order central difference method is followed. Likewise, prediction engine 160 can control the accuracy of such predictions at least in part by selecting appropriately spaced time steps Δt.

[0091] Thus, by determining the node features of all nodes in the fine resolution grid 10.f at the next time step, the simulation system 100 is able to determine the next state 202 of the physical environment. As mentioned above, the simulation system 100 is able to determine the node features of all nodes in the coarse resolution grid 10.c at the next time step by averaging or interpolating the node features associated with the nodes in the fine resolution grid 10.f at the next time step. In embodiments where the node features of the coarse resolution grid 10.c include only geometric features, the simulation system 100 does not need to update the node features of the coarse resolution grid 10.c because such features are static across time steps.

[0092] The simulation system 100 can train the graph neural network 150 based on a set of training data using supervised learning techniques. The training data includes a set of training examples, where each training example specifies: (i) a corresponding training input that can be processed by the graph neural network 150, and (ii) a corresponding target output that the graph neural network 150 is encouraged to generate by processing the training input. The training input includes at a specific time step t k The node features for each node in the fine resolution grid 10.f are trained at and train node features for each node in the coarse resolution grid 10.c In some embodiments, the training node features associated with nodes in the coarse resolution grid 10.c include only geometric features, for example, node types specifying internal or boundary nodes. The target output includes one or more target dynamic features for each node in the fine resolution grid 10.f at time step

[0093] The simulation system 100 is capable of training the graph neural network 150 in multiple training iterations. In each training iteration, the simulation system 100 samples a batch of one or more training examples from the training data and provides them to the graph neural network 150, which is capable of processing the training inputs specified in the training examples to generate corresponding outputs, which are estimates of the target outputs, i.e., the predicted dynamic characteristics of the training inputs. The simulation system 100 is capable of evaluating an objective function L that measures the similarity between: (i) the target output specified by the training examples, and (ii) the output generated by the graph neural network 150, such as a cross entropy or squared error objective function. For example, the objective function L can be based on the predicted dynamic characteristics for the nodes in the fine resolution grid 10.f. The error from the target dynamics characteristic for the node is as follows:

[0094]

[0095] Among them, d θ is a function representing the model of the graph neural network 150 and θ is a neural network parameter of the graph neural network 150. The simulation system 100 can use an objective function for each node and each time step as shown in formula (3), or average the objective function over multiple nodes and / or multiple time steps. The simulation system 100 can determine the gradient of the objective function, for example, using a back-propagation technique, and can use the gradient to update the network parameter values ​​of the graph neural network 150 to optimize the objective function, for example, using any appropriate gradient descent optimization algorithm such as Adam. The simulation system 100 can also determine a performance metric of the graph neural network 150 based on a set of validation data that was not used during training the graph neural network 150.

[0096] To generate training data, the simulation system 100 can use a simulation engine (e.g., a physics engine such as COMSOL Multiphysics from COMSOL Inc.) to simulate the state of the physical environment within one or more time steps. Specifically, the simulation system 100 simulates the state of the physical environment on a grid having a higher resolution than the fine resolution grid 10.f processed by the graph neural network 150. The simulation system 100 then generates a lower resolution version of the simulation by interpolating the simulation (e.g., bilinearly or bicubicly) to the resolution of the fine resolution grid 10.f and the coarse resolution grid 10.c to generate training data based on the lower resolution version of the simulation. Specifically, the simulation system 100 can determine the training input and target output for each training example based on the lower resolution version of the simulation. Generating training data in this manner can improve the accuracy of the training data, thereby enabling the graph neural network 150 trained on the training data to achieve higher simulation accuracy.

[0097] Figure 4 is a diagram showing an example of a low resolution simulation 410, a high resolution simulation 420, and an interpolated lower resolution version 430 of the high resolution simulation 420. The simulations are Karman vortex streets and were simulated with COMSOL. Figure 4 The grayscale in shows the x-component of the velocity field. The low-resolution simulation 410 grid is not fine enough to resolve all flow features, and the characteristic vortex shedding is suppressed. The high-resolution simulation 420 on a finer grid correctly resolves the dynamics. The high-accuracy predictions of the high-resolution simulation 420 are interpolated onto a lower-resolution version 430 of the high-resolution simulation 420 so that the vortex shedding is still visible. The lower-resolution version 430 has the same resolution as the fine-resolution grid 10.f and can be used by the simulation system 100 to generate training examples. In this way, the graph neural network 150 is able to implicitly learn the effects of a smaller scale without making any changes to the model code, and is able to achieve more accurate predictions at inference time than those based on a coarse-scale classical solver.

[0098] After training the graph neural network 150, the simulation system 100 can be used to simulate the state of different types of physical environments. For example, from a single time step prediction with hundreds or thousands of nodes during training, the simulation system 100 can effectively generalize to different types of physical environments, different initial conditions, thousands of time steps, and at least an order of magnitude more nodes.

[0099] Figure 2A1 is a diagram illustrating the operation of an example fine resolution update block 122.f, which is used by the updater 120 to perform a node embedding update on the fine resolution grid 10.f. Figure 2A As shown, each node 11.f.0 in the fine resolution grid 10.f receives information from each of the neighboring nodes 11.f.1-6 connected to the node 11.f.0 via edges.

[0100] Each fine resolution update block 122.f comprises one or more neural network layers and is configured to process the data defining the fine resolution grid 10.f to generate an updated node embedding for each node in the fine resolution grid 10.f Specifically, the one or more first neural network layers of the fine resolution update block 122.f are configured to process inputs comprising: (i) edge embeddings of edges in the fine resolution grid 10.f and (ii) the corresponding node embeddings for the node pairs connected by the edges and To generate the updated edge embedding for the edge In addition, the one or more second neural network layers of the fine resolution update block 122.f are configured to process inputs comprising: (i) node embeddings of nodes in the fine resolution grid 10.f and (ii) the corresponding updated edge embeddings for each edge connected to the node To generate an updated node embedding for the node For example, the fine resolution update block 122.f can generate updated node embeddings as:

[0101]

[0102] Among them, F f and S f Respectively represent the operations of the one or more first neural network layers and the one or more second neural network layers of the fine resolution update block 122.f. For example, the one or more first neural network layers and the one or more second neural network layers of the fine resolution update block 122.f can each include a corresponding multi-layer perceptron (MLP) with residual connections.

[0103] Each fine resolution update block 122.f can be a message passing block with a different set of network parameters. That is, each fine resolution update block 122.f can be identical to each other, i.e., have the same neural network architecture, but have separate neural network parameter sets. Alternatively, the updater 120 can implement a single fine resolution update block 122.f as a message passing block and call that block 122.f one or more times when implementing a single fine resolution update block 112.f in the update block 122 sequence.

[0104] Figure 2B 1 is a diagram illustrating the operation of an example coarse resolution update block 122.c, which is used by the updater 120 to perform a node embedding update on the coarse resolution mesh 10.c. Figure 2B As shown, each node 11.c.0 in the coarse resolution grid 10.c receives information from each of the neighboring nodes 11.c.1-5 connected to the node 11.c.0 by an edge.

[0105] Each coarse resolution update block 122.c comprises one or more neural network layers and is configured to process data defining the coarse resolution grid 10.c to generate an updated node embedding for each node in the coarse resolution grid 10.c. Specifically, the one or more first neural network layers of the coarse resolution update block 122.c are configured to process inputs comprising: (i) edge embeddings of edges in the coarse resolution grid 10.c and (ii) the corresponding node embeddings for the node pairs connected by the edges and To update the edge embedding for the edge In addition, the one or more second neural network layers of the coarse resolution update block 122.c are configured to process inputs comprising: (i) node embeddings of nodes in the coarse resolution grid 10.c and (ii) the corresponding updated edge embeddings for each edge connected to the node To generate an updated node embedding for the node For example, the coarse resolution update block 122.c can generate updated node embeddings as:

[0106]

[0107] Among them, F c and S cRespectively represent the operation of one or more first neural network layers and one or more second neural network layers of the coarse resolution update block 122.c. For example, the one or more first neural network layers and one or more second neural network layers of the coarse resolution update block 122.c can each include a corresponding multi-layer perceptron (MLP) with residual connections.

[0108] Each coarse resolution update block 122.c can be a message passing block with a different set of network parameters. That is, each coarse resolution update block 122.c can be identical to each other, i.e., have the same neural network architecture, but have separate neural network parameter sets. Alternatively, the updater 120 can use a single coarse resolution update block 122.c as a message passing block and call the block 122.c one or more times when implementing the single coarse resolution update block 122.c in the update block 122 sequence.

[0109] Figure 2C 1 is a diagram illustrating the operation of an example upsampling update block 122.u, which is used by the updater 120 to perform a node embedding update on the fine resolution grid 10.f using information on the coarse resolution grid 10.c. Figure 2C As shown, each node 11.f in the fine resolution mesh 10.f receives information from each node 11.c.1-3 in the coarse resolution mesh 10.c that is a vertex of a cell 14.c surrounding the node 11.f.

[0110] Each upsampling update block 122.u is configured to generate a grid G ​​defining an upsampling grid. u =(V u , E u ) data. A set of nodes V of the upsampling grid u =V f ∪V c Includes every node from the fine resolution grid 10.f and every node from the coarse resolution grid 10.c. The set of edges E of the upsampled grid u Included are edges between nodes of the fine resolution grid 10.f and nodes of the coarse resolution grid 10.c. Typically, the upsampling update block 122.u uses the edges of the upsampling grid to transfer information from nodes in the coarse resolution grid 10.c to nodes in the fine resolution grid 10.f.

[0111] The upsampling update block 122.u can generate the edges of the upsampling grid as follows. c For each node in V, the upsampling update block 122.u identifies a cell of the fine resolution grid 10.f that includes the node of the coarse resolution grid 10.c. The upsampling update block 122.u identifies the cell j=j(i)∈V asf Then, the upsampling update block 122.u instantiates a corresponding edge k in the upsampling mesh between a node of the coarse resolution mesh 10.c and each identified node in the fine resolution mesh 10.f. ij ∈E u The upsampling update block 122.u then generates an edge embedding for each edge in the upsampling grid based on, for example, the corresponding positions of the nodes connected by the edge, the difference between the corresponding positions of the nodes connected by the edge, the magnitude of the difference between the corresponding positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof.

[0112] Each upsampling update block 122.u comprises one or more neural network layers and is configured to process the data defining the upsampling grid to generate an updated node embedding for each node in the fine resolution grid 10.f Specifically, the one or more first neural network layers of the upsampling update block 122.u are configured to process an input comprising: (i) edge embeddings of edges in the upsampling grid and (ii) the corresponding node embeddings of the first node in the coarse resolution grid 10.c and the second node in the fine resolution grid 10.f connected by the edge and To update the edge embedding for this edge In addition, the one or more second neural network layers of the upsampling update block 122.u are configured to process inputs comprising: (i) node embeddings of nodes in the fine resolution grid 10.f and (ii) the corresponding updated edge embeddings for each edge in the upsampled grid connected to the node To generate an updated node embedding for the node For example, the upsampling update block 122.u can generate updated node embeddings as:

[0113]

[0114] Among them, F u and S u Respectively represent the operations of the one or more first neural network layers and the one or more second neural network layers of the upsampling update block 122.u. For example, the one or more first neural network layers and the one or more second neural network layers of the upsampling update block 122.u can each include a corresponding multi-layer perceptron (MLP) with residual connections.

[0115] Each upsampling update block 122.u can be a message passing block with a different set of network parameters. That is, each upsampling update block 122.u can be identical to each other, i.e., have the same neural network architecture, but have separate neural network parameter sets. Alternatively, the updater 120 can implement a single upsampling update block 122.u as a message passing block and call the block 122.u one or more times when implementing the single upsampling update block 122.u in the update block 122 sequence.

[0116] Figure 2D 1 is a diagram illustrating the operation of an example downsampling update block 122.d, which is used by the updater 120 to perform a node embedding update on the coarse resolution grid 10.c using information on the fine resolution grid 10.f. Figure 2D As shown, each node 11.c in the coarse resolution mesh 10.c receives information from each node 11.f.1-3 in the fine resolution mesh 10.f that is a vertex of a cell 14.f surrounding the node 11.c.

[0117] Each downsampling update block 122.d is configured to generate a downsampling grid G ​​defined by d =(V d , E d ) data. A set of nodes V of the down-sampled grid d =V f ∪V c Includes every node from the fine resolution grid 10.f and every node from the coarse resolution grid 10.c. The set of edges E of the downsampled grid d Included are edges between nodes of the fine resolution grid 10.f and nodes of the coarse resolution grid 10.c. In general, the downsampling update block 122.d uses the edges of the downsampling grid to transfer information from nodes in the fine resolution grid 10.f to nodes in the coarse resolution grid 10.c.

[0118] The downsampling update block 122.d can generate the edges of the downsampling grid as follows. f For each node in , the downsampling update block 122.d identifies a unit cell in the coarse resolution grid 10.c that includes the node of the fine resolution grid 10.f. The downsampling update block 122.d identifies a unit cell j=j(i)∈V c Then, the downsampling update block 122.d instantiates a corresponding edge k in the downsampling mesh between a node in the fine resolution mesh 10.f and each identified node in the coarse resolution mesh 10.c. ij ∈E dThe downsampling update block 122.d then generates an edge embedding for each edge in the downsampling grid based on, for example, the corresponding positions of the nodes connected by the edge, the difference between the corresponding positions of the nodes connected by the edge, the magnitude of the difference between the corresponding positions of the nodes connected by the edge (e.g., the distance between the nodes connected by the edge), or a combination thereof.

[0119] Each downsampling update block 122.d comprises one or more neural network layers and is configured to process the data defining the downsampling grid to generate an updated node embedding for each node in the coarse resolution grid 10.c Specifically, the one or more first neural network layers of the downsampling update block 122.d are configured to process an input comprising: (i) edge embeddings of edges in the downsampling grid; and (ii) the corresponding node embeddings of the first node in the fine resolution grid 10.f and the second node in the coarse resolution grid 10.c connected by an edge and To generate the updated edge embedding for the edge In addition, the one or more second neural network layers of the downsampling update block 122.d are configured to process inputs comprising: (i) node embeddings of nodes in the coarse resolution grid 10.c and (ii) the corresponding updated edge embeddings for each edge in the downsampled grid connected to the node To generate an updated node embedding for the node For example, the downsampling update block 122.d can generate updated node embeddings as:

[0120]

[0121] Among them, F d and S d Respectively represent the operations of the one or more first neural network layers and the one or more second neural network layers of the downsampling update block 122.d. For example, the one or more first neural network layers and the one or more second neural network layers of the downsampling update block 122.d can each include a corresponding multilayer perceptron (MLP) with residual connections.

[0122] Each down-sampling update block 122.d can be a message passing block with a different set of network parameters. That is, each down-sampling update block 122.d can be identical to each other, i.e., have the same neural network architecture, but have separate neural network parameter sets. Alternatively, the updater 120 can use a single down-sampling update block 122.d as a message passing block and call the block 122.d one or more times when implementing the single down-sampling update block 122.d in the update block 122 sequence.

[0123] Figure 3A and 3B 10.c is a block diagram of an example updater module 120 topology for updating node embeddings for nodes in a fine resolution grid 10.f and a coarse resolution grid 10.c using different update block 122 sequences. Updates on the fine resolution grid 10.f are indicated by solid arrows, while updates on the coarse resolution grid 10.c are indicated by dashed arrows. The topology allows the updater 120 to perform efficient message passing. Specifically, the coarse resolution update block 122.c is significantly faster than the fine resolution update block 122.f because the number of nodes and edges on the coarse resolution grid 10.c is smaller than the fine resolution grid 10.f. The coarse resolution update block 122.c is also able to propagate information further on the coarse resolution grid 10.c.

[0124] Thus, the updater 120 is able to implement an efficient update scheme by performing a small number (e.g., 1 to 4) updates to the fine resolution grid 10.f using a small number (e.g., 1 to 4) of fine resolution update blocks 122.f to aggregate local features, downsampling the coarse resolution grid 10.f using a downsampling update block 122.d, performing a large number (e.g., 10 to 100) updates to the coarse resolution grid 10.c using a large number (e.g., 10 to 100) of coarse resolution update blocks 122.c, upsampling the fine resolution grid 10.f using an upsampling update block 122.u, and computing small-scale dynamics by performing a small number (e.g., 1 to 4) of updates to the fine resolution grid 10.f using a small number (e.g., 1 to 4) of fine resolution update blocks 122.f. Such an update scheme is referred to as a "block loop" in which the updater 120 performs updates to the fine resolution grid 10.f, performs downsampling, performs updates to the coarse resolution grid 10.c, and then performs upsampling back to the fine resolution grid 10.f. Updater 120 can perform any number of such block cycles, as described below.

[0125] exist Figure 3A, the updater 120 uses a sequence of N+4 update blocks 122 that implements a single block cycle. In this case, a first fine resolution update block 122.f.1 is followed by a down-sampled update block 122.d, a sequence of multiple (N) coarse resolution update blocks 122.f.1-N, an up-sampled update block 122.u, and a second fine resolution update block 122.f.2. The sequence of update blocks 122 can be collectively referred to as "fd-Nc-uf", where "f" represents a fine resolution update block 122.f, "c" represents a coarse resolution update block 122.c, "u" represents an up-sampled update block 122.u, and "d" represents a down-sampled update block 122.d.

[0126] exist Figure 3B In , the updater 120 uses a sequence of 11 update blocks 122 that implements two block cycles. In this case, the sequence of update blocks 122 can be represented as "fd-2c-ufd-2c-uf".

[0127] Figure 5 is a flow chart of an example process for simulating the state of a physical environment using a graph neural network. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a simulation system appropriately programmed according to the present specification, such as Figure 1A The simulation system 100 is capable of performing the process 500.

[0128] For each time step in the plurality of time steps, the simulation system performs the following operations.

[0129] The simulation system obtains data defining a fine resolution grid and a coarse resolution grid, each of which represents a state of a physical environment at a current time step (502). The fine resolution grid and the coarse resolution grid each have a corresponding set of nodes and edges, which can span a physical environment, a region of a physical environment, or represent one or more objects in the physical environment. The fine resolution grid has a higher resolution than the coarse resolution grid, for example, the fine resolution grid has more nodes than the coarse resolution grid. The grid can be a one-dimensional grid, a two-dimensional grid, a three-dimensional grid, or a grid of three or more dimensions. In some embodiments, the grid is a triangular grid, i.e., has triangular unit cells. The data defining the fine resolution grid and the coarse resolution grid at the current time step include current node embeddings for nodes in the fine resolution grid and current node embeddings for nodes in the coarse resolution grid. The data can also include current edge embeddings for edges in the fine resolution grid and current edge embeddings for edges in the coarse resolution grid.

[0130] The simulation system can obtain data defining a fine resolution grid by obtaining one or more current node features for each node in the fine resolution grid, wherein the one or more current node features characterize the state of the physical environment at a location in the physical environment corresponding to the node. For example, the node features at an initial time step can be provided by a user, such as through an API, and then the simulation system can execute process 500 to obtain the node features for each subsequent time step. In some embodiments, the node features include one or more of the following: fluid density, fluid viscosity, pressure, or tension at a location in the physical environment corresponding to the node at the current time step. The simulation system can then use an encoder module of a graph neural network to process one or more node features of each node in the fine resolution grid to generate a current node embedding for the node. The simulation system can also use an encoder module to generate a current edge embedding for each edge in the fine resolution grid based on pairs of current node features and / or corresponding positions of nodes connected to the edge.

[0131] The simulation system can obtain data defining a coarse resolution grid in a similar manner. In some embodiments, the current node features of the nodes in the coarse resolution grid are averaged and / or interpolated from the current node features of the nodes in the fine resolution grid. In some embodiments, the current node features of the nodes in the coarse resolution include only geometric (e.g., static) features that do not change with each time step. For example, the geometric features can include node types that specify internal nodes or boundary nodes. In these cases, the simulation system can reuse the node features of the nodes in the coarse resolution grid from the previous time step.

[0132] The simulation system uses an updater module of the graph neural network to process data defining the fine resolution grid and the coarse resolution grid to update current node embeddings for nodes in the fine resolution grid (505).

[0133] The updater module includes: (i) one or more fine resolution update blocks, (ii) one or more coarse resolution update blocks, (iii) one or more up-sampling update blocks, and (iv) one or more down-sampling update blocks. The updater module can implement various different update block sequences, for example, in the form of one or more block loops. For example, to implement the block loop, the updater module can include a sequence of one or more fine resolution update blocks, a down-sampling update block, one or more coarse resolution update blocks, and an up-sampling update block.

[0134] Each fine-resolution update block is configured to process data defining a fine-resolution grid using a graph neural network layer to update a current node embedding of each node in the fine-resolution grid. For example, the fine-resolution update block can update an edge embedding of each edge in the fine-resolution grid based on: (i) the edge embedding of the edge, and (ii) the corresponding node embeddings of the nodes in the fine-resolution grid connected by the edge. The fine-resolution update block can then update the node embedding of each node in the fine-resolution grid based on: (i) the node embedding of the node in the fine-resolution grid, and (ii) the corresponding edge embedding of each edge connected to the node.

[0135] Each coarse resolution update block is configured to process data defining a coarse resolution grid using a graph neural network layer to update a current node embedding of each node in the coarse resolution grid. For example, the coarse resolution update block can update an edge embedding of each edge in the coarse resolution grid based on: (i) the edge embedding of the edge, and (ii) the corresponding node embeddings of the nodes in the coarse resolution grid connected by the edge. The coarse resolution update block can then update the node embedding of each node in the coarse resolution grid based on: (i) the node embedding of the node in the coarse resolution grid, and (ii) the corresponding edge embedding of each edge connected to the node.

[0136] Each upsampling update block is configured to generate data defining an upsampling grid. The upsampling grid includes: (i) each node from the fine resolution grid and each node from the coarse resolution grid, and (ii) multiple edges between the nodes of the fine resolution grid and the nodes of the coarse resolution grid. For example, for each node in the coarse resolution grid, the upsampling update block can identify a unit cell of the fine resolution grid that includes the node of the coarse resolution grid. Then, the upsampling update block can identify one or more nodes in the fine resolution grid that are vertices of the unit cell that includes the node of the coarse resolution grid. Then, the upsampling update block can instantiate corresponding edges between the nodes of the coarse resolution grid and each identified node in the fine resolution grid in the upsampling grid. Then, the upsampling update block generates an edge embedding for each edge in the upsampling grid based on the corresponding positions between a pair of nodes connected by the edge in the upsampling grid, for example, the distance between a pair of nodes connected by the edge in the upsampling grid.

[0137] Each upsampling update block is further configured to process the data defining the upsampling grid using a graph neural network layer to update the current node embedding of each node in the fine-resolution grid. For example, the upsampling update block can update the edge embedding of each edge in the upsampling grid based on: (i) the edge embedding of the edge, and (ii) the corresponding node embeddings of a first node in the coarse-resolution grid and a second node in the fine-resolution grid connected by the edge. Then, the upsampling update block can update the node embedding of each node in the fine-resolution grid based on: (i) the node embedding of the node in the fine-resolution grid, and (ii) the corresponding edge embedding of each edge connecting the node in the fine-resolution grid to the corresponding node in the coarse-resolution grid.

[0138] Each downsampling update block is configured to generate data defining a downsampling grid. The downsampling grid includes: (i) each node from the fine resolution grid and each node from the coarse resolution grid, and (ii) multiple edges between the nodes of the fine resolution grid and the nodes of the coarse resolution grid. For example, for each node of the fine resolution grid, the downsampling update block can identify a unit cell of the coarse resolution grid that includes the node of the fine resolution grid. Then, the downsampling update block can identify one or more nodes of the coarse resolution grid that are vertices of the unit cell that includes the node of the fine resolution grid. Then, the downsampling update block can instantiate corresponding edges between the nodes of the fine resolution grid and each identified node of the coarse resolution grid in the downsampling grid. Then, the downsampling update block generates an edge embedding for each edge in the downsampling grid based on the corresponding positions between a pair of nodes connected by the edge in the downsampling grid, for example, the distance between a pair of nodes connected by the edge in the downsampling grid.

[0139] Each downsampling update block is further configured to process the data defining the downsampling grid using a graph neural network layer to update the current node embedding of each node in the coarse resolution grid. For example, the downsampling update block can update the edge embedding of each edge in the downsampling grid based on: (i) the edge embedding of the edge, and (ii) the corresponding node embeddings of a first node in the coarse resolution grid and a second node in the fine resolution grid connected by the edge. The downsampling block can then update the node embedding of each node in the coarse resolution grid based on: (i) the node embedding of the node in the coarse resolution grid, and (ii) the corresponding edge embedding of each edge connecting the node in the coarse resolution grid to the corresponding node in the fine resolution grid.

[0140] The simulation system uses the updated node embeddings of the nodes in the fine resolution grid to determine the state of the physical environment at the next time step (506). For example, the simulation system can use a decoder module to process the updated node embeddings of each node in the fine resolution grid to generate one or more corresponding dynamic features corresponding to each node in the fine resolution grid. The simulation system can then use a prediction engine to determine the state of the physical environment at the next time step based on: (i) the dynamic features of the nodes in the fine resolution grid, and (ii) the node features of the nodes in the fine resolution grid at the current time step.

[0141] Typically, a graph neural network has been trained on a set of training examples to generate accurate predictions of the physical environment it is modeling. For example, for high accuracy predictions, a simulation system can generate a target simulation of the state of the training physical environment at one or more time steps using a simulation engine (e.g., a physics engine), where the target simulation has a higher resolution than the fine resolution grid processed by the graph neural network. The simulation system can then generate a lower resolution version of the target simulation by interpolating the target simulation to the same resolution as the fine resolution grid processed by the graph neural network. The simulation system can then use the lower resolution version of the simulation grid to generate one or more of the training examples.

[0142] The above-described systems and methods can be adapted to be implemented on a computing system including a first processor (or processor block) and a second processor (or processor block) having different relative capabilities and communicating with each other. Specifically, the second processor has a relatively higher processing power or memory than the first processor. As an example, such a computing system may include a first general-purpose processor and a second processor having one or more neural network accelerators. A neural network accelerator is a dedicated hardware for accelerating neural network calculations, such as a GPU (graph processing unit) or a TPU (tensor processing unit). Typically, a neural network accelerator is configured to perform hardware matrix multiplication, for example, using parallel computing. A neural network accelerator can include a set of one or more multiply-accumulate units (MACs) to perform such operations. As another example, the first processor may include a first general-purpose processor having a first computing power, which is defined, for example, in FLOPS (floating point operations per second) and / or the amount of memory available for computing. The second processor may include a second general-purpose processor having a second higher computing power, for example, a higher number of FLOPS and / or a higher amount of memory available for computing. As a further example, the first processor may include a processor having a first number of neural network accelerators, and the second processor may include a processor having a second larger number of neural network accelerators.

[0143] In such a computing system, the second processor can be used for fine resolution grid 10f. updates, and the first processor can be used for coarse resolution grid 10.c updates. That is, the graph neural network 150 can be distributed among the first processor and the second processor to optimally allocate computing resources for fine resolution 10.f and coarse resolution 10.c grid updates. For example, one or more fine resolution update blocks 122.u can be implemented on the second processor and one or more coarse resolution update blocks 122.c can be implemented on the first processor. Since the fine resolution update block 122.f is generally more computing-intensive than the coarse resolution update block 122.c, this allows the simulation system 100 to simulate the state of the physical environment more efficiently.

[0144] Thus, in some embodiments, the simulation system 100 processes data defining the fine resolution grid 10.f by implementing one or more fine resolution update blocks 122.f on the second processor, and processes data defining the coarse resolution grid 10.c by implementing one or more coarse resolution update blocks 122.c on the first processor. One or more upsampling update blocks 122.u can be implemented on the first processor and / or the second processor. Similarly, one or more downsampling update blocks 122.d can be implemented on the first processor and / or the second processor.

[0145] Although the processors (processor blocks) can operate in parallel, it is not efficient to do so, for example, because the inputs and outputs are only defined on the fine resolution grid 10.f, the first and last updates on other grids will be wasted. Therefore, in some embodiments, the simulation system 100 first processes the data defining the fine resolution grid 10.f by implementing one or more fine resolution update blocks 122.f on the second processor; then processes (using either processor) the data defining the down-sampled grid to update the current node embedding of each node in the coarse resolution grid 10.c; then processes the data defining the coarse resolution grid 10.c by implementing one or more coarse resolution update blocks 10.c on the first processor; then processes (using either processor) the data defining the up-sampled grid to update the current node embedding of each node in the fine resolution grid 10.f. The step of processing the data defining the coarse resolution grid 10.c by implementing one or more coarse resolution update blocks on the first processor can include performing multiple updates on the data defining the coarse resolution grid 10.c on the first processor.

[0146] Some embodiments of the above-described systems and methods can be used for real-world control, such as controlling a mechanical agent, such as a robot, to perform a task in a real-world environment, for example, using the simulation system 100 for model-based predictive control or as part of an optimal control system for controlling the agent. As an example, the simulation system 100 can be used in this manner to assist a robot in manipulating a deformable object.

[0147] In more detail, the physical environment can be a real-world environment including physical objects, such as objects to be picked up and / or manipulated by the robot. The simulation system 100 can be used to control the robot. Specifically, obtaining data characterizing the state of the physical environment at the current time step can include determining a representation of the position, shape, or configuration of the physical object, for example, by capturing an image of the object. For example, the simulation system 100 can determine the node features of the nodes in the fine resolution 10.f and coarse resolution 10.c grids from the representation of the physical object by: based on determining the node features, then generating a node embedding of the node. The simulation system 100 can determine the state of the physical environment at the next time step by determining a predicted representation of the position, shape, or configuration of the physical object, for example, when subjected to a force or deformation, such as from an actuator of the robot.

[0148] The simulation system 100 can use the predicted representation at the next time step to control the robot to manipulate the physical object, for example, using an actuator. For example, the simulation system 100 can use the predicted representation to control the robot to manipulate the physical object toward a target position, target shape, or target configuration of the physical object by controlling the robot to optimize an objective function that depends on the difference between the predicted representation and the target position, shape, or configuration of the physical object. Controlling the robot can include the simulation system 100 providing a control signal to the robot based on the predicted representation to cause the robot to perform an action, for example, using an actuator to manipulate the physical object to perform a task.

[0149] Some examples of the simulation system 100 involve controlling a robot, such as an actuator of the robot, using a reinforcement learning process with a reward based at least in part on an objective function value to learn to perform a task involving manipulation of a physical object. Alternatively or additionally, this may involve the simulation system 100 controlling the robot using a model predictive control (MPC) process or using an optimal control process.

[0150] Fig. 6A and 6Bis an experimental data plot showing the mean square error (MSE) versus the minimum edge length (edge ​​min) for: (i) a reference simulator (COMSOL), (ii) two variants of the MeshGraphNets (MGN) learning solver with 15 message passing steps (mps) and 25 mps, respectively, and (iii) a fine resolution mesh of example simulation systems 100-1 and 100-2 using two different updater module topologies with 15 mps and 25 mps, respectively. Each of the simulation systems 100-1 and 100-2 uses the same coarse resolution mesh with a minimum edge length of 10 -2 The updater module of the first simulation system 100-1 includes a sequence of fifteen blocks that implement a single block cycle "fd-11c-uf", which amounts to 15 mps. The updater module of the second simulation system 100-2 includes a sequence of twenty-five blocks that implement two block cycles "3f-d-6c-u-3f-d-6c-u-3f", which amounts to 25 mps.

[0151] A set of training data for the MGN model and example simulation system consists of one thousand trajectories of incompressible flow through a long cylinder in a channel simulated using COMSOL. Each trajectory consists of 200 time steps. Parameters such as the radius and position of the obstacle, the initial velocity of the inflow, and the mesh resolution vary between trajectories. It is worth noting that the mesh resolution covers a wide range from a hundred to tens of thousands of nodes.

[0152] The effect of the grid resolution on the predictions of each simulated system was evaluated on a set of validation data consisting of five hundred trajectories with varying grid resolutions but with the initial conditions held constant. The minimum edge lengths (edge ​​min) of these grids ranged from 10 -2 to 10 -3 Since analytical solutions are generally not available for non-trivial simulation settings, high-resolution simulations are often used as a proxy for the “true value” solution to the underlying partial differential equation (PDE). Here, the true value reference trajectory (u ref ) was generated by running COMSOL at maximum resolution on this validation dataset (edge ​​min = 10 -2 ). The error is measured by performing the next step prediction on the validation dataset using either the learned model or the classical solver at a given grid resolution, linearly interpolating the true value trajectory onto the simulation grid and computing the MSE.

[0153] Fig. 6AThe results in show that the MSE of simulation systems 100-1 and 100-2 is significantly reduced compared to the MGN baseline, keeping the total number of mps unchanged. The second simulation system 100-2 with 25 mps manages to successfully closely track the spatial convergence curve of the reference simulator. Therefore, simulation system 100 effectively solves the message passing bottleneck of the underlying problem and is able to achieve higher accuracy at the same number of mps as other graph neural network models. The message passing speed becomes a bottleneck for MGN performance for high-resolution grids; however, this bottleneck is relieved using simulation system 100 with a multi-scale grid approach.

[0154] exist Figure 6B In the present invention, the MGN models (15 mps and 25 mps) and the simulation systems 100-1 and 100-2 are trained on training datasets with mixed grid resolutions, but have high accuracy predictions as described above (e.g., see Figure 4 ). This indicates that the learned solver is able to learn an effective model of the subgrid dynamics and make accurate predictions even at very coarse grid resolutions. This effect extends to 10 -2 , which corresponds to a very coarse mesh with only about a hundred nodes. However, this approach does not alleviate the message propagation bottleneck of the MGN model, and the error increases above the convergence curve when the edge length is below 0.0016. Therefore, if a high resolution output mesh is desired, the accuracy of using MGN is still limited. For methods that perform well on both low and high resolution meshes, it is possible to use simulation system 100 with a high accuracy label. For the second simulation system 100-2 with 25 mps, the error remains below the reference solver curve at all resolutions, with all the performance advantages of simulation system 100.

[0155] This specification associates the term "configured to" with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, when operated, causes the system to perform those operations or actions. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0156] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible, non-transitory storage medium, for execution by a data processing device or for controlling its operation. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0157] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment and machines for processing data, such as a programmable processor, a computer or multiple processors or computers. The apparatus can also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0158] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines or code portions. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0159] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0160] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or a combination of a dedicated logic circuit and one or more programmed computers.

[0161] A computer suitable for executing a computer program can be based on a general or special purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented or incorporated therein by a dedicated logic circuit. Typically, the computer will also include or be operably coupled to receive data from or transmit data to one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0162] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks.

[0163] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending a text message or other form of message to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0164] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0165] The machine learning model can be implemented and deployed using a machine learning framework such as the TensorFlow framework.

[0166] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0167] A computing system can include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on a corresponding computer, and has a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to a user device, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from the device. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0168] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although the above-mentioned features may be described as functioning in certain combinations or even initially claimed, in some cases, one or more features in the claimed combination may be deleted from the combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0169] Similarly, although operations are described in a particular order in the drawings and referenced in the claims, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0170] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for simulating a state of a physical environment executed by one or more computers, the method comprising, for each time step in a plurality of time steps: Data defining a fine resolution grid and a coarse resolution grid are obtained, the fine resolution grid and the coarse resolution grid each representing the state of the physical environment at a current time step, wherein The fine resolution grid has a higher resolution than the coarse resolution grid; The data defining the fine resolution grid and the coarse resolution grid are processed using a graph neural network, wherein the graph neural network comprises: (i) one or more fine resolution update blocks, (ii) one or more coarse resolution update blocks, and (iii) one or more upsampling update blocks, wherein: Each fine-resolution update block is configured to process data defining the fine-resolution grid using a graph neural network layer to update a current node embedding of each node in the fine-resolution grid; Each coarse resolution update block is configured to process data defining the coarse resolution grid using a graph neural network layer to update a current node embedding of each node in the coarse resolution grid; and Each upsampling update block is configured as: generating data defining an up-sampled grid, the up-sampled grid comprising: (i) each node from the fine resolution grid and each node from the coarse resolution grid, and (ii) a plurality of edges between nodes of the fine resolution grid and nodes of the coarse resolution grid; and Processing the data defining the upsampled grid using a graph neural network layer to update a current node embedding for each node in the fine resolution grid; and The state of the physical environment at a next time step is determined using the updated node embeddings for nodes in the fine resolution grid.

2. The method according to claim 1, wherein: For each node of the coarse resolution grid, generating the up-sampled grid comprises: identifying a unit cell of the fine resolution grid that includes the node of the coarse resolution grid; identifying one or more nodes in the fine resolution grid, the one or more nodes being vertices of the unit cell comprising the node of the coarse resolution grid; and In the upsampled grid, corresponding edges between the nodes of the coarse-resolution grid and each identified node in the fine-resolution grid are instantiated.

3. The method of claim 2, further comprising, for each edge in the up-sampled grid: An edge embedding is generated for the edge based on a distance between a pair of nodes connected by the edge in the upsampled grid.

4. A method according to any preceding claim, wherein: Processing the data defining the upsampled grid using a graph neural network layer to update a current node embedding for each node in the fine resolution grid includes: updating an edge embedding of each edge in the upsampled mesh based on: (i) the edge embedding of the edge, and (ii) corresponding node embeddings of a first node in the coarse-resolution mesh and a second node in the fine-resolution mesh connected by the edge; and A node embedding for each node in the fine resolution grid is updated based on: (i) the node embedding for the node in the fine resolution grid, and (ii) a corresponding edge embedding for each edge connecting the node in the fine resolution grid to a corresponding node in the coarse resolution grid.

5. A method according to any preceding claim, wherein: Each upsampling block updates current node embeddings of nodes in the fine resolution grid based at least in part on current node embeddings of nodes in the coarse resolution grid.

6. A method according to any preceding claim, wherein: The graph neural network further comprises one or more downsampling update blocks, wherein each downsampling update block is configured to: generating data defining a down-sampling grid, the down-sampling grid comprising: (i) each node from the fine resolution grid and each node from the coarse resolution grid, and (ii) a plurality of edges between nodes of the fine resolution grid and nodes of the coarse resolution grid; and The data defining the downsampled grid is processed using a graph neural network layer to update a current node embedding for each node in the coarse-resolution grid.

7. The method according to claim 6, wherein: Generating the downsampled grid comprises, for each node of the fine-resolution grid: identifying a unit cell of the coarse resolution grid that includes the node of the fine resolution grid; identifying one or more nodes in the coarse resolution mesh, the one or more nodes being vertices of the unit cell comprising the node of the fine resolution mesh, and In the downsampled grid, a corresponding edge between the node of the fine resolution grid and each identified node in the coarse resolution grid is instantiated.

8. The method of claim 7, further comprising, for each edge in the down-sampling grid: An edge embedding is generated for the edge based on a distance between a pair of nodes connected by the edge in the downsampled grid.

9. The method according to any one of claims 6 to 8, wherein: Using a graph neural network layer to process the data defining the downsampled grid to update the current node embedding of each node in the coarse resolution grid includes: updating an edge embedding of each edge in the downsampled grid based on: (i) the edge embedding of the edge, and (ii) corresponding node embeddings of a first node in the coarse-resolution grid and a second node in the fine-resolution grid connected by the edge; and A node embedding for each node in the coarse-resolution grid is updated based on: (i) the node embedding for the node in the coarse-resolution grid, and (ii) a corresponding edge embedding for each edge connecting the node in the coarse-resolution grid to a corresponding node in the fine-resolution grid.

10. The method according to any one of claims 6 to 9, wherein: Each downsampling block updates current node embeddings of nodes in the coarse-resolution grid based at least in part on current node embeddings of nodes in the fine-resolution grid.

11. A method according to any preceding claim, wherein: The graph neural network has been trained on a set of training examples, wherein one or more of the training examples are generated by operations including: generating, using a simulation engine, a target simulation of a state of a training physics environment at one or more time steps, wherein the target simulation has a higher resolution than the fine resolution grid processed by the graph neural network; generating a lower resolution version of the target simulation by interpolating the target simulation to the same resolution as the fine resolution grid processed by the graph neural network; and The training examples are generated using the lower resolution version of the simulation mesh.

12. A method according to any preceding claim, wherein: Obtaining data defining the state of the physical environment at the current time step comprises, for each node in the fine resolution grid: obtaining one or more node features of the node, wherein the node corresponds to a location in the physical environment, and wherein the node features characterize a state of the corresponding location in the physical environment; and The node features are processed using one or more neural network layers of the graph neural network to generate a current embedding for the node.

13. The method according to claim 12, wherein: For each node in the fine resolution grid, the node feature of the node includes one or more of the following: a fluid density feature, a fluid viscosity feature, a pressure feature, or a tension feature.

14. The method according to any one of claims 12 to 13, wherein: The graph neural network further comprises a decoder block, and wherein determining the state of the physical environment at the next time step comprises: processing the updated node embeddings for each node in the fine resolution grid to generate one or more respective dynamic features corresponding to each node in the fine resolution grid; and The state of the physical environment at the next time step is determined based on: (i) dynamic characteristics of nodes in the fine resolution grid, and (ii) nodal characteristics of nodes in the fine resolution grid at the current time step.

15. A method according to any preceding claim, wherein: The fine resolution grid and the coarse resolution grid are each three-dimensional grids.

16. A method according to any preceding claim, wherein: The fine resolution mesh and the coarse resolution mesh are each triangular meshes.

17. A method according to any preceding claim, wherein: The fine resolution grid and the coarse resolution grid each span the physical environment.

18. A method according to any preceding claim, wherein: For each time step, the number of nodes in the fine resolution grid is greater than the number of nodes in the coarse resolution grid.

19. A method according to any preceding claim, performed on a computing system comprising a first processor and a second processor, wherein: The second processor has a higher processing capability or memory than the first processor, and the method comprises: processing data defining the fine resolution grid by implementing the one or more fine resolution update blocks on the second processor; and Data defining the coarse resolution grid is processed by implementing the one or more coarse resolution update blocks on the first processor.

20. The method according to claim 19, when dependent on claim 6, further comprising: processing data defining the fine resolution grid by implementing the one or more fine resolution update blocks on the second processor; Then processing data defining the downsampled grid to update a current node embedding for each node in the coarse resolution grid; Then processing data defining the coarse resolution grid by implementing the one or more coarse resolution update blocks on the first processor; Then Data defining the upsampled grid is processed to update a current node embedding for each node in the fine resolution grid.

21. A method of controlling a robot using the method according to any one of claims 1 to 20, in, The physical environment includes a real-world environment, and the real-world environment includes physical objects; wherein obtaining data defining the fine resolution grid and the coarse resolution grid, the fine resolution grid and the coarse resolution grid each characterizing the state of the physical environment at the current time step comprises: determining a representation of the position, shape, or configuration of the physical object at the current time step; Wherein, determining the state of the physical environment at the next time step comprises: determining a predicted representation of the position, shape, or configuration of the physical object at the next time step; and The method further comprises, at each time step: The robot is controlled using the predicted representation at the next time step to manipulate the physical object.

22. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 21.

23. A system comprising: one or more computers; as well as One or more storage devices, the one or more storage devices being communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 21.

Images