Predicting structures of substances using diffusion models

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting structures of substances. One of the methods includes: generating an initial structure of a substance, wherein the initial structure of the substance defines a respective initial location for each of a plurality of components of the substance; generating a final structure of the substance that defines a respective final location for each of the plurality of components of the substance, wherein updating the initial structure of the substance comprises, at each of multiple updating iterations: processing one or more diffusion inputs for the updating iteration that each comprise a current structure of the substance as of the updating iteration using a diffusion neural network to generate one or more diffusion outputs for the updating iteration; and updating the current structure of the substance using the one or more diffusion outputs for the updating iteration.

Description

[0001] Attorney Docket No.56113-0545WO1 PREDICTING STRUCTURES OF SUBSTANCES USING DIFFUSION MODELS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No.63 / 541,255, filed on September 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to predicting structures of substances.

[0003] The structure of a substance refers to the arrangement of the constituent components of the substance. Some substances, e.g., crystalline materials, have a crystal structure. In such substances, the constituent components (such as atoms, ions, or molecules) are arranged in an ordered pattern that is repeated over the entire substance. A crystal structure can be characterized by its unit cell, which is a simplest repeating unit containing one or more constituent components in a specific spatial arrangement. The unit cells are stacked in three- dimensional space to form the substance.

[0004] The predictions can be made using neural networks. Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0005] This specification describes a substance discovery system implemented as computer programs on one or more computers in one or more locations can be used to discover and validate novel substances by predicting the structure of the substances.

[0006] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The techniques described in this specification can efficiently discover substances, e.g., crystalline materials or other materials, that are both novel and stable over a reverse diffusion process. Compared with other existing materials discovery techniques, e.g., graph-based approaches or random structure search, the structures generated by using the described techniques are more diverse Attorney Docket No.56113-0545WO1 and have a higher fidelity. The described techniques are scalable to generate an arbitrary number of, e.g., one million, ten million, or more, predicted substance structures given a fixed time budget. Moreover, the described techniques are scalable to predict large and complex structures, e.g., structures for substances having a larger number of constituent components, e.g., having millions of atoms.

[0007] Some existing processes for novel substance evaluation involve a time-consuming and computationally expensive process of laboratory experimentation, and can require significant time and computational costs, sometimes taking hundreds or thousands of human expert hours for validation. From another aspect, because higher fidelity structures can be generated, the techniques described in this specification can save the time and reduce computational costs that would otherwise be used to evaluate less stable novel substances and other novel substances that do not meet formation energy criteria, stability criteria, and so on.

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG.1 shows an example substance discovery system.

[0010] FIG.2 is an example illustration of a unified representation of materials.

[0011] FIG.3 is a flow diagram of an example process for generating a final structure of a substance using a diffusion neural network.

[0012] FIG.4 is an example illustration of generating a final structure of a substance using a diffusion neural network.

[0013] FIG.5 illustrates ten example substances discovered using the substance discovery system of FIG.1.

[0014] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0015] FIG.1 shows an example substance discovery system 100. The substance discovery system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. Attorney Docket No.56113-0545WO1

[0016] The substance discovery system 100 includes a diffusion neural network 120 that can predict final structures of substances.

[0017] The diffusion neural network 120 can predict the structure of any of a variety of substances. Each substance is composed of and can include a plurality of components.

[0018] For example, the substance can be a crystalline material, and the structure can be a crystal structure. In this example, the components of the substance include one or more of: atoms, molecules, or ions. Atoms are single neutral particles. Molecules are neutral particles made of two or more atoms bonded together. An ion is a positively or negatively charged particle.

[0019] As another example, the substance can be a molecule, and the structure can be a chemical structure. In this example, the components include atoms of different types.

[0020] As another example, the substance can be a drug or a bio-active agent, and the structure can be a chemical structure. In this example, the components include atoms of different types.

[0021] As another example, the substance can be a protein, and the structure can be a chemical structure. In this example, the components include amino acids. A protein is specified by one or more sequences (“chains”) of amino acids. An amino acid is an organic compound which includes an amino functional group and a carboxyl functional group, as well as a side chain (i.e., group of atoms) that is specific to the amino acid.

[0022] The diffusion neural network 120 can be any appropriate diffusion neural network that has been trained, e.g., by the substance discovery system 100 or another training system, to enable the diffusion neural network 120 to perform its described functions.

[0023] For example, the diffusion neural network 120 can have been trained on a set of training substance structures based on minimizing the following objective: which measures a noise ^ ∼ ^^^0, ^^ௗ^ and a noise estimate ^^^generated by the diffusion neural network 120 from processing a noise corrupted version of a structure of a substance ^^ that is generated by adding the randomly sampled Gaussian noise ^ to the structure of the substance ^^, where ^^௧∈ ℝ is a noise level for a time step t selected from, e.g., randomly sampled from ^^ ∈ ^^].

[0024] As an example, the set of training substances can include the perovskite substance structures included in the Perov-5 dataset. As another example, the set of training substances can include the carbon substance structures included in the Carbon-24 dataset. As a further Attorney Docket No.56113-0545WO1 example, the set of training substances can include the general inorganic substance structures included in the MP-20 dataset.

[0025] For example, the diffusion neural network 120 can have a convolutional neural network architecture, e.g., a U-Net architecture or other convolutional architecture, that includes one or more convolutional residual blocks. As another example, the diffusion neural network 120 can have an attention neural network architecture that includes one or more attention blocks, e.g., self-attention blocks, gated attention blocks, or cross-attention blocks. As yet another example, the diffusion neural network 120 can include both convolutional residual blocks and attention blocks.

[0026] As a few particular examples, the diffusion neural network 120 can have one of the 3D U-Net architectures desctibed in Çiçek, Özgün, et al., 3D U-Net: learning dense volumetric segmentation from sparse annotation. Medical Image Computing and Computer- Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. Springer International Publishing, 2016, and Ho, Jonathan, et al., Video diffusion models. Advances in Neural Information Processing Systems 35 (2022): 8633-8646.

[0027] To predict the structure of a substance, the substance discovery system 100 initializes the structure of the substance, i.e., generates an initial structure 122 of the substance, and then uses the diffusion neural network 120 to generate a final structure 124 of the substance, based on performing a reverse diffusion process to update the initial structure 122 of the substance over multiple updating iterations. The final structure 124 of the substance is therefore generated after the last updating iteration of the reverse diffusion process.

[0028] In some implementations, the substance discovery system 100 can generate the initial structure 122 of the substance by sampling a value for each parameter in the initial structure of the substance from a corresponding noise distribution, e.g., a Gaussian distribution, or a different noise distribution.

[0029] In some implementations, the diffusion neural network 120 can be configured as an unconditional diffusion model, and the reverse diffusion process can be an unconditional reverse diffusion process, i.e., without being conditioned on any conditioning input. In these implementations, the final structures 124 generated by the substance discovery system 100 may approximate samples of a distribution of training substance structures that were using during the training of the diffusion neural network 120.

[0030] In other implementations, the diffusion neural network 120 can be configured as a conditional diffusion model, and the reverse diffusion process can be a conditional reverse Attorney Docket No.56113-0545WO1 diffusion process. That is, the substance discovery system 100 obtains a conditioning input prior to the beginning of the reverse diffusion process and then uses the conditioning input as a guidance during the reverse diffusion process. In these implementations, the final structures 124 generated by the substance discovery system 100 is guided by the conditioning input.

[0031] The conditioning input can include a composition conditioning input that characterizes one or more desired composition properties for the substance. Optionally, the conditioning input can further include an auxiliary conditioning input that characterizes one or more desired auxiliary properties for the substance.

[0032] The reverse diffusion process updates the respective locations of a plurality of components of the substance. The initial structure 122 of the substance defines, for each of the plurality of components of the substance, a respective initial location of the component. The final structure 124 of the substance defines, for each of the plurality of components of the substance, a respective final location of the component.

[0033] In other words, the same component may be included in both the initial structure 122 of the substance and the final structure 124 of the substance, the initial structure 122 may define that the component has a first location, while the final structure 124 may define that the component has a second location that is different from the first location and that is determined as a result of the reverse diffusion process performed by using the diffusion neural network 120.

[0034] The initial structure 122 or the final structure 124 may define that, for at least one of the plurality of components of the substance, the respective location is a null location. By assigning a component to a null location, the initial structure 122 or the final structure 124 indicates that the component is excluded from the substance.

[0035] The initial 122 and final structures 124 of the substance both have the same structured data format that can represent the arrangement of the plurality of components of the substance. Advantageously, such a structured data format is a scalable and flexible representation that can represent any structure. The structured data format introduces minimal or no intrinsic errors (e.g., unlike voxel image representations), and is able to support both up scaling to large sets of substances and down scaling to a single compound system of interest (e.g., silicon carbide).

[0036] FIG.2 is an example illustration 200 of the structured data format. For ease of discussion, the description of FIG.2 focuses on that the structured data format is used to represent crystal substances. It will be appreciated that the structured data format can be used to represent a broader range of substances (e.g., drugs, molecules, and proteins). Attorney Docket No.56113-0545WO1

[0037] The structured data format is based on the periodic table. To represent the crystal substance, the structured data format stores, for each of a plurality of components of the substance, one or more parameters that map the component to a chemical element in the periodic table. For example, the structured data format can represent the atoms in a unit cell of the crystal substance by storing location parameters that define the locations ^^, ^^, ^^ of each atom at the entry of the structured data format that maps to a corresponding chemical element in the periodic table. A unit cell is the smallest repeating unit of a crystal.

[0038] More specifically, in this structured data format, a structure of a crystal substance canbe represented by parameters stored in a 4-dimensional tensor ^^ ∶= ℝ^×ு×^×^, where ^^ =9 and ^^ = 18 correspond to the number of periods (horizontal (vertical columns), respectively, in the periodic table, ^^ (any non-negative to the maximum number of atoms that can be included in the crystal substance for each chemical element in the periodic table, and ^^ = 3 corresponds to the ^^, ^^, ^^ locations of each atoms in a unit cell of the crystal substance.

[0039] This structured data format is therefore capable of representing any crystal structure ^^ ∈ ^^ with no greater than ^^ atoms per chemical element in the periodic table. As mentioned above, the structured data format can also represent a broader range of substances. To that end, ^^ and ^^ can be set to different numbers.

[0040] For example, to represent a specific element (e.g., carbon in graphene) or two- chemical compounds (e.g., silicon carbide), instead of setting ^^ and ^^ to the full periods andgroups of the periodic table (that is, 8 and 18, respectively), it is possible to set ^^ = 1, ^^ =1 (for one specific element) or ^^ = 9, ^^ = 2 (for elements from two groups) to representspecific substances of interest. Likewise, ^^ can also be adjusted according to the actual number of elements expected to be included in a specific substance of interest.

[0041] In some implementations, for each atom in the unit cell, the location can be defined in a Cartesian coordinate system. For example, on the left hand side, FIG.2 illustrates that the bottom right atom that corresponds to chemical element Sodium (Na) in the crystal substance is located at [0.5, 0, 0], hence the structured data format shown on the right hand side shows that it stores location parameters [0.5, 0, 0] at the entry that maps to the corresponding chemical element Sodium (Na) in the periodic table. For ease of illustration, FIG.2 shows only 1 / 8 of the unit cell of the crystal structure on the left hand side.

[0042] In some implementations, for each atom in the unit cell, the location can be defined in a fractional coordinate system (crystal coordinate system). In crystallography, a fractional Attorney Docket No.56113-0545WO1 coordinate system is a coordinate system in which basis vectors used to the describe the space are the lattice vectors of a crystal pattern. Fractional coordinates are expressed as fractions of the unit cell in each of the three directions ^^, ^^, ^^ separated by the angles ^^, ^^, ^^.

[0043] In these implementations, in addition to the location parameters that define the locations ^^, ^^, ^^ of each atom at the entry of the structured data format that maps to a corresponding chemical element in the periodic table, the structured data format also stores additional unit cell parameters ( ^^, ^^, ^^) ∈ ℝଷand ( ^^, ^^, ^^) ∈ ℝଷthat specify the lengths and angles, respectively, between the edges of the unit cell, as shown in FIG.2.

[0044] In either coordinate system, a null location can be defined using special ^^, ^^, ^^ locations to indicate the absence of an atom in a substance. For example, an atom that is not included in the substance can have location parameters that define a special location of ^^ = ^^ = ^^ = −1; ^^ = ^^ = ^^ = ∞; or the like.

[0045] Performing updating iterations is described in more detail below with reference to FIGS.3-4. After the last updating iteration, the substance discovery system 100 generates the final structure 124 of the substance. The final structure 124 of the substance generated as a result of the reverse diffusion process can be used in any of a variety of ways.

[0046] For example, the substance discovery system 100 can store the final structure 124 of the substance in an output data repository, e.g., a substance database, or provide the final structure of the substance for use for some other purpose. For example, the substance discovery system 100 can then output the final structure 124 of the substance for presentation, e.g., on a client device.

[0047] In some implementations, the substance discovery system 100 includes a substance evaluation sub-system 130. The substance evaluation sub-system 130 can evaluate the substance having the final structure 124 by determining any of a variety of metrics of the substance that has the final structure 124. As an example, the metrics can include the formation energy metric of the substance. As another example, the metrics can include the stability metric of the substance. As a further example, the metrics can include one or more of the structure validity metric, the recall coverage metric, the precision coverage metric, or the property statistics metric, as explained in more detail in Xie, Tian, et al. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197 (2021).

[0048] The substance evaluation sub-system 130 can determine the metrics of the substance by using any of a variety of methods. As an example, the substance evaluation sub-system Attorney Docket No.56113-0545WO1 130 can computationally determine the metrics of the substance by performing a high-fidelity computational simulation of the substance that has the final structure 124. As another example, the substance evaluation sub-system 130 can perform ab-initio simulations (e.g., based on density functional theory (DFT) calculations) to determine the metrics of the substance. As a further example, the substance evaluation sub-system 130 can request and receive results for experimental evaluation results of the metrics of the substance by, e.g., providing the final structure 124 of (and possibly other data characterizing) the substance for synthesis and testing the substance and obtaining results from physical tests of the synthesized substance.

[0049] As a particular example of how the metrics of the substance of the substance can be determined based on performing density functional theory (DFT) calculations, the substance evaluation sub-system 130 can use a convex hull phase diagram to ensure the accuracy of the metrics.

[0050] Specifically, in this example, the substance evaluation sub-system 130 can use DFT to compute decomposition energy ^^ ^^. ^^ ^^ measures a compound’s thermodynamic decomposition enthalpy into its most stable compositions on a convex hull phase diagram. The convex hull is formed by linear combinations of the most stable (lowest energy) phases for each known composition of chemical elements. As a result, decomposition energy allows the substance evaluation sub-system 130 to compare substances (compounds) generated by two generative models that differ in composition by separately computing their decomposition energy with respect to the convex hull formed by a larger substance database. The distribution of decomposition energies will reflect a generative model’s ability to generate relatively stable substances. The substance evaluation sub-system 130 can further compute the number of novel stable ( ^^ ^^ < 0) substances from a first set ^^ with respect to convex hull as and compare this quantity to a

[0051] Any of a variety of actions can then be performed based at least in part on the determined metrics of the substance. For example, when the metrics of the substance having the final structure 124 meets certain criteria, the substance evaluation sub-system 130 can add data characterizing the substance to a substance database. In general, the substance evaluation sub-system 130 can update the substance database to include every substance having a respective final structure determined by the substance discovery system 100. Attorney Docket No.56113-0545WO1

[0052] In some implementations, the substance discovery system 100 includes a substance synthesis sub-system 140. The substance synthesis sub-system 140 can physically synthesize the substance having the final structure 124 for experimental validation. In these implementations, physical tests can then be performed on a physically synthesized instance to determine one or more properties of the instance.

[0053] In practice, the physical synthesis of substances can require significant time, expertise, laboratory resources, etc. The substance synthesis sub-system 140, when included, can therefore use the metrics determined by the substance evaluation sub-system 130 for a plurality of substances to select a subset of the plurality of substances for physical synthesis. In some implementations, the substance synthesis sub-system 140 will physically synthesize only the substances included in the subset.

[0054] For example, the substance discovery system 100 can use the diffusion neural network 120 to generate the final structures for a large number of substances (e.g., at least one million, two million, or more substances) and then use the substance synthesis sub- system 140 to physically synthesize a smaller fraction of the substances (e.g., no more than two thousand, one thousand, or fewer substances) that have metrics determined by the substance evaluation sub-system 130 that satisfy certain criteria (e.g., have a formation energy within a particular range, attain a certain threshold of stability, etc.). By using the metrics determined by the substance evaluation sub-system 130 to screen the substances for physical synthesis, the substance discovery system 100 can more efficiently synthesize and verify novel substances.

[0055] The substance discovery system 100 can, in some examples, validate one or more substances against a set of target substance properties for a product. For example, it may be desirable for a component of a product to have one or more physical properties within some respective range of values and / or above / below a respective threshold value. Such values include, for example, one or more of: a stability; a conductance / resistance; a hardness; a stiffness; a band gap; one or more optical properties; and / or the like. The system 100 can compare predicted physical properties of the substances with the set of target substance properties to select one or more substances that satisfy the set of target substance properties. The one or more selected substances can, in some examples, then be synthesized, and their physical properties verified. One or more of the substances that satisfy the set of target substance properties are selected and incorporated into the product, which is then manufactured. For example, a component of the product can be manufactured using the selected substance. Attorney Docket No.56113-0545WO1

[0056] FIG.3 is a flow diagram of an example process 300 for generating a final structure of a substance using a diffusion neural network. The substance includes a plurality of components. The plurality of components can include atoms, molecules, or ions, to name just a few examples. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a substance discovery system, e.g., the substance discovery system 100 depicted in FIG.1, appropriately programmed in accordance with this specification, can perform the process 300.

[0057] To generate the final structure of the substance, the system uses the diffusion neural network to repeatedly perform an iteration of the process 300 to update a current structure of the substance at each of multiple updating iterations over a reverse diffusion process. That is, the process 300 can be performed at each of the multiple updating iterations over the reverse diffusion process. By repeatedly performing iterations of the process 300, the system can generate the final structure of the substance.

[0058] Prior to first iteration of the process 300, i.e., prior to the first updating iteration, the system initializes the structure of the substance, i.e., generates an initial structure of the substance. The initial structure of the substance is the same dimensionality as the final structure of the substance, i.e., includes the same number of parameters as the final structure of the substance, but has different values for at least some of these parameters than the final structure of the substance.

[0059] The initial structure of the substance (or, analogously, the final structure of the substance) has the structured data format as described above with reference to FIG.2. As noted above, to represent a substance, the structured data format stores, for each of the plurality of components of the substance, one or more parameters that map the component to a chemical element in the periodic table.

[0060] The parameters included in the initial structure of the substance can be represented as a 4-dimensional tensor ^^ ∶= ℝ^×ு×^×^, where ^^ can be any integer between 0 and 9, inclusive on both ends, ^^ 0 and 18, inclusive on both ends, ^^ can be any non-negative integer, and ^^ = 3.

[0061] In some implementations, the system can generate the initial structure of the substance by sampling a value for each parameter in the initial structure of the substance from a corresponding noise distribution, e.g., a Gaussian distribution, or a different noise distribution. Attorney Docket No.56113-0545WO1

[0062] In some implementations, the system can generate the initial structure of the substance by setting a value for each of some of the parameters (e.g., the parameters stored in the ^^, ^^, and ^^ dimensions of the 4-dimensional tensor ^^) in the initial structure of the substance to a fixed value, while sampling a value for each of others of the parameters (e.g., the parameters stored in the ^^ dimension of the 4-dimensional tensor ^^) in the initial structure of the substance from a corresponding noise distribution, e.g., a Gaussian distribution, or a different noise distribution. That is, the initial structure of the substance includes some parameters that each has a fixed value, and some other parameters that each has a value being sampled from a corresponding noise distribution.

[0063] In some implementations, prior to first iteration of the process 300, i.e., prior to the first updating iteration, the system obtains a conditioning input. For example, the system can receive the conditioning input as one or more user inputs, or from another system.

[0064] The conditioning input can include a composition conditioning input ^^ that characterizes one or more desired composition properties for the substance. For example, the one or more desired composition properties for the substance can include one or more of: (i) desired types of components of the substance, (ii) desired ratios between the desired types of components of the substance, or (iii) for each desired type of component, a maximum number of components of the desired type to be included in the substance.

[0065] For example, the conditioning input can be represented as a 2-dimentional tensor ^^ =ℝு×^ that defines the desired types of chemical components and the desired ratios betweentypes of chemical components be included in the substance (e.g., carbon and silicon). As another example, the conditioning input can be represented as a 3-dimentional tensor ^^ = ℝ^×ு×^that also defines, for each desired type of components, a maximum number of components of the desired type to be included in the substance (e.g., ^^ ^^ସ^^ସ).

[0066] Optionally, the conditioning input can further include an auxiliary conditioning input ^^ ^^ ^^ that characterizes one or more desired auxiliary properties for the substance. For example, the one or more desired auxiliary properties for the substance can include one or more of: (i) a formation energy of the substance, (ii) a bandgap of the substance, or (iii) a textual description of the substance. More generally, the conditioning input can include any type of data, including, e.g., natural language text, that characterizes one or more aspects of the substance.

[0067] The system processes one or more diffusion inputs for the updating iteration using the diffusion neural network to generate one or more diffusion outputs for the updating iteration Attorney Docket No.56113-0545WO1 (step 302). Each diffusion input includes at least a current structure of the substance as of the updating iteration. If the updating iteration is the first updating iteration in the reverse diffusion process, the current structure of the substance is the initial structure of the substance. For any subsequent updating iteration, the current structure of the substance is the updated structure of the substance that has been generated in the immediately preceding updating iteration.

[0068] Each diffusion output defines a noise estimate ^^ for the current structure of the substance. The noise estimate is an estimate of the noise that needs to be added, i.e., added to the final structure of the substance being generated by the system, to arrive at the current structure of the substance.

[0069] In implementations where the reverse diffusion process is an unconditional reverse diffusion process, the system can process a diffusion input that includes (i) the current structure of the substance ^^௧, and (ii) a timestep index ^^ that corresponds to the updating iteration in the multiple updating iterations, using the diffusion neural network to generate a diffusion output ^^^( ^^௧, t) where θ represents (the parameters of) the diffusion neural network.

[0070] In implementations where the reverse diffusion process is a conditional reverse diffusion process and the conditioning input includes a composition conditioning input ^^, the system can process a first diffusion input that includes (i) the composition conditioning input ^^, (ii) the current structure of the substance ^^௧, and (iii) a timestep index ^^ that corresponds to the updating iteration in the multiple updating iterations, using the diffusion neural network to generate a first diffusion output ^^^( ^^௧, t| c).

[0071] For example, the diffusion neural network can include an encoder neural network that encodes the composition conditioning input into an embedding, and a backbone neural network that processes the embedding and the other data included in the first diffusion input to generate the first diffusion output.

[0072] Optionally, when the conditioning input also includes an auxiliary conditioning input ^^ ^^ ^^, the system can additionally process a second diffusion input that includes (i) the composition conditioning input ^^, (ii) the auxiliary conditioning input ^^ ^^ ^^, (iii) the current structure of the substance ^^௧, and (iv) a timestep index ^^ that corresponds to the updating iteration in the multiple updating iterations, using the diffusion neural network to generate a second diffusion output ^^^( ^^௧, t|c, aux).

[0073] The system updates the current structure of the substance ^^௧using the one or more diffusion outputs for the updating iteration (step 304). That is, the system updates the values Attorney Docket No.56113-0545WO1 of at least some of the parameters (e.g., the values of the parameters stored in the ^^ dimension of the 4-dimensional tensor ^^) included in the current structure of the substance. The system can generate an updated structure of the substance ^^௧ି^based on de-noising, i.e., removing any noise from, the current structure of the substance ^^௧, in accordance with the noise estimates defined in the one or more diffusion outputs generated by the diffusion neural network.

[0074] For example, in implementations where the reverse diffusion process is an unconditional reverse diffusion process, the system can generate an updated structure of the substance ^^௧ି^from the current structure of the substance ^^௧based on the following equation: where ^^(0, ^^ௗ) is the updating iteration, ^^௧is a pre- ^^௧is a time varying noise level that depends on ^^௧and ^^௧, where ^^௧∈ ℝ is a pre-defined noise level for the updating iteration.

[0075] As another example, in where the reverse diffusion process is a conditional reverse diffusion process, the system can process generate an updated structure of the substance ^^௧ି^from the current structure of the substance ^^௧based on a modified version of the equation above where ^^^( ^^௧, t) is replaced with ^^^( ^^௧, t| c) (when the conditioning input includes a composition conditioning input ^^), ^^^( ^^௧, t|c, aux) when the conditioning input includes an auxiliary conditioning input ^^ ^^ ^^), or ^^^̂( ^^௧, t|c, aux) (when the conditioning input includes both a composition conditioning input ^^ and an auxiliary conditioning input ^^ ^^ ^^).

[0076] For example, ^^^̂( ^^௧, t|c, aux) = (1 ^ω) ^^^( ^^௧, t|c, aux) − ω ^^^( ^^௧, t|c) where ω is a pre-defined weight that controls the strength of auxiliary information conditioning relative to composition conditioning. Thus, in this example, the updated structure of the substance ^^௧ି^will be computed based on the current structure of the substance ^^௧, in accordance with the pre-defined weights assigned to the first and second diffusion outputs that are generated by the diffusion neural network from processing the current structure of the substance ^^௧.

[0077] When the updating iteration is not the last updating iteration in the reverse diffusion process, the system can perform a next iteration of the process 300, where the updated structure of the substance will be used as the current structure of the substance in the next iteration of the process 300. Attorney Docket No.56113-0545WO1

[0078] Alternatively, when the updating iteration is the last updating iteration, the system can generate the final structure of the substance based on the updated structure of the substance. For example, the system can output the updated structure of the substance generated in the last updating iteration as the final structure of the substance. As another example, the system can further process the updated structure of the substance generated in the last updating iteration to generate the final structure of the substance.

[0079] For example, the system can filter out, i.e., remove, any component of the substance that has a null location as defined by the updated structure of the substance, such that the final structure of the substance includes parameters having values that define only the valid (i.e., not null) locations for remaining components of the substance.

[0080] As another example, the system can determine if the updated structure of the substance generated in the last updating iteration includes parameters having values that define a same location for two or more of the plurality of components of the substance. That is, the system determines whether any components have overlapping locations as defined by the updated structure of the substance. The system can then filter out, i.e., remove, the two or more components that are located at the same location, such that the final structure of the substance includes parameters having values that define distinct locations for remaining components of the substance.

[0081] FIG.4 is an example illustration 400 of generating a final structure of a substance using a diffusion neural network, where the substance is a crystal substance and the components include atoms. In particular, FIG.4 illustrates performing an unconditional reverse diffusion process to generate a final structure of a substance ^^^420 from an initial structure of the substance ^^்410 where the time ^^ runs from ^^ = ^^ to ^^ = 0.

[0082] As illustrated, the initial structure of the substance ^^்410 includes parameters having initial values that define an initial (e.g., random) location for each component of the substance. The unconditional reverse diffusion process gradually updates the values of the parameters to define updated location for each of at least some of the components of the substance. The final structure of the substance ^^^420 includes parameters having final values that define a final location for each component of the substance.

[0083] The final structure of the substance ^^^420 may include parameters having final values that define that the final locations for at least some components of the substance are a null location, indicating that these components are excluded from the substance. For example, in FIG.4, the final structure of the substance ^^^420 defines that atoms corresponding to the Attorney Docket No.56113-0545WO1 chemical element Manganese (Mn) have the null location. Thus, atoms corresponding to Manganese (Mn) will be excluded from the substance, i.e., the substance does not include any atoms that correspond to Manganese (Mn).

[0084] FIG.5 illustrates ten example substances discovered using the substance discovery system of FIG.1.

[0085] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0086] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0087] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a Attorney Docket No.56113-0545WO1 protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0088] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0089] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0090] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0091] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or Attorney Docket No.56113-0545WO1 optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0092] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0093] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0094] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0095] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework.

[0096] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described Attorney Docket No.56113-0545WO1 in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0097] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0098] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0099] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Attorney Docket No.56113-0545WO1

[0100] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0101] What is claimed is:

Claims

Attorney Docket No.56113-0545WO1 CLAIMS 1. A method performed by one or more computers, the method comprising: generating an initial structure of a substance, wherein the initial structure of the substance defines a respective initial location for each of a plurality of components of the substance; generating, based on updating the initial structure of the substance, a final structure of the substance that defines a respective final location for each of the plurality of components of the substance, wherein updating the initial structure of the substance comprises, at each of a plurality of updating iterations: processing one or more diffusion inputs for the updating iteration that each comprise a current structure of the substance as of the updating iteration using a diffusion neural network to generate one or more diffusion outputs for the updating iteration; and updating the current structure of the substance using the one or more diffusion outputs for the updating iteration.

2. The method of claim 1, wherein the final structure of the substance is a crystal structure of a material, and wherein the plurality of components comprise one or more of: atoms, molecules, or ions.

3. The method of claim 1, wherein the final structure of the substance is one of: a chemical structure of a drug, a chemical structure of a molecule, or a chemical structure of a protein.

4. The method of any one of claims 1-2, wherein the one or more diffusion inputs for the updating iteration each comprise timestep data that corresponds to the updating iteration.

5. The method of any one of claims 1-4, wherein at least one of the one or more diffusion inputs for the updating comprises a composition conditioning input that characterizes one or more desired composition properties for the substance.Attorney Docket No.56113-0545WO1 6. The method of claim 5, wherein the one or more desired composition properties for the substance comprise one or more of: desired types of components of the substance, desired ratios between the desired types of components of the substance, or for each desired type of components, a maximum number of components of the desired type to be included in the substance.

7. The method of any one of claims 1-6, wherein at least one of the one or more diffusion inputs for the updating iteration comprises an auxiliary conditioning input that characterizes one or more desired auxiliary properties for the substance.

8. The method of claim 7, wherein the one or more desired auxiliary properties for the substance comprise one or more of: a formation energy of the substance, a bandgap of the substance, or a textual description of the substance.

9. The method of any one of claims 1-8, wherein the one or more diffusion outputs for the updating iteration each define a noise estimate for the current structure of the substance.

10. The method of claim 9, wherein updating the current structure of the substance using the one or more diffusion outputs for the updating iteration comprises: updating, based on the noise estimates defined by the one or more diffusion outputs, and in accordance with pre-defined weights assigned to the one or more diffusion outputs, the current structure of the substance to generate an updated structure of the substance.

11. The method of any one of claims 1-8, wherein an updated structure of the substance as of a last updating iteration defines a same predetermined location for each of two or more of the plurality of components of the substance.

12. The method of claim 9, wherein generating the final structure of the substance comprises: filtering out the two or more components that are located at the same predetermined location in the updated structure of the substance as of the last updating iteration.Attorney Docket No.56113-0545WO1 13. The method of any preceding claim when also dependent on claim 2, wherein the initial structure of the substance comprises, for each of the plurality of components of the substance, one or more parameters that map the component to a chemical element in a periodic table.

14. The method of any preceding claim when also dependent on claim 2, wherein the initial locations are defined in a Cartesian coordinate system.

15. The method of any preceding claim when also dependent on claim 2, wherein the initial locations are defined in a fractional coordinate system.

16. The method of any preceding claim when also dependent on claim 2, further comprising evaluating the substance having the final structure based on density function theory (DFT) calculations to determine a formation energy metric, a stability metric, or both of the substance having the final structure.

17. The method of claim 16, wherein evaluating the substance having the final structure based on the density function theory (DFT) calculations comprise using a convex hull phase diagram, wherein the convex hull is formed by linear combinations of lowest energy phases for each known composition.

18. The method of any preceding claim, further comprising: performing an action based at least in part on the formation energy metric, the stability metric, or both of the substance.

19. The method of claim 18, wherein performing the action based at least in part on the formation energy metric, the stability metric, or both of the substance comprises: physically synthesizing the substance for experimental validation.

20. The method of claim 19, further comprising: performing physical tests on a physically synthesized instance to determine one or more properties of the instance.Attorney Docket No.56113-0545WO1 21. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-20.

22. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-20.

Images