antisymmetric neural network

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using antisymmetric neural networks.

Description

Background Technology

[0001] This manual relates to neural networks.

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output in response to a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network—that is, the next hidden layer or output layer. Each layer of the network generates its output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0003] This specification describes a system for processing inputs using an antisymmetric neural network, implemented as a computer program on one or more computers at one or more locations. Generally, an antisymmetric neural network is a neural network that is antisymmetric with respect to its inputs. That is, if the relative positions of any two vectors in the network input are swapped, the output of the neural network after the swap will be equal in magnitude but opposite in sign to the output of the neural network before the swap.

[0004] In one aspect, a computer-implemented method is described, comprising obtaining input data characterizing a system having multiple electrons, the input data including electronic features of each of the multiple electrons and pair features of each of a plurality of pairs of the multiple electrons; and processing the input data using an antisymmetric neural network having multiple intermediate layers. The antisymmetric neural network can be configured to process the input data to generate predicted values ​​of one or more properties of the system as output; and each of the multiple intermediate layers of the antisymmetric neural network can be configured to generate a corresponding layer output for that intermediate layer from the corresponding layer input by applying a corresponding permutation equivalence function to the corresponding layer input of that intermediate layer.

[0005] One application of the systems and methods is predicting the wavefunction values ​​of chemical systems such as molecules. Some known methods for solving this problem are accurate but do not scale well and are computationally impractical for molecules other than very small ones. They are also inaccurate on nonequilibrium geometries, which is important in practice. Variational Monte Carlo methods are computationally less intensive but less accurate. Some neural network-based methods utilize supervised learning, trained on examples generated from existing computational methods.

[0006] By comparison, the implementation of the described system and method can produce accurate results by including pairwise (interchangeable) interactions between each pair of electrons and using features from each (individual) electron. In principle, this approach is also computationally difficult. However, the described system and method also recognize that the computational load can be mitigated by using antisymmetric neural networks, achieved through the use of intermediate layers that implement permutation equivalence functions. This provides an expressible yet computationally feasible Ansatz.

[0007] In some implementations for intermediate layers (more specifically, each of a plurality of intermediate layers other than the final intermediate layer), the corresponding layer inputs include separate input streams for each of the electrons and for each of the electron pairs, the layer outputs include separate output streams for each of the electrons and for each of the electron pairs, and the intermediate layer is configured to operate on each input stream using a corresponding flow sublayer. For a first intermediate layer of a plurality of intermediate layers, the input stream for each electron can be an electronic feature of the electron, and the input stream for each electron pair can be a pair feature of the electron pair. This approach utilizes the streams for electrons and electron pairs to facilitate the computational requirements of the control system.

[0008] In some implementations, for each of a plurality of intermediate layers, a specific flow sublayer corresponding to each electron is configured to: receive an input stream for the corresponding electron from the corresponding layer input; generate a combined input for the corresponding electron from the input stream from the corresponding layer input; concatenate the input stream generated by the specific flow sublayer with the combined input to generate a cascaded input stream for the corresponding electron; and process the cascaded input stream to generate a cascaded output stream for the corresponding electron. (In other words, depending on what is considered as input and output, a specific flow sublayer may receive an input stream for the corresponding electron from the corresponding layer input; may process the input stream to generate an initial output stream; may generate a combined output for the corresponding electron from the initial output stream generated by the flow sublayer of the specific intermediate layer; and may concatenate the initial output stream generated by the specific flow sublayer with the combined output to generate a cascaded output stream for the corresponding electron.) The cascaded output stream may include an output stream for the corresponding electron and / or, for at least one of the intermediate layers, the output stream for the corresponding electron may include the cascaded output stream and a residual output stream generated by the flow sublayer for the corresponding electron in a previous intermediate layer.

[0009] In some implementations, processing the (cascaded) input stream to generate a cascaded output stream includes applying a linear transformation to the (cascaded) input stream to generate a transformed stream, and applying a nonlinear activation function to the transformed stream to generate a (cascaded) output stream. Generating a combined signal (input) for the corresponding electron may include, for each of a plurality of electron spins, calculating an average value of the input stream corresponding to the electron having the electron spin, and including the average value in the combined input. This may also include, for each of the plurality of electron spins, calculating an average value of the input stream corresponding to the pair of electrons including the corresponding electron and the electron having the electron spin, and including the average value in the combined input.

[0010] Antisymmetric neural networks can be configured to generate predictions from the output of the last intermediate layer in a plurality of layers. Generating predictions may include generating a corresponding (matrix) input for each of the plurality of determinants from the output of the last intermediate layer, determining a corresponding output for each determinant from the corresponding input for each determinant, and determining the prediction from the corresponding output for each determinant. Such matrix inputs may include a square matrix of (all) permutations of the output stream of each electron. In implementations, generating the corresponding inputs (to the determinants) may also include applying a linear transformation (specifically, different for spin-up and spin-down states) to the output of the last intermediate layer to generate a final transformed output, and applying each of a plurality of exponentially decaying envelopes to some or all of the final transformed output. This can help implement boundary conditions that force the wavefunction away from the nucleus to zero.

[0011] In this implementation, the system provides a mapping from electronic characteristics—e.g., the positions of the sets of spin-up and spin-down electrons—to the value of the wavefunction ψ of a chemical system—e.g., a molecule. Therefore, the wavefunction can be mapped or characterized, for example, to determine the associated energy, such as the ground-state energy of a chemical system—e.g., a molecule. The associated electron probability density is given by ψ. 2 Provided.

[0012] The system can be trained to optimize for varying energies, for example, minimizing the energy to determine the ground-state wavefunction. This can be done by optimizing the parameters of an antisymmetric neural network using stochastic gradient descent on an objective function that estimates the energy from the wavefunction.

[0013] The wave function of a chemical system—such as a molecule—can be used in many ways.

[0014] For example, wavefunctions can be used to design molecules or other chemical systems (on a computer), such as screening multiple composite structures by energy-based ground-state wavefunctions to assess which structures are likely to be relatively more stable than others. Optionally, molecules or other chemical systems can then be synthesized based on the structures identified as relatively more stable.

[0015] In another example, the wavefunction can be used to select synthetic routes for molecules or other chemical systems. For instance, there may be more than one possible synthetic route, and the ground-state energies of one or more intermediates in each synthetic route can be determined and compared to assess which synthetic route might be easier. Alternatively, there may be more than one possible synthetic route, and the energies of one or more different conformations of the same intermediate in each synthetic route can be determined and compared to assess which synthetic route might be easier (the techniques described herein can be used to determine the accurate wavefunction of a bent or twisted molecule). Optionally, the easier synthetic route can then be used to synthesize the molecule or other chemical system.

[0016] In another example, the synthetic route for a molecule or other chemical system may be known, but the reaction mechanism may be unknown. One or more wavefunctions and / or ground-state energies can be determined for one or more components of one or more steps in a hypothetical mechanism to assess the likelihood of the mechanism, and the mechanism of the reaction can be identified by comparing these. Once the reaction mechanism is known, the reaction can be improved by tweaking the mechanism; alternatively, a suitable mechanism can then be used to synthesize the molecule or other chemical system.

[0017] In another example, the conformation of a molecule or other chemical system can be identified by comparing the wavefunctions and / or energies of different assumed conformations. The molecule or other chemical system can then be modified to change the conformation or make the desired conformation more likely; alternatively, the molecule or other chemical system can then be synthesized with the desired conformation.

[0018] In another example, the wavefunction and / or ground-state energy can be determined for one or both of the ligand and its target. The target can be a biomolecular target, and the ligand can be a candidate drug or a catalyst. The wavefunction and / or ground-state energy can be used to predict which ligands will interact strongly with the target; one or more ligands can then be synthesized for real-world evaluation.

[0019] In another example, wavefunctions and energies can be determined for two or more different physical or electronic conformations of a molecule or other chemical system and for the difference between energies used to characterize the molecule / chemical system, for example, to identify unknown molecule / chemical systems or to design (and optionally then manufacture) molecule / chemical systems with specific electromagnetic absorption or emission properties.

[0020] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0021] Calculating the exact wavefunction of a chemical system of interest is infeasible because it is difficult to perform NP-hard calculations; however, approximations can be found, for example, using polynomial scaling algorithms. A challenge with many of these algorithms is the choice between a wavefunction approximation or Ansatz algorithm that requires a trade-off between efficiency and accuracy.

[0022] While neural networks generally demonstrate impressive capabilities as approximators of exact functions, problems involving electronic structure require wavefunctions that obey Fermi-Dirac statistics. This makes existing neural network structures unsuitable for approximating the wavefunctions (or other properties related to electronic structure) of chemical systems.

[0023] This specification introduces an antisymmetric neural network that can approximate the wavefunction (or other properties) of a multi-electron system in a manner that obeys Fermi-Dirac statistics. By utilizing the antisymmetric neural network, the described technique enables higher accuracy than other variable quantum Monte Carlo wavefunction approximators on a wide variety of atoms and small molecules without using data other than atomic positions and charges. The described technique outperforms other ab initio quantum chemistry methods, allowing for accurate and direct optimization of the wavefunctions of previously intractable molecules and solids.

[0024] Antisymmetric neural networks can be constructed and optimized to enable high-precision quantum chemical calculations for challenging systems. For example, antisymmetric neural networks enable the simple and straightforward VMC method to compete with DMC, AFQMC, and CCSD(T) methods for equilibrium geometries, and to outperform CCSD(T) for many imbalance geometries.

[0025] Advantageously, a single network architecture with a set of training parameters has been able to achieve high accuracy on many different chemical systems. Therefore, using an antisymmetric neural network means that it is not necessary to select a basis set or perform basis set extrapolation for new chemical systems, thus removing common error sources in computational quantum chemistry.

[0026] More generally, when the predictions of a neural network need to be antisymmetric, using an antisymmetric neural network—that is, a neural network that is antisymmetric about the input of the neural network—can lead to improvements in accuracy for many different tasks.

[0027] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0028] Figure 1 An example neural network system is shown.

[0029] Figure 2 An example architecture of an antisymmetric neural network is shown.

[0030] Figure 3 This is a flowchart of an example process for generating output for network input.

[0031] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0032] This specification describes a system implemented as a computer program on one or more computers at one or more locations that uses an antisymmetric neural network to process inputs. Generally, an antisymmetric neural network is a neural network that is antisymmetric with respect to its inputs. That is, if the relative positions of any two vectors in the network input are swapped, the output of the neural network after the swap will be equal in magnitude but opposite in sign to the output of the neural network before the swap.

[0033] A neural network can be configured to receive any digital data input from a variety of digital data inputs and generate any score from a variety of digital data inputs.

[0034] Specifically, in the following description, a neural network is described as being configured to receive data characterizing a chemical system—such as an atom, molecule, or other group of one or more nuclei and multiple electrons—as input, and to generate predictions of one or more properties of the chemical system as output.

[0035] However, neural networks can also be configured to process other kinds of data from other systems that include different components, where the aforementioned antisymmetric properties are beneficial. For example, the input to a neural network could be data characterizing the participants in a zero-sum game, and the output of the neural network could be data representing a predictive value function of the zero-sum game.

[0036] As a specific example, although the following description describes the output of a layer in an antisymmetric neural network as being based on the spin of electrons in the system, in different contexts the output can be based on different properties of the system's components.

[0037] Figure 1 An example neural network system 100 is shown. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0038] The neural network system 100 is a system for obtaining input data 102 that characterizes a system having multiple components—such as a chemical system having multiple electrons.

[0039] The neural network system 100 then uses an antisymmetric neural network 110 to process the input data 102. The antisymmetric neural network 110 is configured to process the input data 102 to generate predicted values ​​112 of one or more properties of the system as output. When the system is a chemical system, one or more properties may include the wave function of the chemical system. That is, the antisymmetric neural network 110 is configured to process the input data 102 characterizing the chemical system to generate an output 112 as an estimate of the wave function of the chemical system.

[0040] Because the wave function of any chemical system must obey Fermi-Dirac statistics, that is, the wave function must be antisymmetric if the positions and spin coordinates of any two electrons in the chemical system are interchanged simultaneously, the neural network 110 must also be antisymmetric in order to accurately estimate the wave function.

[0041] As another example, the predicted value can be or can include the ground state energy of a chemical system.

[0042] Input data 102 includes component features (e.g., electronic features) of each of a plurality of components and pair features of each of a plurality of pairs of the plurality of components.

[0043] As a specific example, for each electron in a chemical system, electronic features can include distance-based features, which are based on relative positions—such as the three-dimensional relative positions of the electrons and each atomic nucleus in the chemical system. Specifically, for a given electron and a given atomic nucleus, electronic features can include a difference vector that measures the difference between the positions of the given electron and the given atomic nucleus within the chemical system—that is, equal to r. i –R I The distance vector, where r is the distance vector. i R is a vector representing the position of the i-th electron in a chemical system. I This is a vector representing the position of the i-th atomic nucleus within the chemical system. Optionally, in addition to the distance vector, the electronic feature may also include the absolute distance between a given electron and a given atomic nucleus, i.e., the norm of the distance vector.

[0044] Further optionally, for each electron in the chemical system, the electronic characteristics may also include characteristics that identify the electron's spin, i.e., characteristics that identify whether the electron is an up-spin electron or a down-spin electron.

[0045] As another specific example, for each pair of electrons in a chemical system, the pair characteristics may include distance-based characteristics based on the relative positions of the electrons in that pair within the chemical system. Specifically, the pair characteristics may include a difference vector measuring the positions of the electrons in that pair within the chemical system, i.e., equal to r i -r j The distance vector, where r is the distance vector. iIt is a vector representing the position of the first electron i in this pair within the chemical system, and r j This is a vector representing the position of the second electron j in the pair within the chemical system. Optionally, in addition to the distance vector, the pair feature may also include the absolute distance between the two electrons in the pair, i.e., the norm of the distance vector.

[0046] Since the distance between two electrons, or the distance between an electron and an atomic nucleus, is a non-smooth function at zero, the neural network 110 can represent the non-smooth behavior of the wavefunction—the wavefunction tip—when two particles coincide. Accurately modeling these tips is important for correctly estimating the energy and other properties of a chemical system. Therefore, including a distance vector and optionally an absolute distance as input can help the neural network 110 accurately estimate one or more output properties of a chemical system, although the wavefunction can still be approximated without them. Furthermore, directly including the absolute distance between particles as input removes the need to include a separate Jastrow factor (representing electron-electron correlation) after the determinant.

[0047] Further optionally, for each pair of electrons in the chemical system, the pair feature may also include a feature that identifies the spin of each electron in the pair, that is, a feature that identifies whether each electron in the pair is a spin-up electron or a spin-down electron.

[0048] As described above, the antisymmetric neural network 110 maps input data 102 to predicted values ​​112 of one or more properties of the system.

[0049] The opposing neural network 110 has multiple intermediate layers, and the predicted value 112 is generated from the output of the last intermediate layer in the neural network 110. The intermediate layers are referred to as "intermediate" because the output generated by the intermediate layers—that is, the output of the last intermediate layer in the neural network 110—is an intermediate feature used to generate the predicted value 112, rather than the predicted value 112 directly as the output of the neural network 110.

[0050] To ensure that the output of the neural network 110 is antisymmetric, each of the intermediate layers is configured to generate the corresponding layer output for that intermediate layer by applying the corresponding permutation equivalence function to the corresponding layer input of that intermediate layer.

[0051] In this context, the substitution equivalence function means that if two layers of input features corresponding to different electrons with the same spin are substituted, then the corresponding output electron features of different layers are substituted in the same way. In other words, if the input streams of two spin-up electrons are swapped, then the output streams of two spin-up electrons will also be swapped relative to the output streams that would be generated if the input streams were not swapped. Similarly, if the input streams of two spin-down electrons are swapped, then the output streams of two spin-down electrons will also be swapped. Mathematically, the function is equivalent for all subsets of the set of electron substitutions, that is, the set of all substitutions in which the substituted electrons have the same spin.

[0052] Once the final intermediate layer has generated its layer output, the neural network 110 generates from the layer output of the final intermediate layer the corresponding input for each of the multiple functions that compute the determinants of different matrices, determines the corresponding output of each determinant from the corresponding input of the determinant, and determines the predicted value from the corresponding output of each determinant.

[0053] Because each intermediate layer used to generate the input to the determinant function is permutationally equivalent, and because the output of a neural network depends on the determinant computed from the output of these permutationally equivalent functions, the output of neural network 110—i.e., the predicted value 112—is antisymmetric with respect to the interchange of features between any two components of the input system.

[0054] The output generated from the features of the chemical system and the architecture of the neural network 110 are referenced below. Figure 2 and Figure 3 To describe in more detail.

[0055] Figure 2 An example architecture of an antisymmetric neural network 110 is shown.

[0056] In particular, Figure 2 In the example, input data 102 is generated for chemical system 200, and neural network 110 processes the input data 102 to generate a predicted value 212 of the wave function of chemical system 200.

[0057] Although for the sake of explanation Figure 2 The diagram shows three electrons r1, r2, and r i And a single atomic nucleus R1, but it should be understood that chemical system 200 may have more electrons, more atomic nuclei, or both than shown in the figure.

[0058] Input data 102 includes the electronic characteristics of each of the n electrons in chemical system 200 and the pair characteristics of each (different) pair of electrons in chemical system 200.

[0059] exist Figure 2In the input data 102 for the i-th electron, the electronic characteristics are represented as follows: The electron pair characteristics of electron pairs i and j are represented as follows:

[0060] Typically, n electrons are divided into spin-up electrons (n↑) and spin-down electrons (n↓). Before the input data 102 is processed by the neural network 110, the spin-up electrons are ordered from 1 to n↑, while the spin-down electrons are ordered from 1 to n↓.

[0061] As mentioned above, for each atomic nucleus R in a chemical system, the electronic characteristics of a given electron r include a distance vector and an absolute distance. The pair characteristics of any given electron pair also include a distance vector and an absolute distance.

[0062] like Figure 2 As shown, the neural network 110 uses L intermediate layers 210 to process the input data 102.

[0063] Each of the L intermediate layers receives layer input and generates layer output.

[0064] For each of the first L-1 intermediate layers, the layer output is used to generate the layer input to the next layer.

[0065] For the final intermediate layer, namely intermediate layer L, the layer output of the intermediate layer is used to generate the input to multiple determinant functions, as will be described in more detail below.

[0066] Typically, for each layer l in the intermediate layers, the corresponding layer input of the intermediate layer includes a separate input stream for each electron. and a separate input stream for each pair of electrons

[0067] Similarly, for each intermediate layer l, the layer output includes a separate output stream for each electron. and a separate output stream for each pair of electrons

[0068] For the first intermediate layer among the L intermediate layers, i.e., intermediate layer 1, the input current of each electron is the electronic characteristic of that electron. Furthermore, the input current for each electron pair is the electronic characteristic of that electron pair.

[0069] For each intermediate layer after the first intermediate layer 1 in the L intermediate layers, the input flow of each electron is the output flow of the electrons generated in the preceding intermediate layer, and the input flow of each electron pair is the output flow of the electron pair generated in the preceding intermediate layer.

[0070] Each of the intermediate layers is configured to operate on each input stream with a corresponding flow sublayer, which is configured to generate an output stream for the corresponding electron or electron pair from the input stream of the corresponding electron or electron pair.

[0071] As will be apparent from the following description, due to the use of the input stream and the corresponding sub-layer, each intermediate layer applies a permutation equivalent function to its corresponding layer input. In other words, if the order of the input streams is modified, each intermediate layer will still return the same layer output due to the use of the sub-layers.

[0072] Operations occurring within a given intermediate layer Figure 2 The operations shown are illustrated and described below with reference to the operations performed by intermediate layer l+1. However, more generally, these operations are performed by each intermediate layer (in some implementations, except for the last intermediate layer described below).

[0073] Within the intermediate layer l+1, each sublayer corresponding to one of the electrons (i.e., not one of the electron pairs) is configured to receive an input stream for the corresponding electron from the corresponding layer input, generate a combined input for the corresponding electron from the input stream in the corresponding layer input, and generate an output stream for the corresponding electron from the input stream for the electron and the combined input for the electron.

[0074] Typically, sublayers generate cascaded inputs for corresponding electrons by cascading the input stream for the corresponding electron with the combined input for the corresponding electron. The cascaded inputs are then processed to generate cascaded output streams for the corresponding electrons.

[0075] Specifically, the sublayer processes the cascaded input by applying a linear transformation to the cascaded input—for example, by multiplying the cascaded input by the weight matrix used by the sublayer and then optionally adding the learned bias, or by performing a convolution between the sublayer's weight kernel and the cascaded input—to generate a transformed stream. The sublayer can then apply a nonlinear activation function—for example, tanh or a rectified linear unit function—to the transformed stream to generate a cascaded output stream (the tanh nonlinearity is smooth, which helps with numerical stability).

[0076] In some implementations, the cascaded output stream is the output stream corresponding to the electron.

[0077] However, in some other embodiments, at least some intermediate layers generate an output stream for a given electron, which includes a cascaded output stream and a residual output stream generated by the sublayer for the corresponding electron in the previous intermediate layer—that is, an output stream from the previous intermediate layer (or, for the first intermediate layer, the electronic characteristics of the electron). For example, the output stream of a given electron can be a cascade, sum, or average of cascaded output streams and residual output streams.

[0078] like Figure 2 As shown, in order to generate a combined input for a given electron from the layer output of layer l, the sublayer corresponding to the electron calculates the average value of the input stream corresponding to the electron with the electron spin for each possible electron spin. In other words, the sublayer calculates (i) the input stream corresponding to the spin-up electron. The average value g l↑ and (ii) the input flow corresponding to spin-down electrons The average value g l↓ .

[0079] The sublayer then includes the average value from the combined input.

[0080] Optionally, for each possible electron spin, the sublayer can also calculate the corresponding average value of the input stream corresponding to the electron pair, which includes the corresponding electron and the electron pair with the possible electron spin. In other words, the sublayer calculates (i) the input stream corresponding to the electron pair, which includes the corresponding electron and the spin-up electron. average And (ii) the input stream corresponding to electron pairs including corresponding electrons and spin-down electrons. average

[0081] The sublayer then includes the average value from the combined input.

[0082] For example, the combined input could be a calculated average—for example, g. l↑ g l↓ , and — cascade.

[0083] Unlike individual electrons, for each electron pair, the specific flow sublayer corresponding to that pair is configured to receive a corresponding input stream from the corresponding layer input of the intermediate layer, and process the input stream to generate an initial output stream; that is, no combined or cascaded inputs are generated.

[0084] In some implementations, the initial output stream is the output stream used for the corresponding response.

[0085] In some other implementations, for at least one of the intermediate layers, the output stream for a given pair includes an initial output stream for that pair and a residual output stream generated by the flow sublayer for the given pair in the previous intermediate layer. For example, the output stream for the given pair may be a concatenation, summation, or averaging of the cascaded output stream and the residual output stream.

[0086] Therefore, information from other single electron streams and convections is fed into the single electron streams. However, to reduce the computational overhead of the neural network 110, information is not transferred between convections. These can then be implemented as multilayer perceptrons operating in parallel. Due to the architecture of the neural network 110, even with this reduction in computational overhead, the neural network 110 can still generate accurate predictions.

[0087] As described above, neural network 110 uses the layer output of the last intermediate layer L to generate the predicted value as the output of neural network 110.

[0088] Specifically, the neural network 110 generates the corresponding input of each of the multiple determinants from the layer output of the last intermediate layer, and determines the corresponding output of the determinant from the corresponding input of each determinant.

[0089] Typically, the input to each determinant is a square matrix of values, and the output of a determinant is the determinant of a square matrix. As will be seen from the description below, since the input to each determinant is a different matrix, each determinant will typically output a different scalar determinant value.

[0090] More specifically, and as Figure 2 As shown, the multiple determinants comprise multiple pairs of determinants, wherein each pair of determinants includes one determinant operating on the matrix generated from the output stream for spin-up electrons and another determinant operating on the matrix generated from the output stream for spin-down electrons. Specifically, in Figure 2 In the example, one determinant in each pair operates on an n↑ multiplied by n↑ matrix, while the other determinant operates on an n↓ multiplied by n↓ matrix. Each matrix can be antisymmetric in the sense that it interchanges two rows or two columns of the electron characteristic interchange matrix for any two electrons in the input data.

[0091] To generate an input matrix of a given determinant, the neural network 110 applies a final spin-dependent linear transformation to the layer output of the last intermediate layer (and more specifically, to the output stream of electrons in the layer output of the last intermediate layer) to generate a final transformed output, and then generates the input of the given determinant, i.e., the determinant input matrix, based on applying each of a plurality of exponentially decaying envelopes to some or all of the final transformed outputs. The application of exponentially decaying envelopes forces the wavefunction (ψ) away from the atomic nucleus to a boundary condition of zero.

[0092] The linear transformation is called the spin-dependent transformation because within each pair of determinants, a different transformation is applied to the spin-up electrons than to the spin-down electrons to generate the input to the determinant of that pair.

[0093] In other words, for the k-th pair of determinants, in order to generate the elements ij of the input matrix for the determinants of that pair, the system generates the elements ij in the final transformation output for the elements ij, as follows:

[0094]

[0095] Where α represents spin-up or spin-down, and and These are the weights and biases of the final linear transformation of the k-th pair of determinants (and, due to their dependence on α, differ depending on whether the linear transformation is applied to a spin-up or spin-down matrix), and It is the output stream used for the j-th spin-up or spin-down electron.

[0096] Then, the system... Applying a weighted sum of exponentially decaying envelopes, the elements ij in the input matrix are generated as follows:

[0097]

[0098] Where the sum is at the radius of the chemical system, and and These are the parameters that are learned, which control the anisotropy to decay to zero away from each atomic nucleus.

[0099] Then, the neural network 110 determines the predicted value from the corresponding output of each determinant. For example, the neural network 110 may determine the predicted value as a weighted sum of the products of the determinant outputs of the determinants in that pair of determinants.

[0100] As can be seen from the above description, in some embodiments, only the output streams corresponding to electrons, rather than electron pairs, in the layer output of layer L are used to calculate the predicted values. Therefore, in these embodiments, the final intermediate layer L may not include sublayers corresponding to electron pairs (and only include sublayers corresponding to individual electrons in the chemical system), and thus can generate layer outputs that include only output streams for individual electrons and not for any electron pairs.

[0101] In some cases, the determinant can be computed in the logarithmic domain, and the output of the neural network can therefore be the absolute value of the wavefunction and the logarithm of the sign of the wavefunction. This can help improve numerical stability. In this case, the energy is computed in the logarithmic domain as a function of logψ when the system is trained (see later).

[0102] Figure 3This is a flowchart of an example process 300 for generating predicted values. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed neural network system—for example, Figure 1 The neural network system 100 can execute process 300.

[0103] The system obtains input data characterizing a chemical system having multiple electrons (step 302). The input data includes electronic features for each of the multiple electrons and pair features for each of the multiple pairs of electrons. For example, the input data may include pair features for each possible electron pair or a subset of possible electron pairs that can be generated by selection from the multiple electrons in the chemical system.

[0104] The system uses an antisymmetric neural network to process the input data to generate predicted values ​​of the system's properties (step 304).

[0105] More specifically, the antisymmetric neural network is configured to process the input data through multiple intermediate layers to generate the layer output of the final intermediate layer (step 306), and then generate a prediction of the property from the layer output of the final intermediate layer (step 308).

[0106] This system can train an antisymmetric neural network to generate accurate predictions of the properties of the subject system on training data, thereby minimizing the objective function using a training technique based on appropriate gradient descent. The objective function can be any appropriate objective function for a specific task, to which the neural network is trained to perform that specific task, which typically measures the error between the predictions generated by the neural network for training input data and known or target values ​​of the properties of the system represented by the training input data.

[0107] As a specific example, when the property is the wave function of a chemical system, the objective function can be a loss function that measures the energy expectation of the wave function Ansantz. That is, the system can be trained to train a neural network to generate a predicted wave function value that minimizes the energy expectation of the wave function Ansantz.

[0108] When calculating the determinant in the logarithmic field, the gradient of the objective function (energy) L with respect to the parameters θ of the antisymmetric neural network can be determined as follows:

[0109]

[0110] Where X is the set of 3D coordinates of all electrons. And the E evaluated for ψ(X) L It can be determined as

[0111]

[0112] Where V(X) is the potential energy of state X, and index i operates in each of the three dimensions for each electron of X.

[0113] As an example of a gradient-based technique that can be used to train neural networks to optimize for such an objective, the system can use a version of the Kronecker Factorized Approximate Curvature (KFAC) to train the neural network, which is an approximation of the natural gradient descent technique suitable for training neural networks.

[0114] For illustrative purposes only, in some implementations, the number of intermediate layers can be small, for example, less than 10 or less than 5. The width used for the electron flow array can be less than 100 hidden units, although this can depend on the number of electrons, and generally using more hidden units can increase accuracy.

[0115] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, this means that software, firmware, hardware, or a combination thereof are already installed on the system, which, when in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0116] Embodiments of the subject matter and functional operation described in this specification may be implemented as digital electronic circuits, as tangibly implemented computer software or firmware, as computer hardware including the structures disclosed in this specification and their structural equivalents, or as a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or to control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0117] The term "data processing apparatus" refers to data processing hardware and includes all types of devices, apparatuses, and machines for processing data, including, by way of example, programmable processors, computers, or multiple processors or computers. The apparatus may be or also include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an operating environment for computer programs, such as code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0118] A computer program—which may also be referred to or described as a program, software, software application, app, module, software module, script, or code—can be written in any form of programming language, including compiled or interpreted languages, or declarative or programmatic languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data—for example, one or more scripts stored in a markup language document—stored in a single file dedicated to the program involved, or stored in multiple coordinating files—for example, a file that stores portions of one or more modules, subroutines, or code. A computer program can be deployed to run on one computer or on multiple computers located at a site, or distributed across multiple sites and interconnected via a data communication network.

[0119] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way or at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an index database may include multiple collections of data, each of which can be organized and accessed differently.

[0120] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0121] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry—such as FPGAs or ASICs—or a combination of special-purpose logic circuitry and one or more programmed computers.

[0122] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving and / or transferring data, such as disks, magneto-optical disks, or optical disks. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game controller, global positioning system (GPS) receiver, or portable storage device—e.g., a universal serial bus (USB) flash drive, etc.

[0123] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0124] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device—such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor—to display information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device—such as a smartphone running a messaging application—and receiving response messages from the user in exchange.

[0125] The data processing apparatus for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for handling the common and computationally intensive portions of machine learning training or production—i.e., inference—workloads.

[0126] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0127] Embodiments of the subject matter described herein may be implemented in a computing system, including backend components—such as a data server; or middleware components—such as an application server; or frontend components—such as a client computer having a graphical user interface, web browser, or app through which a user can interact with the implementation of the subject matter described herein; or any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected via any form or medium of digital data communication—such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0128] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The relationship between clients and servers is generated by computer programs running on their respective computers, and they have a client-server relationship. In some embodiments, the server sends data—e.g., HTML pages—to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from it. Data generated at the user device, such as the results of user interactions, may be received at the server from the device.

[0129] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments may also be combined or implemented in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0130] Similarly, although the operations depicted in the drawings and referenced in the claims are described in a specific order, this should not be construed as requiring the operations to be performed in the specific order or sequence shown, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0131] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for processing input data using an antisymmetric neural network, comprising processing each of multiple chemical systems having multiple atomic nuclei and multiple electrons: Obtain input data characterizing the chemical system, the input data including electronic characteristics for each of the plurality of electrons and pair characteristics for each of the plurality of pairs of electrons; and The input data is processed using an antisymmetric neural network with multiple intermediate layers, wherein: The antisymmetric neural network is configured to process the input data to generate predicted values ​​as output, including one or more properties of the chemical system, such as the wavefunction and / or ground-state energy. Each of the plurality of intermediate layers of the antisymmetric neural network is configured to generate a corresponding layer output for the intermediate layer by applying a corresponding permutation equivalence function to the corresponding layer input of the intermediate layer. Generating the predicted value includes: The output of the last intermediate layer generates the corresponding input for each of the multiple determinants; Determine the corresponding output of the determinant from the corresponding input used for each determinant; and The predicted value is determined from the corresponding output of each determinant in the determinant.

2. The method according to claim 1, wherein, For the intermediate layer: The corresponding layer inputs include separate input streams for each of the electrons and for each electron pair. The layer output includes separate output streams for each of the electrons and for each electron pair. The intermediate layer is configured to operate on each input stream using the corresponding stream sublayer.

3. The method according to claim 2, wherein, For the first intermediate layer of the plurality of intermediate layers, the input stream for each of the electrons is the electronic feature for that electron, and the input stream for each of the electron pairs is the pair feature for that electron pair.

4. The method according to claim 2, wherein, For each of the plurality of intermediate layers, the sub-layer corresponding to each of the electrons is configured as follows: The input stream for the corresponding electron is received from the corresponding layer input; A combined input for the corresponding electron is generated from the input stream from the corresponding layer input; The input stream generated by the sublayer is cascaded with the combined input to generate a cascaded input stream for the corresponding electron; as well as The cascaded input stream is processed to generate a cascaded output stream for the corresponding electron.

5. The method according to claim 4, wherein, The cascaded output stream is the output stream used for the corresponding electron.

6. The method according to claim 4, wherein, For at least one of the intermediate layers, the output stream for the corresponding electron includes the cascaded output stream and the residual output stream generated by the flow sublayer for the corresponding electron in the previous intermediate layer.

7. The method according to claim 4, wherein, Processing the cascaded input streams to generate cascaded output streams includes: A linear transformation is applied to the cascaded input stream to generate a transformed stream; and A nonlinear activation function is applied to the transform stream to generate the cascaded output stream.

8. The method according to claim 4, wherein, Generating the combined input for the corresponding electron includes: For each of the multiple electron spins, calculate the average value of the input current corresponding to the electron with the electron spin; and The average value is included in the combined input.

9. The method according to claim 8, wherein, Generating the combined input for the corresponding electron includes: For each of the plurality of electron spins, calculate the average value of the input stream corresponding to the electron pair including the corresponding electron and the electron having the electron spin; and The average value is included in the combined input.

10. The method according to claim 2, wherein, For each of the plurality of intermediate layers, the corresponding flow sublayer for each of the pairs is configured as follows: Receive the input stream for the corresponding layer input from the intermediate layer; and The input stream is processed to generate an initial output stream.

11. The method according to claim 10, wherein, The initial output stream is the output stream used for the corresponding action.

12. The method according to claim 10, wherein, For at least one of the intermediate layers, the output stream for the corresponding layer includes the initial output stream and the residual output stream generated by the sublayer for the corresponding layer in the previous intermediate layer.

13. The method according to claim 1, wherein, The electronic characteristics include a difference vector that measures the positional differences between the electrons and atomic nuclei within the system.

14. The method according to claim 13, wherein, The electronic characteristic includes the norm of the difference vector.

15. The method according to claim 1, wherein, The electronic characteristics include features that identify the spin of the electron.

16. The method according to claim 1, wherein, The pair features include a difference vector that measures the positional difference between electrons in the pair within the system.

17. The method according to claim 16, wherein, The pair of features includes the norm of the difference vector.

18. The method according to claim 1, wherein, The pair features include features that identify the spin of the electrons in the pair.

19. The method according to claim 1, wherein, when also subordinate to claim 14, Generating the corresponding input for each of the multiple determinants from the layer output of the last intermediate layer includes: The final spin-dependent linear transformation is applied to the layer output of the last intermediate layer to generate the final transformed output; and The corresponding input for each of the plurality of determinants is generated by applying each of the multiple exponentially decaying envelopes to some or all of the final transform output.

20. A system comprising one or more computers and one or more storage devices and storage instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform operations according to any one of claims 1 to 19.

21. A computer-readable storage medium for storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 19.

Images