Neural networks for processing graph data

Abstract

The present invention relates to neural networks for processing graph data. Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, are configured to: receive graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges; generate vertex input data representing properties of each vertex in the input graph and pairing input data representing properties of pairs of vertices in the input graph from the graph data; and generate order-invariant features of the input graph using a neural network, wherein the neural network comprises: a first subnetwork configured to generate a first alternative representation of the vertex input data and a first alternative representation of the pairing input data from the vertex input data and the pairing input data; and a combining layer configured to receive the input alternative representations and process the input alternative representations to generate the order-invariant features.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201680034801.7, filed on August 12, 2016. Technical Field

[0003] This specification relates to neural networks for processing graph data. Background Art

[0004] A neural network is a machine learning model that uses one or more layers of the model to generate an output (e.g., a classification) for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input for the next layer in the network (i.e., the next hidden layer or output layer of the network). Each layer of the network generates an output from the received input according to the current values ​​of a set of corresponding parameters. Summary of the Invention

[0005] In general, this specification describes neural networks for processing graph data.

[0006] Generally speaking, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following acts: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pairing input data representing characteristics of pairs of vertices in the input graph; and generating order-invariant features for the input graph using a neural network, wherein the neural network comprises: a first subnetwork configured to generate, from the vertex input data and the pairing input data, a first alternative representation of the vertex input data and a first alternative representation of the pairing input data; and a combination layer configured to receive the input alternative representations and process the input alternative representations to generate the order-invariant features.

[0007] Implementations may include one or more of the following features. The order-invariant features are processed by one or more neural network layers to generate a classification of the graph data. The graph data represents a molecule, and each vertex in the graph is an atomic element in the molecule and each edge is a type of bond between two atomic elements in the molecule. The order-invariant features are processed to determine a likelihood that the molecule represented by the graph data will bind to a specific target molecule. The characteristics of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of a ring of which the atomic element is a part. The characteristics of each pair of vertices include at least one or more of the following: the type of bond between the atomic elements represented by the vertex or the length of a path between the atomic elements represented by the vertex. The multiple layers further include one or more other subnetworks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data, and provide the second alternative representation as input to the combination layer. The first sub-network is configured to, for each vertex in the input graph: apply a first vertex function to the vertex input data of the vertex to generate a first vertex output; apply a second vertex function to the pair input data of each pair including the vertex to generate a corresponding output of the each pair; combine the corresponding output of each pair with a swap function to generate a second vertex output; and generate the first alternative representation of the vertex input data from the first vertex output and the second vertex output. The first sub-network is configured to, for each pair of vertices in the input graph: apply the first pair function to the pair input data of the pair to generate a first pair output; generate a second pair output from the vertex input data of the vertices in the pair; and generate the first alternative representation of the pair input data of the pair from the first pair output and the second pair output. Generating the second pairing output includes: applying a non-commutative function to vertex input data of a first vertex in the pair as a first input to the non-commutative function and vertex input data of a second vertex in the pair as a second input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to vertex input data of the second vertex as the first input to the non-commutative function and vertex input data of the first vertex as the second input to the non-commutative function to generate a second non-commutative output; and applying a commutative function to the first non-commutative output and the second non-commutative output to generate the second pairing output.Generate triplet input data representing characteristics of triplets of vertices in the graph from the graph data; wherein the first subnetwork of the plurality of layers is configured to generate a first alternative representation of the triplet input data; and wherein the first subnetwork is further configured to, for each pairing of vertices: apply a second pairing function to the triplet input data for each triple where the pairing is a first element and a third element of a triple to generate a corresponding output for each triple; combine the corresponding output for each triple with a swap function to generate a third pairing output; and wherein generating the first alternative representation of the pairing input data includes generating the first alternative representation of the pairing input data from the first pairing output, the second pairing output, and the third pairing output. Generating triplet input data representing characteristics of triplets of vertices in the graph from the graph data; wherein the first subnetwork is further configured to, for each triplet of vertices in the input graph: apply a first triplet function to the triplet input data of the triplet to generate a first triplet output; generate a second triplet output from the paired input data of the vertex pairs in the triplet; and generate a first alternative representation of the triplet input data from the first triplet output and the second triplet output. Generating the second triplet output includes: identifying a first pair of vertices including the first vertex and the second vertex in the triplet; identifying a second pair of vertices including the first vertex and the third vertex in the triplet; identifying a third pair of vertices including the second vertex and the third vertex; applying a non-commutative function to the paired input data of the first pair of vertices as the first input of the non-commutative function, the paired input data of the second pair of vertices as the second input of the non-commutative function, and the paired input data of the third pair of vertices as the third input of the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the paired input data of the third pair of vertices as the first input of the non-commutative function, the paired input data of the second pair of vertices as the second input of the non-commutative function, and the paired input data of the first pair of vertices as the third input of the non-commutative function to generate a second non-commutative output; and applying a commutative function to the first non-commutative output and the second non-commutative output to generate the second triplet output.

[0008] Certain embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A neural network can generate a classification from input data representing an undirected graph of vertices connected by edges. Typically, converting a graph into data to be processed by a neural network requires sorting the graph in a particular order. Some neural networks can output different classifications for different sequences of a graph, even if the elements of the graph remain the same. The neural networks described in this specification allow the graph of vertices to be sorted in any order while still having the same neural network output classification. For example, a neural network can receive a graph representing a molecule and efficiently process the graph to generate an output, such as a classification defining the likelihood that the molecule will bind to a particular target molecule. As a result of making the order of the elements of the network's output graph independent of the order in which they are represented as input, the neural network must learn fewer parameters, which results in better performance and a simpler model.

[0009] It should be understood that various aspects can be implemented in any convenient form. For example, various aspects and embodiments can be implemented by a suitable computer program that can be carried on a suitable carrier medium, which can be a tangible carrier medium (such as a disk) or an intangible carrier medium (such as a communication signal). Various aspects can also be implemented using suitable equipment, which can take the form of a programmable computer running a computer program. The details of one or more embodiments of the subject matter of this specification are set forth in the following figures and description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An example system for processing graphics data is illustrated.

[0011] Figure 2 is a flow chart of an example method for processing graph data using a neural network.

[0012] Figure 3 is a flow chart of an example method for generating an alternative representation of vertex input data.

[0013] Figure 4 is a flow chart of an example method for generating alternative representations of paired input data.

[0014] Figure 5 is a flow chart of an example method for generating paired output.

[0015] Figure 6 is a flow chart of an example method for generating another pairing output.

[0016] Figure 7 is a flow chart of an example method for generating an alternative representation of triplet input data.

[0017] Figure 8 is a flowchart of an example method for generating triplet output.

[0018] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] Figure 1 Illustrated is an example system 100 for processing graphics data. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0020] System 100 includes a neural network 101 and a graph data parser 104. Graph data parser 104 may receive graph data 102, which represents an undirected graph having vertices connected by edges. Graph data parser 104 generates input data for neural network 101 from the graph data, and neural network 101 processes the input data to generate a classification 112 for the graph.

[0021] In some embodiments, a graph represents a molecule—the vertices of the graph represent atomic elements in the molecule and the edges represent bonds connecting the atomic elements together. For example, a water molecule (H2O) can be represented using graph data as having three vertices and two edges connecting those vertices: HOH, and the neural network 101 can process the graph representing the water molecule to generate a score representing the likelihood that the water molecule will bind to a target protein.

[0022] The graph data may also include additional data about the vertices and edges. For example, the additional data may specify, for each edge connecting a pair of vertices, the type of bond represented by the edge, such as a single bond, a double bond, or a triple bond. The additional data may specify the properties of the atomic elements represented by the vertices, which will be referred to below. Figure 2 This is described further.

[0023] The graphics data parser 104 generates vertex input data 114, pair input data 116, and optionally triple input data 118 from the graphics data 102, referring to Figure 2 Describe all of these.

[0024] The sub-network 106 in the neural network 101 processes the vertex input data 114, the paired input data 116, and the triple input data 118 to generate an alternative representation of the vertex input data 124, an alternative representation of the paired input data 122, and an alternative representation of the triple input data 120. The sub-network 106 applies multiple functions to the input data 102-124 to generate the alternative representations such that order invariance is maintained. Figures 3 to 8The use of sub-network 106 to generate alternative representations is further described.

[0025] In some embodiments, rather than having only one subnetwork 106, neural network 101 also includes one or more other subnetworks. In these embodiments, the subnetworks are arranged in a sequence. The subnetworks can collectively process alternative representations to generate additional alternative representations of vertex, pair, and triple input data. That is, each subnetwork following subnetwork 106 in the sequence processes the alternative representations generated by the previous subnetwork in the sequence to generate another set of alternative representations. The last subnetwork in the sequence can provide the alternative representations generated by that subnetwork as input to combination layer 108.

[0026] The combination layer 108 can process alternative representations, such as alternative representations 120-124 or the output alternative representation from the last sub-network, to generate order-invariant features of the graph data 102. In some embodiments, if there are multiple sub-networks, the combination layer 108 processes the alternative representation output from each sub-network.

[0027] The combining layer 108 can generate an output based on moments obtained from the input alternative representations. A moment can be a measure of the shape of the alternative representation. For example, the first moment of a particular alternative representation can be the mean, the second moment can be the variance, and the third moment can be the skewness. The combining layer 108 can obtain moments from each input alternative representation. The combining layer 108 can then apply a set of parameters to each moment, for example, calculating a dot product of the set of parameters and each moment, to generate the output of the combining layer 108, namely, order-invariant features of the graph data. The set of parameters can be adjusted during training, which will be described below. In some embodiments, the combining layer 108 can generate order-invariant features by calculating a histogram of the values ​​of each dimension in the alternative representations 120-124.

[0028] Order-invariant features are features that remain the same regardless of the order in which the graph is sorted and input to neural network 101. Classifying order-invariant features of graphs can reduce errors during classification. As an example, a molecule may have three atomic elements: X, Y, and Z, occupying a multidimensional space. The graph data for the molecule may be sorted as XYZ in a first orientation, and may also be sorted as ZYX in a second orientation. Due to the order in which the graph is sorted, some neural networks may generate different features for differently sorted molecules. However, neural network 101 will generate the same order-invariant features for molecules sorted in the first orientation and molecules sorted in the second orientation. Processing graph data (regardless of how it is sorted) through the layers of neural network 101 as described herein enables the generation of order-invariant features of the graph data, and therefore, enables consistent classification of graph data sorted in any manner.

[0029] The classification layer 110 can then process the output of the combination layer 108 to generate a classification 112 for the graph data 102. For example, the classification layer 110 can apply a set of corresponding parameters to the input of the layer to generate the classification 112. In some embodiments, rather than generating the classification 112 at one layer (i.e., the classification layer 110), the neural network 101 can use multiple layers arranged in a sequence to generate the classification 112 from the order-invariant features. In some embodiments, if the graph data 102 represents a molecule, the classification 112 can be a score representing the likelihood that the molecule represented by the input graph will bind to a particular target molecule.

[0030] To perform classification, system 100 can train neural network 101 using conventional neural network training techniques. For example, neural network 101 can be trained on training data that specifies molecules and, for each molecule, specifies whether the molecule binds to a specific target molecule (e.g., a target protein). System 100 can adjust the parameters used in each layer and subnetwork of neural network 101 by backpropagating errors. After training, neural network 101 can classify input graph data to generate a classification.

[0031] Figure 2 is a flowchart of an example method for processing graph data at a neural network. For convenience, the method 200 will be described with respect to a system (e.g., a system having one or more computing devices executing software to implement the method 200). Figure 1 The method 200 is described with reference to the system 100 ).

[0032] The system receives graph data representing an input graph (step 202). The graph includes a plurality of vertices connected by edges. The graph also corresponds to additional data describing properties of the vertices and edges. For example, if the graph is molecular, the corresponding additional data may specify the properties of the atomic elements represented by each vertex, the properties of the bonds between the atomic elements, or other molecular properties.

[0033] The system generates vertex input data and pairing input data from the graph data (step 204). The vertex input data and pairing input data can be provided to the subnetwork. To generate the vertex input data, the system can identify vertices in the graph. The vertex input data can also include additional data about the vertex. For example, in the context of molecules, the additional data can include whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, the size of the ring of which the atomic element is a part, or other atomic element properties.

[0034] To generate pairing input data, the system can identify pairs of vertices in a graph. Each vertex can be paired with any other vertex in the graph to form a pair. The pairing input data can also include additional data about the pairs of vertices. For example, the additional data can include the type of bond between the atomic elements represented by the vertices or the length of the path between the atomic elements represented by the vertices. The pairing input is unordered. If X and Y are vertices, then the pairing input for XY is the same as YX.

[0035] In some embodiments, the system generates triple input data from graph data. Similar to generating vertex input data and pair input data, the system can identify triplets of vertices in a graph. Triple input data can also include additional data for the vertex triplets. For example, in the context of molecules, the additional data can include the length of the path from the first vertex to the second vertex in the triplet, the length of the path from the second vertex to the third vertex in the triplet, or other molecular characteristics. Triple input is only partially unordered. An input of XYZ must be the same as ZYX, but can be different from YXZ.

[0036] Vertex input data, pairing input data and triple input data can each be a vector of numbers representing the input graph. For example, if the input graph represents a molecule, the vertex input data can represent the type of molecule by the number of atoms. Pairing input data can represent a pair of specific vertices by mapping to the position of the pair of vertices in the input graph and one or more numbers of the type of the bond between the pair of vertices. In some embodiments, pairing input data is a vector, and the position in the vector corresponds to the position in the input graph like an adjacency matrix. Similarly, triple input data can represent a specific vertex triple by mapping to one or more numbers of the position of the vertex triples in the input graph. Other characteristics of the input graph can be mapped to numerical values ​​in a similar manner. In some embodiments, the characteristics of a vertex, a pairing of a vertex or a triple of a vertex are represented as a one-hot vector. As an illustration, the bond type of a single bond, a double bond or a triple bond can be represented as 100, 010 or 001 respectively.

[0037] The system generates alternative representations from the vertex input data, the pair input data, and optionally the triple input data by processing these data using one or more subnetworks (step 206). The system applies multiple functions to the vertex, pair, and triple input data to generate alternative representations. In all alternative representations, the pair data is unordered and the triple data is partially unordered. If X, Y, and Z are vertices, then the alternative pair representations of XY and YX must be the same and the alternative representations of the triples XYZ and ZYX must be the same. This will be referenced below. Figures 3 to 8 Several functions are further described and all maintain unordered pair data and partially unordered triple data.

[0038] The system processes the alternative representations using the combined layer to generate order-invariant features (step 208). Figure 1 As described, the combination layer can apply a set of trained parameters to the alternative representations to generate order-invariant features.

[0039] The system generates a classification of the input graph from the order-invariant features by processing the order-invariant features using one or more classification layers (step 210). Figure 1 As described, one or more classification layers may apply a set of trained parameters to the order-invariant features to generate classifications.

[0040] Figure 3 is a flow chart of an example method 300 for generating an alternative representation of vertex input data using a subnetwork. For convenience, the method 300 will be described with respect to a system (e.g., a system having one or more computing devices executing software to implement the method 300). Figure 1 The method 300 is described with reference to the system 100 ).

[0041] The system may perform the following operations 302 - 308 for each vertex of the system's input graph to generate an alternative representation of the vertex input data.

[0042] For each vertex, the system applies a first vertex function to the vertex input data of the vertex (step 302) to generate a first vertex output. The first vertex function can be a linear or nonlinear function. For example, the first vertex function can be an inverse tangent function. The first vertex function can also be Relu(WX+b), where X is the vertex input data as a vector of length n, W is a weight matrix of size mxn, b is a bias vector of length m, and Relu is an element-wise nonlinear function, such as a rectifier. The parameters of the first vertex function can be adjusted during training.

[0043] The system applies a second vertex function (304) to paired input data including each pair of vertices to generate a corresponding output for the pair of vertices. In particular, for a given vertex X, the system identifies each pair of vertices in the input graph including vertex X. For each pair, the system applies the second vertex function to the paired input data to generate an output for the pair. As an illustration, the second vertex function may perform a matrix multiplication between a particular parameter matrix and the paired input data including each pair of vertices. The second vertex function may also apply an element-wise nonlinear function to the output of the matrix multiplication. Parameters of the second vertex function (e.g., entries of the parameter matrix) may be adjusted during training of the system.

[0044] The system combines the outputs of each pair with a swap function to generate a second vertex output (step 306). A swap function is a function that returns the same output regardless of the order of its inputs. For example, the system can combine the outputs to generate the second vertex output by calculating the sum or average of the outputs.

[0045] The system generates an alternative representation of the vertex input data for the vertex from the first vertex output and the second vertex output, for example using a commutative or non-commutative function (step 308). For example, the system can calculate the sum of the first vertex output and the second vertex output or can multiply the first and second vertex outputs by weight parameters that can be learned during training.

[0046] Figure 4 is a flow chart of an example method for generating an alternative representation of paired input data using a subnetwork. For convenience, the method 400 will be described with respect to a system (e.g., a system having one or more computing devices executing software to implement the method 400). Figure 1 The method 400 is described with reference to the system 100 ).

[0047] The system may perform the following operations 402 - 406 for each pairing of vertices in the input graph to the system to generate an alternative representation of the paired input data.

[0048] For each pairing, the system applies a first pairing function to the pairing input data for that pair (step 402) to generate a first pairing output. The first pairing function can be a linear or nonlinear function. For example, the first pairing function can be an inverse tangent function. The first pairing function can also be Relu(WX+b), where X is the pairing input data as a vector of length n, W is a weight matrix of size mxn, b is a bias vector of length m, and Relu is an element-wise nonlinear function, such as a rectifier. The parameters of the first pairing function can be adjusted during training.

[0049] The system generates a second pairing output from the vertex input data of the vertices in the pair (step 404). For a given pair of vertices (X, Y), the system can identify the vertex input data of vertex X and the vertex input data of vertex Y. The identified vertex input data can be used as described below with reference to Figure 5 A second pairing output is generated as further described.

[0050] The system generates an alternative representation of the paired input data of the pair from the first paired output and the second paired output, for example using a commutative or non-commutative function (step 406). For example, the system may calculate the sum of the first paired output and the second paired output or may generate a weighted sum, where the weight is a parameter that can be adjusted during training. Optionally, the system also uses the third paired output to generate an alternative representation, which will be referred to below. Figure 6 This is described further.

[0051] Figure 5 is a flow chart of an example method for generating a second pairing output using a subnetwork. For convenience, the method 500 will be described with respect to a system (e.g., Figure 1 The method 500 is described with reference to the system 100 ).

[0052] For a particular pair of vertices, the system applies a non-commutative function to vertex input data for one of the vertices in the pair and, as a second input, vertex input data for the second vertex in the pair (step 502). For example, for a non-commutative function f, the system calculates f(X, Y), where X is the vertex input data for vertex X and Y is the vertex input for vertex Y. As a result, the non-commutative function generates a first non-commutative output. An example of a non-commutative function is y = Relu(W(X, Y) + b), where Relu is a non-linear function, W is a weight matrix, b is a bias vector, and (X, Y) is the concatenation of input vectors X and Y. Because y(a, b) is not equal to y(b, a), y is non-commutative.

[0053] The system then applies the same non-commutative function (step 504) to the vertex input data of the second vertex as the first input and the vertex input data of the first vertex as the second input to generate a second non-commutative function output. That is, in the above example, the system calculates f(Y, X).

[0054] The system applies a swap function to the first non-swappable output and the second non-swappable output (step 506) to generate Figure 4 The commutative function may be a summation function. The parameters of the non-commutative and commutative functions may be adjusted during training.

[0055] Figure 6 is a flow chart of an example method for generating a third pairing output using a subnetwork. For convenience, the method 600 will be described with respect to a system (e.g., a system having one or more computing devices executing software to implement the method 600) Figure 1 The method 600 is described with reference to the system 100 ).

[0056] For each pair of vertices, the system applies a second pairing function (step 602) to the triplet input data for each triplet that includes the pair of vertices as its endpoints to generate a corresponding output for the triplet of vertices. That is, given a pair of vertices (X, Y), the system can identify triplet input data for each triplet of vertices in the input graph with endpoints X and Y (e.g., (X, A, Y), (X, B, Y), (X, C, Y), etc.). The identified triplet input data can be used as input to the second pairing function. As an example, the second pairing function can perform a matrix multiplication between a specific parameter matrix and the triplet input data for each triplet that includes the pair of vertices. The parameters of the second pairing function can be adjusted during training of the system.

[0057] The system combines the corresponding output of each triple with the swap function to generate a third pairing output (step 604).For example, the system can combine the outputs by performing a sum or average of the outputs to generate the second vertex output.

[0058] The system uses commutative or non-commutative functions to Figure 4 The first pairing output, Figure 5 The second paired output and the third paired output generate an alternative representation of the paired input data for the pair of vertices (step 606). For example, the system can calculate the sum of the first vertex output and the second vertex output or can generate a weighted sum, where the weight is a parameter that can be adjusted during training.

[0059] Figure 7 is a flow chart of an example method for generating an alternative representation of triplet input data using a subnetwork. For convenience, the method 700 will be described with respect to a system (e.g., a system having one or more computing devices that execute software to implement the method 700). Figure 1The method 700 is described with reference to the system 100 ).

[0060] The system may perform the following operations 702 - 706 for each triple of vertices in the system's input graph to generate an alternative representation of the triplet input data.

[0061] For each triple, the system applies a first triple function to the triple input data of the triple (step 702) to generate a first triple output. The first triple function can be a linear function or a nonlinear function. For example, the first triple function can be an inverse tangent function. The first triple function can also be Relu(WX+b), where X is the paired input data as a vector of length n, W is a weight matrix of size mxn, b is a bias vector of length m, and Relu is an element-wise nonlinear function, such as a rectifier. Similar to the above reference Figure 3 The described first vertex function can adjust parameters of the first triplet function during training.

[0062] The system generates a second triplet output from the paired input data of the vertex pairs in the triplet (step 704). For a given triplet of vertices (X, Y, Z), the system can identify paired input data for vertex pairs (X, Y), (X, Z), and (Y, Z). The identified paired input data can be used to generate the second triplet output, which will be referred to below. Figure 8 Further comparison is described.

[0063] The system generates an alternative representation of the triplet of triplet input data from the first triplet output and the second triplet output, for example using a commutative or non-commutative function (step 706). For example, the system can calculate the sum of the first triplet output and the second triplet output or can generate a weighted sum, where the weight is a parameter that can be adjusted during training.

[0064] Figure 8 is a flow chart of an example method for generating a second triplet output using a subnetwork. For convenience, the method 800 will be described with respect to a system (e.g., Figure 1 The method 800 is described with reference to the system 100 ).

[0065] For a particular triplet of vertices, the system may identify a first pair of vertices comprising the first vertex and the second vertex in the triplet (step 802), identify a second pair of vertices comprising the first vertex and the third vertex in the triplet (step 804), and identify a third pair of vertices comprising the second vertex and the third vertex in the triplet (step 806).

[0066] The system applies a non-commutative function to the following three inputs (step 808): paired input data of a first pair of vertices as a first input; paired input data of a second pair of vertices as a second input; and paired input data of a third pair of vertices as a third input. For example, for a non-commutative function f and a triple XYZ, the system calculates f([X,Y],[X,Z],[Y,Z]), where [X,Y] is the paired input data of vertices X and Y, [Y,Z] is the paired input data of vertices Y and Z, and [X,Z] is the paired input data of vertices X and Z. As a result, the non-commutative function generates a first non-commutative output. An example of a non-commutative function is Relu(W(X,Y)+b), where Relu is a non-linear function, W is a weight matrix, b is a bias vector, and (X,Y) is the concatenation of input vectors X and Y.

[0067] The system then applies the same non-commutative function to the paired input data of the third pair of vertices as the first input, the paired input data of the second pair of vertices as the second input, and the paired input data of the first pair of vertices as the third input (step 810). That is, the system calculates f([Y, Z], [X, Z], [X, Y]). This causes the non-commutative function to generate a second non-commutative output.

[0068] The system applies a swap function to the first non-swappable output and the second non-swappable output (step 812) to generate Figure 7 The second triplet output of . For example, the commutative function can be a summation function. The parameters of the non-commutative and commutative functions can be adjusted during training.

[0069] Embodiments of the subject matter and functional operations described in this specification may be implemented using digital electronic circuits, tangibly implemented computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of a data processing device). Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, a computer storage medium is not a propagation signal.

[0070] The term "data processing apparatus" encompasses all types of equipment, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0071] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines or code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0072] As used in this specification, "engine" or "software engine" refers to an input / output system that provides a software implementation that is different from the input. An engine can be a functional coding block, such as a library, platform, software development kit ("SDK") or object. Each engine can be implemented on any suitable type of computing device including one or more processors and a computer-readable medium, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone or other fixed or portable device. Additionally, two or more engines can be implemented on the same computing device or on different computing devices.

[0073] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and devices can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0074] As an example, a processor suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or a central processing unit of any other type. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operationally coupled to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or transfer data to the mass storage devices, or both. However, a computer may not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.

[0075] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0076] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) monitor, an LCD (liquid crystal display) monitor, or an OLED display) for displaying information to the user and an input device, such as a keyboard, a mouse, or a presence-sensitive display or other surface, for providing input to the computer. Other kinds of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to a request received from the web browser.

[0077] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with implementations of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0078] A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0079] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and therefore even initially claimed, one or more features from a claimed combination may in some cases be deleted from that combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0080] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0081] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph represents an atomic element in the molecule and each edge represents a bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: a first sub-network configured to generate, from the vertex input data and the pairing input data, a first alternative representation of the vertex input data and a first alternative representation of the pairing input data, wherein the first sub-network is configured to, for each pairing of vertices in the input graph: applying a first pairing function to the pairing input data of the pairing to generate a first pairing output; Generating a second pairing output from the vertex input data of the vertices in the pairing, wherein generating the second pairing output comprises: applying a non-commutative function to vertex input data for vertices in the pair to generate a first non-commutative output, the vertex input data having (i) vertex data for a first vertex in the pair as a first input to the non-commutative function and (ii) vertex data for a second vertex in the pair as a second input to the non-commutative function, wherein one or more parameters of the non-commutative function have been adjusted during the training of the neural network; applying the non-commutative function to vertex input data for the vertices in the pair to generate a second non-commutative output, the vertex input data having (i) vertex data for the second vertex in the pair as a first input to the non-commutative function and (ii) vertex data for the first vertex in the pair as a second input to the non-commutative function; and applying a swap function to the first non-commutative output and the second non-commutative output to generate the second paired output; and generating the first alternative representation of the pairing input data for the pairing from at least the first pairing output and the second pairing output; and a combining layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first input alternative representation; and The order-invariant features are processed through one or more neural network layers to generate a neural network output representing the molecule represented by the graph data.

2. The method according to claim 1, wherein The neural network output is a classification representing the likelihood that the molecule represented by the graphical data will bind to a particular target molecule.

3. The method according to claim 1, wherein The properties of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of the ring of which this atomic element is a part.

4. The method according to claim 1, wherein The characteristics of each pair of vertices include at least one or more of the following: a type of bond between the atomic elements represented by the vertices or a length of a path between the atomic elements represented by the vertices.

5. The method according to claim 1, wherein The plurality of neural network layers further comprises: One or more other sub-networks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data and provide the second alternative representation as input to the combination layer.

6. The method according to claim 1, wherein The first sub-network is configured to, for each vertex in the input graph: applying a first vertex function to the vertex input data for the vertex to generate a first vertex output; applying a second vertex function to the paired input data for each pair including the vertex points to generate a corresponding output for each pair; combining the corresponding outputs of each pairing with a swap function to generate a second vertex output; as well as The first alternative representation of the vertex input data is generated from the first vertex output and the second vertex output.

7. The method according to claim 1, further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; wherein the first sub-network is configured to generate a first alternative representation of the triplet input data; as well as The first subnetwork is further configured to: for each pairing of vertices: applying a second pairing function to the triplet input data for each triplet of the triplet pairings being a first element and a third element of the triplet to generate a corresponding output for each triplet; combining the corresponding outputs of each triple with a swap function to generate a third pairing output; as well as Wherein generating the first alternative representation of the paired input data comprises generating the first alternative representation of the paired input data from the first paired output, the second paired output, and the third paired output.

8. The method according to claim 1, further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; The first sub-network is further configured to, for each triple of vertices in the input graph: applying a first triplet function to the triplet input data of the triplet to generate a first triplet output; generating a second triplet output from the paired input data of the pairs of vertices in the triplet; as well as A first alternative representation of the triplet input data is generated from the first triplet output and the second triplet output.

9. The method according to claim 8, wherein Generating the second triplet output includes: identifying a first pair of vertices comprising a first vertex and a second vertex in the triple; identifying a second pair of vertices comprising the first vertex and a third vertex in the triplet; identifying a third pair of vertices including the second vertex and the third vertex; applying a non-commutative function to the paired input data of the first pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the third pair of vertices as a third input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the paired input data of the third pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the first pair of vertices as a third input to the non-commutative function to generate a second non-commutative output; and A swap function is applied to the first non-commutative output and the second non-commutative output to generate the second triplet output.

10. A system for processing graphic data, comprising: one or more computers; as well as One or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph represents an atomic element in the molecule and each edge represents a bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: a first sub-network configured to generate, from the vertex input data and the pairing input data, a first alternative representation of the vertex input data and a first alternative representation of the pairing input data, wherein the first sub-network is configured to, for each pairing of vertices in the input graph: applying a first pairing function to the pairing input data of the pairing to generate a first pairing output; Generating a second pairing output from the vertex input data of the vertices in the pairing, wherein generating the second pairing output comprises: applying a non-commutative function to vertex input data for vertices in the pair to generate a first non-commutative output, the vertex input data having (i) vertex data for a first vertex in the pair as a first input to the non-commutative function and (ii) vertex data for a second vertex in the pair as a second input to the non-commutative function, wherein one or more parameters of the non-commutative function have been adjusted during the training of the neural network; applying the non-commutative function to vertex input data for the vertices in the pair to generate a second non-commutative output, the vertex input data having (i) vertex data for the second vertex in the pair as a first input to the non-commutative function and (ii) vertex data for the first vertex in the pair as a second input to the non-commutative function; and applying a swap function to the first non-commutative output and the second non-commutative output to generate the second paired output; and generating the first alternative representation of the pairing input data of the pairing from at least the first pairing output and the second pairing output; and a combining layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first input alternative representation; and The order-invariant features are processed through one or more neural network layers to generate a neural network output representing the molecule represented by the graph data.

11. The system according to claim 10, wherein: The neural network output is a classification representing the likelihood that the molecule represented by the graphical data will bind to a particular target molecule.

12. The system according to claim 10, wherein: The properties of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of the ring of which this atomic element is a part.

13. The system according to claim 10, wherein: The characteristics of each pair of vertices include at least one or more of the following: a type of bond between the atomic elements represented by the vertices or a length of a path between the atomic elements represented by the vertices.

14. The system according to claim 10, wherein: The plurality of neural network layers further comprises: One or more other sub-networks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data and provide the second alternative representation as input to the combination layer.

15. The system according to claim 10, wherein: The first sub-network is configured to, for each vertex in the input graph: applying a first vertex function to the vertex input data for the vertex to generate a first vertex output; applying a second vertex function to the paired input data for each pair including the vertex points to generate a corresponding output for each pair; combining the corresponding outputs of each pairing with a swap function to generate a second vertex output; as well as The first alternative representation of the vertex input data is generated from the first vertex output and the second vertex output.

16. The system of claim 10, the operations further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; wherein the first sub-network is configured to generate a first alternative representation of the triplet input data; as well as The first subnetwork is further configured to: for each pairing of vertices: applying a second pairing function to the triplet input data for each triplet of the triplet pairings being a first element and a third element of the triplet to generate a corresponding output for each triplet; combining the corresponding outputs of each triple with a swap function to generate a third pairing output; as well as Wherein generating the first alternative representation of the paired input data comprises generating the first alternative representation of the paired input data from the first paired output, the second paired output, and the third paired output.

17. The system of claim 10, the operations further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; The first sub-network is further configured to, for each triple of vertices in the input graph: applying a first triplet function to the triplet input data of the triplet to generate a first triplet output; generating a second triplet output from the paired input data of the pairs of vertices in the triplet; as well as A first alternative representation of the triplet input data is generated from the first triplet output and the second triplet output.

18. The system according to claim 17, wherein: Generating the second triplet output includes: identifying a first pair of vertices comprising a first vertex and a second vertex in the triple; identifying a second pair of vertices comprising the first vertex and a third vertex in the triplet; identifying a third pair of vertices including the second vertex and the third vertex; applying a non-commutative function to the paired input data of the first pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the third pair of vertices as a third input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the paired input data of the third pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the first pair of vertices as a third input to the non-commutative function to generate a second non-commutative output; and A swap function is applied to the first non-commutative output and the second non-commutative output to generate the second triplet output.

19. One or more non-transitory computer-readable media having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph represents an atomic element in the molecule and each edge represents a bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: a first sub-network configured to generate, from the vertex input data and the pairing input data, a first alternative representation of the vertex input data and a first alternative representation of the pairing input data, wherein the first sub-network is configured to, for each pairing of vertices in the input graph: applying a first pairing function to the pairing input data of the pairing to generate a first pairing output; Generating a second pairing output from the vertex input data of the vertices in the pairing, wherein generating the second pairing output comprises: applying a non-commutative function to vertex input data for vertices in the pair to generate a first non-commutative output, the vertex input data having (i) vertex data for a first vertex in the pair as a first input to the non-commutative function and (ii) vertex data for a second vertex in the pair as a second input to the non-commutative function, wherein one or more parameters of the non-commutative function have been adjusted during the training of the neural network; applying the non-commutative function to vertex input data for the vertices in the pair to generate a second non-commutative output, the vertex input data having (i) vertex data for the second vertex in the pair as a first input to the non-commutative function and (ii) vertex data for the first vertex in the pair as a second input to the non-commutative function; and applying a swap function to the first non-commutative output and the second non-commutative output to generate the second paired output; and generating the first alternative representation of the pairing input data for the pairing from at least the first pairing output and the second pairing output; and a combining layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first input alternative representation; and The order-invariant features are processed through one or more neural network layers to generate a neural network output representing the molecule represented by the graph data.

20. The medium according to claim 19, wherein The neural network output is a classification representing the likelihood that the molecule represented by the graphical data will bind to a particular target molecule.

21. A method for processing graphic data, comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph is an atomic element in the molecule and each edge is a type of bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: A first sub-network, wherein the first sub-network is configured as: For each vertex in the input graph: applying a first vertex function to the vertex input data for the vertex to generate a first vertex output; applying a second vertex function to the paired input data for each pair including the vertex points to generate a corresponding output for each pair; combining the corresponding output of each pairing with a swap function to generate a second vertex output; and generating a first alternative representation of the vertex input data from the first vertex output and the second vertex output; and generating a first alternative representation of the paired input data from the vertex input data and the paired input data; and a combination layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first alternative representation; and process the order-invariant features through one or more neural network layers to generate a predicted output of the graph data characterizing the molecule represented by the graph data.

22. The method according to claim 21, wherein The prediction output includes a score representing the likelihood that the molecule will bind to a particular target molecule.

23. The method according to claim 21, wherein The properties of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of the ring of which this atomic element is a part.

24. The method according to claim 21, wherein The characteristics of each pair of vertices include at least one or more of the following: a type of bond between the atomic elements represented by the vertices or a length of a path between the atomic elements represented by the vertices.

25. The method according to claim 21, wherein The plurality of layers further comprises: One or more other sub-networks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data and provide the second alternative representation as input to the combination layer.

26. The method according to claim 21, wherein The first subnetwork is configured to, for each pairing of vertices in the input graph: applying a first pairing function to the pairing input data of the pairing to generate a first pairing output; generating a second pairing output from the vertex input data for vertices in the pairing; as well as The first alternative representation of the pairing input data of the pairing is generated from the first pairing output and the second pairing output.

27. The method according to claim 26, wherein Generating the second pairing output includes: applying a non-commutative function to vertex input data for a first vertex in the pair as a first input to the non-commutative function and vertex input data for a second vertex in the pair as a second input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the vertex input data for the second vertex as a first input to the non-commutative function and to the vertex input data for the first vertex as a second input to the non-commutative function to generate a second non-commutative output; and A swap function is applied to the first non-commutative output and the second non-commutative output to generate the second paired output.

28. The method of claim 26, further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; wherein the first sub-network is configured to generate a first alternative representation of the triplet input data; as well as The first subnetwork is further configured to: for each pairing of vertices: applying a second pairing function to the triplet input data for each triplet of the triplet pairings being a first element and a third element of the triplet to generate a corresponding output for each triplet; combining the corresponding outputs of each triple with a swap function to generate a third pairing output; as well as Wherein generating the first alternative representation of the paired input data comprises generating the first alternative representation of the paired input data from the first paired output, the second paired output, and the third paired output.

29. The method of claim 21, further comprising: generating triplet input data from the graph data, the triplet input data representing properties of triplets of vertices in the graph; The first sub-network is further configured to, for each triple of vertices in the input graph: applying a first triplet function to the triplet input data of the triplet to generate a first triplet output; generating a second triplet output from the paired input data of the pairs of vertices in the triplet; as well as A first alternative representation of the triplet input data is generated from the first triplet output and the second triplet output.

30. The method according to claim 29, wherein Generating the second triplet output includes: identifying a first pair of vertices comprising a first vertex and a second vertex in the triple; identifying a second pair of vertices comprising the first vertex and a third vertex in the triplet; identifying a third pair of vertices including the second vertex and the third vertex; applying a non-commutative function to the paired input data of the first pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the third pair of vertices as a third input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the paired input data of the third pair of vertices as a first input to the non-commutative function, the paired input data of the second pair of vertices as a second input to the non-commutative function, and the paired input data of the first pair of vertices as a third input to the non-commutative function to generate a second non-commutative output; and A swap function is applied to the first non-commutative output and the second non-commutative output to generate the second triplet output.

31. A system for processing graphic data, comprising: one or more computers; as well as a computer-readable medium coupled to the one or more computers and having instructions stored thereon that, when executed by the one or more computers, cause the one or more computers to perform operations for each of the neural network layers, the operations comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph is an atomic element in the molecule and each edge is a type of bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: A first sub-network, wherein the first sub-network is configured as: For each vertex in the input graph: applying a first vertex function to the vertex input data for the vertex to generate a first vertex output; applying a second vertex function to the paired input data for each pair including the vertex points to generate a corresponding output for each pair; combining the corresponding output of each pairing with a swap function to generate a second vertex output; and generating a first alternative representation of the vertex input data from the first vertex output and the second vertex output; and generating a first alternative representation of the paired input data from the vertex input data and the paired input data; and a combination layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first alternative representation; and process the order-invariant features through one or more neural network layers to generate a predicted output of the graph data characterizing the molecule represented by the graph data.

32. The system of claim 31, wherein: The properties of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of the ring of which this atomic element is a part.

33. The system of claim 31, wherein: The characteristics of each pair of vertices include at least one or more of the following: a type of bond between the atomic elements represented by the vertices or a length of a path between the atomic elements represented by the vertices.

34. The system of claim 31 , wherein: The plurality of layers further comprises: One or more other sub-networks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data and provide the second alternative representation as input to the combination layer.

35. The system of claim 31 , wherein: The first subnetwork is configured to, for each pairing of vertices in the input graph: applying a first pairing function to the pairing input data of the pairing to generate a first pairing output; generating a second pairing output from the vertex input data for vertices in the pairing; as well as The first alternative representation of the pairing input data of the pairing is generated from the first pairing output and the second pairing output.

36. The system of claim 35, wherein: Generating the second pairing output includes: applying a non-commutative function to vertex input data for a first vertex in the pair as a first input to the non-commutative function and vertex input data for a second vertex in the pair as a second input to the non-commutative function to generate a first non-commutative output; applying the non-commutative function to the vertex input data for the second vertex as a first input to the non-commutative function and to the vertex input data for the first vertex as a second input to the non-commutative function to generate a second non-commutative output; and A swap function is applied to the first non-commutative output and the second non-commutative output to generate the second paired output.

37. A computer-readable medium having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving graph data representing an input graph, the input graph comprising a plurality of vertices connected by edges, wherein the graph data represents a molecule, and wherein each vertex in the graph is an atomic element in the molecule and each edge is a type of bond between two atomic elements in the molecule; generating, from the graph data, vertex input data representing characteristics of each vertex in the input graph and pair input data representing characteristics of pairs of vertices in the input graph; Generating order-invariant features of the input graph using a trained neural network, wherein the trained neural network comprises: A first sub-network, wherein the first sub-network is configured as: For each vertex in the input graph: applying a first vertex function to the vertex input data for the vertex to generate a first vertex output; applying a second vertex function to the paired input data for each pair including the vertex points to generate a corresponding output for each pair; combining the corresponding output of each pairing with a swap function to generate a second vertex output; and generating a first alternative representation of the vertex input data from the first vertex output and the second vertex output; and generating a first alternative representation of the paired input data from the vertex input data and the paired input data; and a combination layer configured to receive at least one input alternative representation and process the at least one input alternative representation to generate the order-invariant features, wherein the at least one input alternative representation includes the first alternative representation; and process the order-invariant features through one or more neural network layers to generate a predicted output of the graph data characterizing the molecule represented by the graph data.

38. The computer-readable medium of claim 37, wherein: The properties of each vertex include at least one or more of the following: the atomic element represented by the vertex, whether the atomic element represented by the vertex is a hydrogen bond donor or acceptor, the chirality of the atomic element, the formal charge of the atomic element, the hybridization state of the atomic element, or the size of the ring of which this atomic element is a part.

39. The computer-readable medium of claim 37, wherein: The characteristics of each pair of vertices include at least one or more of the following: a type of bond between the atomic elements represented by the vertices or a length of a path between the atomic elements represented by the vertices.

40. The computer-readable medium of claim 37, wherein: The plurality of layers further comprises: One or more other sub-networks that collectively process the first alternative representation to generate a second alternative representation of the paired input data and a second alternative representation of the vertex data and provide the second alternative representation as input to the combination layer.

Images