Modelling multi-scale interactions in a geometrical algebraic equivariant graph neural network

Abstract

A computer-implemented machine learning method for modelling multi-scale interactions includes determining a geometric vectorization based on a mapping of input data associated with a multi-scale system. Using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system are generated. The neural network includes pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms. The one or more predicted vectors are mapped to the multi-scale interactions of the multi-scale system. The method has applications including, but not limited to, use cases in computational biology, medical AI and healthcare, and orbiting body management for optimizing machine learning processes or supporting decision making.

Description

Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) MODELLING MULTI-SCALE INTERACTIONS IN A GEOMETRICAL ALGEBRAIC EQUIVARIANT GRAPH NEURAL NETWORK CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit to European Patent Application No. EP 24156907.8, filed on February 9, 2024, which is hereby incorporated by reference herein. FIELD

[0002] The present disclosure relates to Artificial Intelligence (AI) and machine learning (ML), and in particular to a method, system, data structure, computer program product and computer-readable medium for modelling multi-scale interactions in geometrical algebraic equivariant graph neural network and its application to atomistic modelling of biological systems and drug design. BACKGROUND

[0003] In medical applications, the biological system at various levels can be modelled. When modelling large biological system as deoxyribonucleic acid (DNA), ribonucleic acid (RNA), messenger RNA (mRNA), Proteins, T-Cell Receptors, B-Cell receptors, the interaction of atoms that are not local might need to be considered. Traditional graph neural networks (GNNs) can propagate the information depending on the connectivity of the graph and on the number of layers in distance that is linear in the number of layers. For these large systems, the number of parameters of the network can easily exceed the memory of the graphics processing unit (GPU) or alternative computational device, and can require large computing times and resultant increased costs to the computational infrastructure. Moreover, architectures based on transformers (e.g., as used in large language models (LLM)) can require further increased cost demands. Additionally, since multiple passes are often performed on the network for both training and inference, current architectures can be limited in their potential to describe long range interactions or expensive in term of memory and computational resources.

[0004] In addition to modelling long-range interactions for biological (and in general also for organic and inorganic materials such as polymers), neural network (NN) architectures can be designed (e.g. training in modern systems using automatic differentiation) to include the symmetries (e.g., most critical symmetry) of a given molecular system. These symmetries can be modelled as group actions (e.g. rotations and translations). When predicting a vectorial quantity, as for example the force acting on an atom, the NN architecture can be equivariant, while when predicting scalar quantities such as mass or energy of molecules, the architecture can be invariant to these group actions. SUMMARYAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0005] In an embodiment, the present disclosure provides a computer-implemented machine learning method for modelling multi-scale interactions. A geometric vectorization is determined based on a mapping of input data associated with a multi-scale system. Using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system are generated. The neural network includes pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms. The one or more predicted vectors are mapped to the multi-scale interactions of the multi-scale system. The method has applications including, but not limited to, use cases in computational biology, medical AI and healthcare, and orbiting body management for optimizing machine learning processes or supporting decision making. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the present disclosure will be described in even greater detail below based on the exemplary figures. The present disclosure is not limited to the exemplary embodiments. All features described and / or illustrated herein can be used alone or combined in different combinations in embodiments of the present disclosure. The features and advantages of various embodiments of the present disclosure will become apparent by reading the following detailed description with reference to the attached drawings which illustrate the following:

[0007] FIG.1 illustrates a schematic representation of the architecture and training of an equivariant neural network (NN) according to one or more embodiments of the present disclosure;

[0008] FIG.2 illustrates representations of multiple types of unitary vectors according to one or more embodiments of the present disclosure;

[0009] FIG.3 illustrates representations of example Clifford group equivariant neural network transformations according to one or more embodiments of the present disclosure;

[0010] FIG.4 illustrates exemplary stages of a pooling process according to one or more embodiments of the present disclosure;

[0011] FIG.5 illustrates a block-diagram of an example operational flow of a multi-bladed projection according to one or more embodiments of the present disclosure;

[0012] FIG.6 illustrates an example unpooling process according to one or more embodiments of the present disclosure;

[0013] FIG.7 illustrates a table of example error rates for a protein simulation performed according to one or more embodiments of the present disclosure;

[0014] FIG.8 illustrates a table of example prediction error rates on various types of simulated datasets according to one or more embodiments of the present disclosure;Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0015] FIG.9 is a schematic diagram of an encoder and decoder with corresponding pooling and unpooling layers according to one or more embodiments of the present disclosure;

[0016] FIG.10 illustrates a visualization of the output of a hierarchy-based Clifford group equivariant message passing neural network (HCGE-MPNN) and ground-truth for multi N-body dataset of one or embodiments of the present disclosure;

[0017] FIG.11 illustrates a visualization of the output of a HCGE-MPNN and ground-truth for motion capture dataset according to one or more embodiments of the present disclosure; and

[0018] FIG.12 is a block diagram of an exemplary processing system, which can be configured to perform any and all operations disclosed herein. DETAILED DESCRIPTION

[0019] In medical applications, modelling the biological system at various levels such as DNA, RNA, mRNA, Proteins, T-Cell Receptors, B-Cell receptors can require considering the interaction of atoms that are not local. This can be done by propagating the information based on the connectivity of the graph and on the number of layers in distance that is linear in the number of layers. However, given the number of passes that can be required for both training and interference, the parameters of the network can easily exceed the memory of the GPU or alternative computational device, requiring large time and bringing to high cost on the computational infrastructure. Additionally, and / or alternatively, the NN architectures neural (e.g., for training in modern systems using automatic differentiation), can require the inclusion of the symmetry of the molecular system. These symmetries can be modelled as group actions (e.g., rotations and translations). Predict a vectorial quantity, as for example the force acting on an atom, can require the NN architecture to be equivariant, while predicting scalar quantities such as mass or energy of molecules can require the architecture to be invariant to these group actions.

[0020] As such, embodiments of the present disclosure can enable the modelling of long- range / multiscale interactions that can be (and historically have been) too computationally complex and computationally intensive to be performed in the human mind while also including physical symmetries such as rotation and translation in three dimensions. Additionally, and / or alternatively, embodiments of the present disclosure can be used for property prediction, used as surrogate model for forces and energy (e.g. for active learning framework), and / or used as a generative model for structure prediction of large molecular systems. Moreover, embodiments of the present disclosure can improve the performance of models between 4-66% in standard benchmark datasets (e.g., n-body dataset, motion capture and others) compared to conventional methods and while also reducing the demand on computational resources.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0021] In a first aspect, the present disclosure provides a computer-implemented method for modelling multi-scale interactions. A geometric vectorization is determined based on a mapping of input data associated with a multi-scale system. Using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system are generated. The neural network includes pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms. The one or more predicted vectors are mapped to the multi-scale interactions of the multi-scale system.

[0022] In a second aspect, the present disclosure provides the method according to the first aspect, wherein the one or more predicted vectors includes a multi-vector, and wherein mapping the one or more predicted vectors to the multi-scale interactions comprises mapping the multi- vector to a scalar component that is invariant to rotation and translation, a vector component, and a bivector component of the molecular system.

[0023] In a third aspect, the present disclosure provides the method according to the first aspect or the second aspect, wherein the multi-scale system represents a molecular system, wherein the scalar component represents an atomic mass of an atom, wherein the vector component represents information in different channels of a position or a momentum of the atom, and wherein the bivector component represents an angle between the atom and another atom or area.

[0024] In a fourth aspect, the present disclosure provides the method according to any of the first to third aspects, wherein the multi-scale system represents a messenger ribonucleic acid (mRNA) system of atoms, wherein the atoms of the mRNA system of atoms is represented in a configuration as components of the multi-vector, wherein the scalar component represents an atomic energy of an atom of the mRNA system of atoms, and wherein the vector component represents a force or an acceleration associated with the atom.

[0025] In a fifth aspect, the present disclosure provides the method according to any of the first to fourth aspects, wherein the one or more predicted vectors comprise a representation of the atom, and wherein mapping the one or more predicted vectors to the multi-scale interactions of the multi-scale system further comprises integrating the representation of the atom into a plurality of atoms of the system of atoms.

[0026] In a sixth aspect, the present disclosure provides the method according to any of the first to fifth aspects, wherein the multi-scale system represents a low earth orbit satellite system, wherein the scalar component represents that mass of a satellite of the satellite system, and wherein the vector component represents the coordinate of the satellite.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0027] In a seventh aspect, the present disclosure provides the method according to any of the first to sixth aspects, further including: prior to generating the one or more predicted vectors, creating the neural network to describe the molecular system, wherein a use of the multi-vector indicates flexibility to map various information either in a specific blade of the multi-blade projection or in an additional channel of the vector component.

[0028] In an eighth aspect, the present disclosure provides the method according to any of the first to seventh aspects, wherein the mapping of the input data is based on Clifford algebra. Generating the one or more predicted vectors is based on the mapping of the input data.

[0029] In a ninth aspect, the present disclosure provides the method according to any of the first to eighth aspects, further including: collecting a training dataset comprising training input data and training output data; and training the neural network using the training dataset to generate one or more training predicted vectors. The one or more training predicted vectors indicate a final state of a simulation or a step in a multi-step simulation.

[0030] In a tenth aspect, the present disclosure provides the method according to any of the first to ninth aspects, wherein training the neural network is based on using a neural network neural potential or a generative process of a diffusion model.

[0031] In an eleventh aspect, the present disclosure provides the method according to any of the first to tenth aspects, wherein the neural network further comprises a geometric algebra or Clifford based neural network, and wherein the geometric algebra or Clifford based neural network is extended with dense or sparse pooling of the pooling and unpooling mechanisms.

[0032] In a twelfth aspect, the present disclosure provides the method according to any of the first to eleventh aspects, wherein the neural network is a message passing neural network.

[0033] In a thirteenth aspect, the present disclosure provides the method according to any of the first to twelfth aspects, wherein the equivariance of the geometric primitives comprises translation and rotation of the geometric primitives.

[0034] In a fourteenth aspect, the present disclosure provides a computer system for modelling multi-scale interactions. The system includes one or more hardware processors, which, alone or in combination, are configured to provide for execution of the following steps: a geometric vectorization is determined based on a mapping of input data associated with a multi- scale system; using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system are generated, wherein the neural network includes pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms; and the one or more predicted vectors are mapped to the multi-scale interactions of the multi-scale system.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0035] In a fifteenth aspect, the present disclosure provides a tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, alone or in combination, provide for execution of a method for modelling multi-scale interactions comprising the following steps: a geometric vectorization is determined based on a mapping of input data associated with a multi-scale system; using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system are generated, wherein the neural network includes pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi- blade projection that defines pooling or clustering of the pooling and unpooling mechanisms; and the one or more predicted vectors are mapped to the multi-scale interactions of the multi- scale system.

[0036] FIG.1 provides an example of the training process for an architecture 100. For instance, the architecture 100 includes a model 130 (e.g., an NN, a HCGE-MPNN) that includes two components: an equivariant NN 132 based on geometric algebra and an equivariant pooling and unpooling model 134 (e.g., an encoder and / or decoder). This model 130 can collect (e.g., obtain, receive) a dataset 101 and undergo a training phase 102 which, once trained, provides a molecular property predictor or molecular generative model 104 for an inference phase.

[0037] A graph of dataset 101 is composed of nodes (for example atoms) and edges (for example bonds or atom connected within a cut-off radius). Additionally, and / or alternatively, for a loose type of graph, the dataset 101 can include a point cloud where the nodes have coordinates in three dimensions but they are not explicitly connected by edges. In other words, the dataset 101 can include nodes and edges for input data, and the input data can be associated with a multi-scale system (e.g., a system with certain properties on one level based on the information from different levels, with each level addressing a separate phenomenon) such as a chain of atoms and the bonds between those atoms, and the input data can be mapped to a geometric vectorization of the multi-scale system.

[0038] During the training phase 102, at each layer of the pooling and unpooling model 130, the nodes of the dataset 101 are pooled when going “up” in level and unpooled when going “down” in levels. For example, at a first level of geometric message passing 108, message passing is performed by the equivariant NN 132 and then geometric pooling 110 (further discussed with respect to FIGS.4, 5, 6, and 9) is performed by the model 134 (e.g., based on the geometric message passing 108). At the next level, geometric massage passing 112 is performed by the equivariant NN 132 (e.g., based on the pooled nodes and edges resulting from the geometric pooling 110) and geometric pooling occurs again up to an Hth time (e.g. geometric pooling 114). Geometric message passing 116 again occurs at the next level.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0039] After the final instance of geometric message passing, which occurs after H number of times (e.g., geometric message passing 116), the resultant geometric message passing 118 is unpooled by model 134 by performing geometric unpooling 120. At the next level (e.g., the Hth level), geometric message passing 122 occurs based on the unpooled nodes and edges of geometric unpooling 120. Based on the result of the geometric message passing 122, the model 134 performs geometric unpooling 124 again, and the model 132 again performs geometric message passing 126 based on the nodes and edges unpooled from geometric unpooling 124. For example, the output of the (H-k+1)-th geometric unpooling (1<= k <= H) can have the topology isomorphic to that of the output of the k-th geometric pooling layer. As indicated by the horizontal lines between geometric message passings of pooling and unpooling steps (e.g., geometric message passing 112 and geometric message passing 122), the training phase 102 can use numerical features defined over nodes and edges of the isomorphic topologies and the instance of the arithmetic for those features can be the concatenation and summation.

[0040] In other words, the general framework of architecture 100 includes two components in model 130: a Clifford group equivariant NN 132 that propagates low and high-order information, and another is a Clifford equivariant pooling / unpooling network 134 that pools / unpools nodes in a graph (e.g., a graph based on dataset 101). An example process can be as follows: once data is given from dataset 101, the data is first processed by the equivariant NN model 132. The output of the network (e.g., NN model 132) is then processed by the pooling model 134 and the model outputs data with a smaller number of nodes. These two processes are repeated up to predefined times (hereinafter denoted the number by H). Then, the same process is performed H number of times, but with unpooling models of model 134 instead. Each of these geometric message passing and geometric pooling pairs 128A and 128B (and therefore the geometric unpooling and geometric message passing pairs 128C and 128D) is referred to as an H block (e.g., H block 128). The loss 136 is computed according to the output and backpropagation is performed using gradient 138 to update the model parameters.

[0041] The first component (e.g., equivariant NN 132) can be a Clifford / geometric group equivariant neural network, which can be embodied by trainable machine-learning models equivariant to Clifford group action. Clifford group equivariant message passing neural networks and geometric algebra transformers are examples of applicable networks, and are described, for example, in Ruhe, David, et al. “Clifford Group Equivariant Neural Networks” arXiv preprint: arXiv:2305.11141 and Brehmer, Johann, et al. “Geometric Algebra Transformer”, arXiv preprint: arXiv:2305.18415, which are incorporated by reference herein.

[0042] FIG.2 provides an example description of multivectors in R3(e.g., used by the equivariant NN 132) where the base is given by the vectors {e_1,e_2,e_3} as the set of unitaryAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) vectors. For example, a general framework can be based on geometric / Clifford algebra (e.g., geometric vectorization) where generalized vectors, called multivectors, model not only low- order information such as magnitude of vector and distance between two vectors, but also high- order information like dihedral angle between multiple objects and volume of area spanned by multiple vectors. For instance, the multivector 202 can provide vectors including a scalar 204, a vector 206, bivector 208, trivector 210. As shown in FIG.3, the equivariant NN 132 (e.g., a Clifford group equivariant NN) can allow for rotation and transformation to be applied to the multivectors 302 before or after the geometric message passing performed by the equivariant NN 132 and yield the same result 304.

[0043] The second component (e.g., pooling and unpooling model 134) can in turn include a pooling model that can perform different types of pooling (e.g., a dropping nodes method and a clustering nodes method) and different types of unpooling (e.g., unpooling the dropped nodes method and unpooling the clustered nodes method). As shown in FIG.4, the pooling model (e.g., pooling and unpooling model 134 of FIG.1) is responsible for making a decision on which nodes in a graph are pooled and reconstructing a graph according to the pooled nodes (e.g., by performing a computation of scores 408). The pooling process 400 can be divided by three processes: a scoring stage 402, a node creation stage 404, and a topology reconstruction stage 406.

[0044] In the scoring stage 402, the model (e.g., model 134 of FIG.1) first takes a graph as an input and computes a confident score over all the nodes of the graph. In the node creation stage 404, new nodes of a pooled graph are generated based on the scores computed in the scoring stage 402. The model can take two ways to perform node creation, including a node dropping method and a node clustering method. After the new nodes are generated, edges are reconstructed in the topology reconstruction stage 406.

[0045] For example, FIG.4 provides an overview of the pooling process 400 performed by a pooling model (e.g., an encoder, the pooling and unpooling model 134 of FIG.1) at each instance of geometric pooling (e.g., geometric pooling 114, 116 and geometric pooling 120, 124 of FIG.1). At the first stage, the scoring stage 402, the model 134 can compute a score at step 408 for the input multivectors (e.g., a multiblade projection that generates the same scores no matter how the Clifford group acts on the multivectors). From there, the model 134 can perform a node clustering method and / or a node dropping method to the multivectors while storing the edges of the clustered and dropped nodes. In the second stage, the node creation stage 404, the model 134 can generate new nodes for the output of the node clustering method and / or the node dropping method by performing a cluster assignment at step 410 to the output of the node clustering method, and perform node dropping at step 412 to the output of the node droppingAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) method. The cluster assignment at step 410 and the node dropping at step 412 can be performed independently, and can perform the two types of the pooling and unpooling methods sequentially. In the third stage, the topology reconstruction stage 406, the model 134 can rewire the edges at step 414 stored during the node creation stage 404 to the new nodes of the node creation stage 404. Accordingly, the pooling and unpooling model can output a graph for the next step of geometric message passing.

[0046] Determining geometric vectorization (e.g., as shown in FIGS.2 and 3), generating the predicted vectors (e.g., resulting from the process as shown in FIG.4), and mapping of the predicted vectors to the multi-scale interactions of the multi-scale system according to embodiments of the present disclosure are computationally complex tasks not capable of being performed in the human mind. For example, determining a geometric vectorization is a complex computational task, especially being based on a mapping of input data associated with a multi- scale system, which further increases its complexity. Also, as the multi-scale system includes multi-scale interactions, such as complex interactions among molecules, generating predicted vectors of such a system is also a computationally complex task utilizing complex machine learning techniques and models, such as a multi-blade projection and pooling and unpooling layers or mechanisms. Further, as the predicted vectors and the multi-scale system introduce such computational complexities, mapping of such vectors to such a system is also a computationally complex task not capable of being performed in the human mind.

[0047] In the dropping nodes method, the model samples a subset of nodes as the nodes for the pooled graph, based on the scores computed in scoring stage 402. Creating the framework Clifford group equivariant can use an invariance of score function to group action. For instance, the confidence score can be the same no matter what action is applied to the input. For this purpose, a multi-bladed projection can be used, with an inner product generalized for multivectors, to compute confident scores: ^⋅^^^^^బ^ ^^ ൫ ^ ^ ൯||

[0048] Eq.1a trainable Clifford group equivariant model and the bullet in the numerator denotes a Clifford product, which is a natural generalization of the inner product. After computing the Clifford product of the two multivectors, the model projects a scalar part (0-th element of the multi-vector), which enables the multi-bladed projection to be invariant to group action, since the scalar part of the Clifford product is proven to be invariant to Clifford action. After the model chooses the nodes to beAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) pooled according to the score y, edges connecting to unchosen nodes are deleted from the graph (e.g., stored).

[0049] For example, FIG.5 provides a block-diagram of the operational flow of the multibladed projection of Eq.1. Once the input 502 is given, the trainable Clifford group equivariant model is computed for each of inputs 504. After that, the invariant part of the inner product is computed for every pair of one of the inputs and its output of the Clifford group equivariant model 506, while the summation of the norm (for example l_2 norm) of 504 is also computed. Finally all the 508 are summed up at block 512 and divided at block 514 by the result of 510.

[0050] The node clustering method can be an extension of Eq.1, but all nodes are explicitly assigned to a pooled node. To achieve this, the model (e.g., model 134) first pre-defines the number of clusters (denoted as K) as the number of nodes for a pooled graph. The model then computes confidence scores, but can have the same number of multibladed projection as the clusters: ൫^^ ⋅ ^^ ^^^ ^^^^ ^^^ ൌ ^ ^ ^ ^ ൯^, ^^ ൌ 1,2, … , ^^,and have a number^ ^బ^ ^ ⋅^^(Eq.2) ^^ ൌ ൭^^^^ ൌ ∑^^ ೖ൫^^൯^ ^:^^^^^^^ ^ , 1 ^ ^^ ^ ^^, 1 ^ ^^ ^ ^^the node feature and edges (adjacent matrix) for a pooled graph are computed as follows: ^^^^^^ ൌ ^^^^^, ^^^^^^ ൌ ^^^^^ ^^

[0052] The overall process can be equivariant to Clifford group action, since Eq.2 is again invariant to Clifford group action.

[0053] The model (e.g., model 134) can perform unpooling for the node dropping method of pooling. When restoring nodes from pooled nodes, the model complements node features for unpooled nodes while keeping the node features of the pooled nodes unchanged. The complemented features can be zero vectors or mean of all the input vectors, to ensure the equivariance of the unpooling operation. Edges are reconstructed from edge information stored when performing pooling operation.

[0054] For example, FIG.6 provides a visualization of an embodiment of the unpooling of the resultant graph 602 of the node dropping method. The resultant graph 602 includes nodes ^^^connected by an edge to ^^ଷ, nodes ^^ଶconnected by edges to nodes ^^ଷand ^^ସ, node ^^ଷconnected by edges to nodes ^^ଶand ^^ସ, and node ^^ସconnected by edges to nodes ^^ଶand ^^ଷ. When theAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) model performs unpooling 604, the output graph 606 includes the nodes and edges of the resultant graph, including new nodes ^^ and ^^, where node ^^ is connected by edges to node ^^ଷand new node ^^, and new node ^^ is connected by edges to node ^^ସand new node ^^.

[0055] The model (e.g., model 134), can perform an extension of unpooling for the output of the node clustering method (e.g., step 410 of FIG.4) that respects the explicit assignments of the nodes to the pooled nodes as follows: ^^௨^^^^^ ൌ ^^ ^^^^^^

[0056] Edges can be stored when performing poolingoperation.

[0057] Embodiments of the present disclosure can allow for the extension of geometric algebra equivariant NN architectures with a pooling and unpooling mechanism that preserves equivariance to a group action to model multi-scale interactions.

[0058] Embodiments of the present disclosure can provide a direct prediction. For example, one or more embodiments can be used for a direct prediction of the final property (e.g., solvability of a molecule or elasticity of a polymer).

[0059] Embodiments of the present disclosure can provide a generative model. For example, one or more embodiments can be used as a component in a denoise auto-encoder of a diffusion model and used to generate molecule’s structures.

[0060] Embodiments of the present disclosure can provide a molecular dynamic. For example, one or more embodiments can be used to encode the force prediction for a molecular dynamic model (e.g., where the force prediction model is trained separately).

[0061] Embodiments of the present disclosure can provide for active learning. For example, the training of the force model (e.g., neural potential) can be implemented in an active learning loop, where new configurations are selected to be computed in the quantum mechanical solver and used as a new training point for the neural potential.

[0062] Embodiments of the present disclosure thus provide for improvements to computers in machine learning systems, such as improvements to molecular modelling for structural dynamics property prediction of mRNA molecules for vaccine design and delivery.

[0063] For instance, RNA can be converted to mRNA, and mRNA can then be translated in proteins. This process can used for drug delivery. An aspect of this process is that mRNA can be designed to improve stability and efficiency, because embodiments of the present disclosure can provide models that describe long range interaction among atoms (e.g., a potentially important aspect of mRNA design). Provided enough data, embodiments of the present disclosure can predict the output property directly (stability, efficiency); without enough data, a proposed design can perform molecular simulation in an efficient (e.g., in terms of computation) andAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) effective (e.g., in terms of description of the interaction) architecture. The data source can be a data set that includes the molecular configurations (inputs), and respective energy and force (labels). Embodiments can be used to model the multi-scale interactions of the atoms, and output the desired property, for example stability and efficiency, by simulating the atomistic model using a molecular dynamic tool. The output desired property can then be used to operate wet lab equipment to test or produce an mRNA vaccine and method of delivery, to have an interactive vaccine design process with an expert providing feedback on the output with iterative operation of the method, or to generate diagnosis or personalized treatments in an automated manner.

[0064] Embodiments of the present disclosure thus provide for improvements to computers in machine learning systems, such as improvements to generation of peptide-major histocompatibility complex and T-cell receptors (peptide-MHC-TCR) binding complexes, and for prediction of binding affinity for of molecular system for drug design using molecular simulation or generative models.

[0065] For instance, peptide-MHC-TCR can describe the effect of T-cell receptors working in the biological system, including the human system. To develop personalized vaccines (e.g., for cancer), the T-cell receptor should be able to recognize cancerous cells. By designing proper proteins that can present themselves and bind to the T-cell, embodiments can then incentivize the immune system to attack the cancerous cells. One or more embodiments of the present disclosure provide improvements to ML – AI models that can be used to model generative models to either directly generate protein with desired properties (binding affinity), or to simulate (e.g., using a molecular dynamic) the property of the proposed molecule. The data source can be a data set that includes the molecular configurations of the MHC complex, the proteins and the T-cell receptors (e.g., as inputs), and respective free energy or binding affinity (e.g., as labels). Embodiments of the present disclosure can then be used to model the multi- scale interactions of the atoms, and output the binding affinity and the proposed protein. The output binding affinity and proposed protein can then be used to produce a drug and method of delivery. For example, the output can be used to operate wet lab equipment to test or produce the drug and method of delivery, to have an interactive drug design process with an expert providing feedback on the output with iterative operation of the method, or to generate diagnosis or personalized treatments in an automated manner.

[0066] Embodiments of the present disclosure thus provide for improvements to computers in machine learning systems, such as multibody tracking and prediction.

[0067] For instance, low earth orbit (LEO) satellites are used for telecommunication and earth observation. A LEO satellite is a moving object that is affected by various forces and disturbances. Accurate and fast prediction can be important when prediction modules need toAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) run inside the satellite. The prediction can depend on the relative position of the interfering factors, and rotation and translation invariance (or equivariance) can be a natural requirement for this setting. Asteroids or space debris can also be modelled in a similar way. The data source can be a data set that includes the historical data of the satellite and affecting terms (e.g., sun, moon, earth positions), and the dataset can be then split into input / output for training. One or more embodiments of the present disclosure can be used to model the multi-body interactions (e.g., when an orbit of one satellite depends on the position of other satellites) and output a predicted trajectory of one more satellites or orbital object (e.g., asteroids). The output predicted trajectory can then be used to affect a change in the path of the one or more satellites, orbital object, or a satellite to-be-added to LEO, for example by operating satellite equipment.

[0068] Embodiments of the present disclosure thus provide for improvements to computers in machine learning systems, such as biopolymer and polymer property prediction.

[0069] For instance, polymers and bio-polymers have properties that change with different compositions of the material. Given a specific 3D structure (confirmation) and with the atom type, one or more embodiments of the present disclosure can predict the property (e.g., tensile strength, durability, viscoelasticity, reactivity, melting point, conductivity) of the given polymer or bio-polymer. Since the polymer is a structure whose structure is not completely predictable at low level, but can only be described statistically, embodiments of the present disclosure can separate the contribution of the main chain and the side chain and their mixture and predict the property as a contribution of the each component. The data source can be a data set that includes the main chain structure, the side chain and the composition and possible the topology, and the properties (e.g., as labels). One or more embodiments of the present disclosure can be used to model single components (e.g., main chain and side chain) separately and then the results are concatenated with additional information and a multilayer perceptron (MLP) can then be used for the final prediction and output the polymer property based on its structure. The output polymer property can then be used to develop new polymer designs and applications (e.g., improved manufacture of paper, textiles, and biodegradable plastics). For example, the output can be used to generate manufacturing plans or blueprints, order materials or to operate manufacturing or lab equipment to produce a polymer.

[0070] Moreover, embodiments of the present disclosure can be practically applied to use cases to effect further improvements in a number of technical fields, in addition to those discussed above, including, but not limited to, medical and healthcare (e.g., digital medicine, personalized healthcare, drug or vaccine development, prescription, treatment, etc.), material development, public safety, and smart cities (e.g., automated traffic or vehicle control, smart districts, smart buildings, smart industrial plants, smart agriculture, energy management, etc.).Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0071] One or more embodiments of the present disclosure can comprise the following steps that can be performed: 1) Collect dataset and select a geometric algebra, typically Cl(3,0,0) or Cl(3,0,1) based on the geometry (Euclidean geometry or projective geometry) of the dataset, and map (embedding) the input and output data in the geometric algebra. 2) Create a network that based on (1) extending geometric algebra (or Clifford) based neural network with dense and / or sparse pooling; and / or (2) using the multi-blade projection to define the pooling or clustering of the nodes that describes the molecular system. 3) Train the neural potential or the generative process (diffusion model) to propose structure (either final state or as a step in a simulation). For example, when the energy of molecular systems matters, the neural potential can be chosen, and the molecular structure can be obtained in a way that it minimizes the energy of the systems. Additionally, and / or alternatively, the diffusion models that perturb the systems’ structures could be also used to generate potential candidates of the molecular systems. The general tasks of those two models can be different; for example, the former model can predict the energy of molecular systems and the latter models can predict the structure of molecular systems, but both methods can be applicable to the structure proposal. Then, the network can be integrated into a large system; and 4) Map back the Geometric Algebra output to the real application domain; the scalar component represents quantities that are invariant to rotation and translation, as for example the mass of the atom, while the vector component can represent information (in different channels) of the position and momentum of the atoms. Lastly bivector could represent information such as angle between atoms or areas. In general the use of multivector introduces flexibility to map various information either in a specific blade or in an additional channel. Thus, the atoms of mRNA are represented in their configuration as vector components and the prediction output the forces (accelerations) as vector component, while the atom energy is presented as scalar. Similarly, in the context of low earth orbit satellites’ prediction, the vector components of a multivector represent the coordinate of a satellite, while the scalar part represents mass of the satellite.

[0072] Embodiments of the present disclosure provide for the following improvements and technical advantages over existing technology: 1) Embodiments can extend geometric algebra (or Clifford) based neutral network with dense or sparse pooling, where the pooling and unpooling conserves equivariance to geometric primitives and in particular E(3) to model multi-scale interactions (e.g., where the NN is a Message passing architecture); andAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) 2) Embodiments can use the multi-blade projection to define the pooling or clustering of the nodes.

[0073] Embodiments of the present disclosure provide for many advantages over the current state of the art, such as improved accuracy. For example, the table of FIG.7 shows that with the hierarchy of one or more embodiments of the present disclosure, the performance of the protein datasets where long-range interactions are considered is improved (e.g., with an error reduction rate of 7%) above the Clifford group-equivariant message passing neural networks (GA) of conventional approaches such as Ruhe, David, et al. “Clifford Group Equivariant Neural Networks” arXiv preprint: arXiv:2305.11141, which is incorporated by reference herein. In the embodiment of FIG.7, dropping nodes and clustering nodes are the node dropping method performed at step 412 and cluster assignment method performed at step 410, respectively, of FIG.4.

[0074] Additionally, one or more embodiments of the present disclosure allow the base model to see nodes that are at distance that is exponential to the number of layers.

[0075] FIG.8 shows the results of an embodiment of the present disclosure on another dataset, where the dataset is an interaction of multiple bodies. It can be considered as a simplified example of satellite orbit prediction. As shown in FIG.8, the prediction error(ൈ 10ିଶ) is provided on various types of simulated datasets. The “Multiple System” contains^^ ൌ 5 different systems. For each column, (^^,^^ / ^^) indicates that each system contains Mcomplexes of average size ^^ / ^^. The results are averaged across 3 runs. “OOM” denotes out of memory.

[0076] An embodiment of the present disclosure, referred to as the hierarchy-based Clifford group equivariant message passing neural networks (HCGE-MPNN) will be further described below. The embodiments forms part of the present disclosure and provides further background and description of exemplary embodiments of the present disclosure, which can overlap to some extent with some of the information provided above. To the extent the terminology used to describe the exemplary embodiments can differ from the terminology used to describe the above embodiments, a person having skill in the art would understand that certain terms correspond to one another in the different embodiments. Features described in the embodiment of HCGE- MPNN can be combined with features described above in various embodiments.

[0077] An embodiment of the present disclosure provides a HCGE-MPNN, e.g., a Clifford group equivariant U-Net with skip connection. The embodiment integrates the expressivity of Clifford group-equivariant layers with hierarchical pooling / unpooling in an encoder-decoder fashion. The architecture of the embodiment admits major classes of pooling methods, sparse and dense pooling methods. The embodiment also introduces a Clifford group invariantAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) projection operator, a generalized projection operator defined on the Clifford space, to make the end to end architecture equivariant to Clifford group action. The embodiment’s method can outperform other conventional (e.g., Clifford-)Equivariant MPNNs by up to 7% in prediction mean square error (MSE) for multi-N body datasets and 22% for motion capture dataset.

[0078] Simulations can be important to modelling and understanding multi-body physical systems, such as inorganic material or biological systems, and can be used to compute properties of the physical systems. However, simulations can be very sensitive to the accuracy of the description of the governing laws, which typically includes a geometric interplay between objects such as symmetry of the systems.

[0079] Recent geometric deep learning has made progress for modelling multi-body physical systems. For instance, multi-body physical systems can be represented with geometric graphs. Geometric graphs can be embedded in a topological space such as a metric space or a manifold. The node features of these graphs can include quantities of geometric features, like positions or velocities, and these quantities can be transformed under spatial operations like rotations, reflections, or translations. E(n)-equivariant graph neural networks (EGNNs) can be designed to preserve these transformation by imposing an invariance and / or equivariance structure tailored to tasks.

[0080] Clifford neural network models have been proposed to model high-order interactions of physical systems. Clifford algebra can represent higher-order elements such as bivectors and trivectors and can be capable of modelling not only low-order information such as magnitude of vector but also high-order information such as area or volume spanned by multiple vectors. The expressive power of the networks with Clifford representation can be shown in diverse applications. However, while the Clifford equivariant neural networks can be capable of higher- order information processing, the prior art networks only conduct flat information propagation, which can hinder propagation of the spatial and dynamical information.

[0081] An embodiment of the present application, HCGE-MPNN, is an end-to-end trainable model capable of discovering important substructures and respecting Clifford group action. HCGE-MPNN comprises an encoder and decoder equivariant to Clifford group, such as ^^^3^ group actions (e.g., that represent translation and rotations) represented as elements in the algebra. The encoder can encode the input of the system from fine-scale to coarse-scale and the decoder can restore both the topological and numeric information of the input systems. The embodiment introduces two pooling operations, sparse and dense pooling, and shows their equivariance by introducing a novel projection operator on the Clifford space. The embodiment is demonstrated in two benchmarks of multi N-body systems and motion capture datasets,Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) showing up to 22% reduction in prediction accuracy (e.g., reduction in prediction error) compared to conventional geometric group equivariant methods.

[0082] Clifford Algebra and Clifford Neural Networks: The embodiment starts by introducing Clifford algebra, also known as geometric algebra, over the Euclidean space ^^^and some of its properties. The Clifford algebra ^^^^^^^^,^^^ is the quotient space over the tensor algebra over ^^^with an equivalence relation q(v) = v ⊗ v (∀v ∈ Rn) defined by a quadratic form q : Rn→ R. Since the anti-commutative relation v ⊗ w + w ⊗ v = q(v + w) − q(v) − q(w) holds for ∀v, w ∈ Rn, Cl(Rn, q) is a finite (up to 2n) dimensional linear space and every element x ∈Cl(Rn, q) can be written with a finite index I as x = ∑^∈ூ ^^^ ∙ ^^^,^ ⊗ ^ ^ ^ ⊗ vi,ki, ci ∈ R, vi,j ∈ R .Here, the expression v ⊗ w of elements v, w ∈ Rnthe geometric product of v, w,which is a natural generalization of the canonical and outer products on Rn. Hereinafter, the notation of the tensor product is used on the tensor algebra for the geometric product. The geometric product defines vector subspaces Cl(m)(Rn, q) (m = 0, 1, ^ ^ ^ , n), called grades, whose elements are m-fold geometric product of v ∈ Rn. In the embodiment, x(m)is written for x ∈ Cl(Rn, q) as the component of x belonging to Cl(m)(Rn, q). An advantage of the embodiment (e.g., using Clifford algebra) is the flexibility to allow for algebraic representation of physical systems and manipulation of geometric quantities. This geometric arithmetic is injected as an inductive bias to various machine learning models to solve problems in some applications, and implementation in PYTORCH, JAX (e.g., GOOGLE JAX), and TensorFlow are also available. Although the prior art models incorporate various modes of Clifford algebra, it is yet unclear which kinds of symmetry the prior art models can preserve, which hinders the prior art models from being applied to diverse application domains.

[0083] Equivariant Clifford Neural Networks: a concept of equivariance for Clifford neural networks has been introduced in the prior art and proposes Clifford group equivariant (graph) neural networks (CGENNs). The concepts are the Clifford group and its associated representation into the endomorphisms of Cl(Rn, q). The Clifford group Γ(Rn, q) is a set of invertible multi-vectors belonging to even or odd grades equipped with the induced geometric product. Then a representation ρ : Γ(Rn, q) → End(Cl(Rn, q)), called an (adjusted) twisted conjugation, is defined and induces the following equivariant property: for any multivariate polynomial F ∈ R[T1, ^ ^ ^ , Tl] , ρ(ω)F (x1, ^ ^ ^ , xl) is = F (ρ(ω)x1, ^ ^ ^ , ρ(ω)xl), ∀x1, ^ ^ ^ , xl∈ Cl(Rn, q), ∀ω ∈ Γ(Rn, q), in which the geometric product is performed as a product. When the target of ρ is restricted to the base space Rn, ρ induces an isomorphism between Γ(Rn, q) and the orthogonal group on Rnup to scalar R×. This means ρ defines a high-order action on the Clifford space, while it models the Euclidean rotational action on the underlying Euclidean space. Although CGENNs can respect the symmetry of physical systems while allowing forAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) representation of higher-order interaction of objects, CGENNs (and non-equivariant Clifford neural networks) still only conduct flat information propagation.

[0084] Notation: each input representing multi-body systems is represented as a graph G(V, E) consisting of N nodes V, representing particles, and the set of edges E ⊂ V×2, where each edge represents the existence of multi-body interactions. Each node i is assigned with a feature (zi(0), hi(0)): zi(0)representing a directional vector in Rmn, such as the concatenation of location and velocity of particles, and hi(0)∈ Rsis the non-directional feature, such as the charges or category types of the atoms. The embodiment embeds both features into Cl(Rn, q), and denotes corresponding embeddings by (z˜i(0), h˜i(0)) ∈ Cl(Rn, q)×(m+s). The embodiment assumes the natural extension of the group action ρ(ω) to the product space Cl(Rn, q)×(m+s)and zi(0)is equivariant to the orthogonal action while hi(0)is invariant to the action.

[0085] FIG.9 is a schematic diagram of an embodiment of the present disclosure of the HCGE-MPNN 900. HCGE-MPNN is composed of an encoder 902 and a decoder 904, and receives a representation of a graph as an input 906 and outputs a representation of a graph as an output 908. Each module has corresponding Clifford group equivariant pooling an unpooling layers ^^^^^. A task of the HCGE-MPNN is to predict the future state of system (e.g., a predicted vector) at a certain time T , given directional state zi(0); that is, to find a model Φ able to predict a future state zi(T )for given T and zi(0). The model Φ should be equivariant to Clifford group action, which means:^^^^^^^^^^^^^ ^ ൌ ^^^^^^^^^^^^^˜ ^^^^^^ , ^^˜^^^^^^^^ே^^^^^^^^^ୀ^) = ^^^^^^^^^^^^^˜^^ , ^^˜^^^^ே^ୀ^), ∀ω ∈ Γ(Rn, q).

[0087] When Φ is based on message passing neural networks, the embodiment formulates HCGE-MPNN as an encoder-decoder architecture with skip connection: ^^^^^௪^ ^,^^^^௪ ൌ ^^^^ െ ^^^^^^^^^^^^൫^^^^ ^^^^ ^^^^^ ^,^^ ൯, ^^^^௨௧^ ^,^^^௨௧ ൌ ^^^^ െ ^^^^^^^^^^^^൫^^^^^^௪^,^^^^௪൯,^ .operations CE-Pool(h)and message passing with Clifford Group Equivariant MPNNs (CGE- MPNNs) iteratively, up to H times. The decoder maps information from the low level (h = H) to the high level graphs (h = 1) using the unpooling operator CE-UnPool(h).

[0088] CE-Pool(h)(1 ≤ h ≤ H) is responsible for making decisions on which nodes in a graph are pooled and reconstructing a graph according to the pooled nodes. The pooling process is composed of three processes: scoring stage, node creation stage, and topology reconstruction stage (e.g., scoring stage 402, node creation stage 404, and topology reconstruction stage 406 as show in FIG.4). In the scoring stage, a model of HCGE-MPNN 900 first takes a graph G(h)as anAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) input 906 and computes scores {ri}i∈V (h) over all the nodes. In the node creation stage, new nodes of a pooled graph are generated based on the scores computed in the scoring stage. The embodiment can execute two ways to perform node creation: sparse pooling method 910 and dense pooling method 912. After the new nodes are generated, edges are reconstructed.

[0089] Clifford group-invariant scoring function: scores {ri}i∈V(h) should be invariant to Clifford group action since pooled graphs should have the same topological structure up to equivariant configurations. To provide this invariance, the embodiment introduces a multi-blade projection over Cl(Rn, q)×C: ^^^ ^^^^

[0090] denotesฮ^^^ఏ^^^^ ⊗ ^^^ఏ^^^^^^^ฮ. The embodiment then has the following theorem for the multi-bladex, p ∈ Rnand C a positive integer. Then, (i) if C = 1 and pθ(x˜) = p˜, xc= x˜ ∈ Cl(1)(Rn, q), then the multi-blade projection is the projection of x over p in Rn. (ii) If pθis equivariant to Γ(Rn, q), the multi-blade projection is invariant to Γ(Rn, q).

[0091] Theorem 3.1 indicates that the multi-blade projection is a generalization of the projection defined on Rnand guarantees to generate the same scores no matter how the Clifford group acts on multi-vectors. The multi-blade projection fθis used as the score function in the rest of the HCGE-MPNN embodiment.

[0092] Sparse pooling and unpooling: sparse pooling 910 (e.g. ^^^^ െ ^^^^^^^^^^^^^^^^^) first computes a score for each node (requiring ^^^^^^ space), and reduces a graph G(h)into G(h+1)by keeping only the top ⌈rN ⌉ scoring ones and dropping the rest. Here r is the ratio that limits the amount of pooled nodes and is given as a hyperparameter. The embodiment stores all the edges ^^^^^and ^^̃^^^^^ , ℎ^^^^^ ^ of dropped nodes. The stored information is used when unpooling (e.g.,unpooling 914) such that ^^^^^ െ ^^^^^^^^^^^^^^^^^^^^^ to ^^^^ି^^:^^^^^ is usedto restore graph topology for ^^^^ି^^and the restored nodes areΓ^^^^, ^^^ invariant nodeas mean of node features at ^^^^^ or zero vectors. Theoverall pooling architecture is similar to TopK-pooling, butis not rotational equivariance since scoring function is not ^^^^^^ invariant.

[0093] Dense pooling and unpooling: dense pooling 912 (e.g., ^^^^ െ ^^^^^^^^^^^ௗ^^^^) is an extension of the multi-blade projection, in which all nodes in ^^^^^are explicitly assigned to aAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) pooled node in ^^^^ା^^. The embodiment first pre-defines a number of clusters, denoted as^^^^ା^^^^^^^^ ൌ ^^^, as the number of nodes for the pooled graph ^^^^ା^^. To ensure the^^^^ െ ^^^^^^^^^^^ௗ^^^^, the embodiment then computes confidence scores with the multi-but with the same number of multi-blade projections ^^^^^^ ^ఏ ൫^^^^൯^^ஸ^ஸ^^^శభ^ asthe clusters ^ൌ ^^^^ା^^^ and defining the ^^^^^ ൈ ^^^^ା^^ assignment matrix ^^^^^ ൌ ^^^^^^^,^^ as follows:^^^^^^^^ ^^^൬^^⊗ ^^ ^^^^^ æ^^^^^^^^^ ^,^ ఏ^ೖ^ ^,^^ ^^ ^ö

[0094] withchannel- are asfollows: ^ 1 ^^^^ା^^ ^1 ∙^ ^ ^ ^^^^^ ⊙ ^^^^^, 1 ∈ ^^^^ ^1^

[0095] (identified below),the embodiment also stores ^^ , and uses the stored edges’ information as edges for ^^^^ା^^to be pooled while restoring node features (e.g., during dense unpooling 916) as follows: ^^^^^ ൌ ^^^^^^^^^ା^^,^^^^^^ ൌ ^^^ ^^^^^^^, ... , ^^^^^^^ ^3^

[0096] Theseis Euclidean group-equivariant, while the method according to an embodiment of the present disclosure is equivariant to the Clifford group action that can respect high-order symmetry of systems.

[0097] For these two sparse and dense (un)pooling methods, the embodiment can impose equivariance to Clifford-group action, by adopting multi-blade projection. The claim is derived as a corollary of Theorem 3.1 (identified above).

[0098] Corollary 3.2: Let pθbe a trainable Γ(Rn, q)-equivariant endomorphism of Cl(Rn, q). Then, the sparse and dense (un)poolings are Γ(Rn, q)-equivariant.

[0099] Training objective: the embodiment defines the training objective of HCGE-MPNN, depending on the pooling modes:Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) 1ே^௨௧^௧^ì ^^^ ฮ^^^^ ^^^^ െ ^^ฮ if sparse ^^^ᇱ ^^ୀ^

[0100] can becapableinstead for the dense pooling mode. The second term (without the weight ^^) in the dense pooling is the mode connectivity loss. The purpose of having this term is to encourage more connections within the pooling nodes and smaller cuts among clusters.

[0101] The multi N-body dataset consists of N charged particles ^^^^ ,^^^ , ^^^^ே^ୀ^that belongs to ெ M disjoint complex systems ^^^^^^ୀ^ , where ^^^ ,^^^ , ^^^ are particles’ position, velocity and change.The task consists of predicting the final position of all the particles after T = 1500 steps, given the initial positions and velocities. The accuracy of the prediction is measured in MSE, and the results are reported in Table 1 below.

[0102] Table 1: Prediction error (×10−2) on multi N-body dataset: the “Multiple System” contains J = 5 different systems. For each column, (M, N / M ) indicates that each system contains M complexes of average size N / M . Results averaged across 3 runs. Single System (J = 1) Multiple Systems (J = 5) (3,3) (5,5) (3,3) (5,5)EGNN 12.69 (0.19) 15.37 (0.13) 13.33 (0.12) 15.48 (0.16) EGHN 11.58 (0.10) 14.42 (0.80) 12.80 (0.56) 14.85 (0.30) CGE-MPNN 9.44 (0.41) 8.18 (0.14) 9.60 (0.21) 7.18 (0.36) HCGE-MPNN 8.75 (0.24) 8.21 (0.13) 9.14 (0.40) 7.29 (0.61) (sparse) Table 1

[0103] The method of an embodiment of the present disclosure outperforms the two baselines and also performs better than CGE-MPNN for most of the settings. Even for the larger systems such as (5, 5) systems, the method is competitive to CGE-MPNN. The results indicate the efficacy of employing hierarchical structure of underlying systems together along with arithmetic of the Clifford algebra as inductive bias for the neural network.

[0104] Motion capture: an embodiment of the present disclosure (e.g., HCGE-MPNN) was evaluated on Carnegie Mellon university’s (CMU’s) motion capture database, e.g., on the walking dataset (subject #35), in the task of predicting future configuration. For each of the models, an experiment was run 3 times and the results were averaged. The results are shown in Table 2 below.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0105] Table 2: Prediction error MSE (×10−2) on motion-capture dataset. Results are averaged across 3 runs. Walk (#35)EGNN 7.9 (0.4) EGHN 7.2 (0.1) CGE-MPNN 7.7 (0.5) HCGE-MPNN (sparse) 6.3 (0.2) HCGE-MPNN (dense) 6.0 (0.1) Table 2

[0106] HCGE-MPNN outperforms the other strong baselines and the dense mode achieves the best among the other models. The number of parameters for HCGE-MPNN are around 70,000 which is comparable to other baselines.

[0107] Clifford equivariant dense pooling and unpooling: dense pooling ^^^^ െ ^^^^^^^^^^^ௗ^^^^is an extension of the multi-blade projection, in which all nodes in ^^^^^are explicitly assigned to a pooled node in ^^^^ା^^. The embodiment first pre-defines aof clusters, denoted as the number of nodes for the pooled graph ^^^^ା^^. To ensure the^^^, the embodiment thௗ^^^^en computes scores with the multi- blade projection, with the same number of multi-blade projections^^^^^^ఏ൫^^^^^൯^as the ^ஸ^ஸ^^^శభ^clusters ൫ൌ ^^^^ା^^൯ and defining the ^^^^^ ൈ ^^^^ା^^ assignmentfollows:^^^^^^^^⊗ ^^ ^^^^^^^ ^^^^^^൬^ ^^^

[0108] with channel-wise normalization. Then, the node feature and edges of ^^^^ା^^are computed as follows: ^ 1 ^^^^^^ ^^^^^^

[0109] also stores ^^^^^, and uses the stored edges’ information as edges for ^^^^ା^^to be pooled while restoring node features as follows:Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) ^^^^^ ൌ ^^^^^^^^^ା^^,^^^^^^ ൌ ^^^ ^^^^^^^, ... , ^^^^^^^ ^6^

[0110] is Euclidean group-group action that can respect high-order symmetry of systems.

[0111] The training objective of HCGE-MPNN can be defined, for example, depending on the pooling modes: 1ே^^^^௨௧^^^^ െ ^^^௧^ì^ ฮ ^ฮ if sparse ^^ ^^^ᇱ ^^ୀ^ if dense ி

[0112] and ^^ = 2 canuses norm instead for the dense pooling mode. The second term (without the weight ^^) in the dense pooling is the mode connectivity loss. The purpose of having this term is to encourage more connections within the pooling nodes and smaller cuts among clusters.

[0113] Proof of theorem 3.1: (i) since ^^ఏ^^^^^ ൌ ^^^ ^^^^^^ ^^^ ൌ ^^^ ∈ ^^^^^^^^^^^, ^^^, ^^^ and ^^^ can bewritten as: ^ ^ ^^^ ൌ^^^^^^^ ,^^^ ൌ^^^^^^^ , ^^^ , ^^^ ∈ ^^

[0114] Here,^^^^is the by taking theproduct, ^^^ ⊗ ^^ఏ^^^^^ ൌ ^^^ ⊗ ^^^^ ^ ^^and therefore get:^ ^^^ ^^ ^^^^^^^^ ^^^^^

[0115] In a manner similar to^ ^^ ^ ^ ^^ ^ ^^^^ ^^^^^Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) Therefore, ൫^^^ ⊗ ^^ఏ^^^^^൯^ ∑ ^^^ ^^ ^ ^^, ^^ ^ൌ ^ୀ^^ ^∑^^^ ^^ൌ ^^ ^ ^^,^^ ^

[0116] (ii) note that following relation

[0117] Then, ^^^^^^^^^^ ⊗ ൫^^ఏ^^^^^^^^^^൯ ൌ ^^^^^^^^^^ ⊗ ^^^^^^^^^ఏ^^^^^ൌ ^^^^^^^^^ ⊗ ^^ఏ^^^^^

[0118] Since Clifford group acts on the scalar component identically, ^^^^^^൫^^ ⊗ ^^ఏ^^^^൯^^^ ൌ^^^ ⊗ ^^ఏ^^^^^^^^Similar argument holds for ฮ^^ఏ^^^^^^^ฮ, and this completes the proof.

[0119] Proof corollary 3.2: sparse case: since the scoring function is invariant to the Cliffordgroup action, the pooled nodes are same across the inputs acted by any ^^ ∈ Γ^^^^, ^^^. Thepooled nodes are sampled from the nodes in a graph, and the features on the sampled nodes areequivariant to Γ^^^^, ^^^. Unpooled nodes on the other hand are equipped with invariant features,such as zero multi-vectors or mean of the multi-vectors in the Clifford space, and the unpoolingoperation is also equivariant to Γ^^^^, ^^^ action.

[0120] Dense case: the assignment matrix S is invariant to Γ^^^^, ^^^ action, since the multi-blade projection is Γ^^^^, ^^^ invariant. By theorem 3.2, linear transformation is Γ^^^^,^^^equivariant, and therefore Eq. 4 is Γ^^^^, ^^^ equivariant. On the other hand, the adjacent matrix^^^^ା^^ in Eq. 5 is invariant to Γ^^^^, ^^^ action, since S is Γ^^^^, ^^^ invariant. Therefore, the densepooling method is Γ^^^^, ^^^ equivariant. Similar argument applies to Eq. 6, therefore the denseunpooling is also Γ^^^^, ^^^ equivariant.

[0121] Experimental settings: multi N-body dataset. The multi N-body dataset consists of Ncharged particles ^^^^ , ^^^ , ^^^^ேெ ^ୀ^that belongs to M disjoint complex systems ^^^^^^ୀ^, where^^ , ^^^ , ^^^ are particles’ position, velocity, and change. Within each systemparticlesrigidly connected in pairs (e.g., bonds), triangles, and tetrahedrons. The dynamics of the M systems is driven by electromagnetic forces between pairs of particles. The task consists of predicting the final position of all the particles after ^^=1500 steps, given the initial positions and velocities. ^^ independent systems are samples, each composed of M independent systems, with the number of participles sampled from uniform distribution with mean N / M. This system is called (M,N / M,J). Within each system ^^^the particles are rigidly connected in pairs (bonds),Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0122] As graph topologies over which message passing is performed with HCGE-MPNN, the embodiment uses the graph topologies Aglobal and Alocal. In this experiment, pθis modeled with CGE-MPNN and uses Alocalas an input of pθto compute the scoring function of CE- Pool(h). Aglobal is used to update node features by CGE-MPNNs composing CE-Encode and CE- Decode. Both Alocaland Aglobalare pooled based on the scores computed over Alocal, and pooled Alocal and Aglobal are used in the next round of pooling process.

[0123] Throughout the experiment, the embodiment sets 2-layers and 14-channels for hidden layers of CGE-MPNN in the model. Pooling depth is set to be 1 and the ratio of pooled nodes (for sparse pooling) is 0.5. For dense pooling, the number of cluster is set to be same as the number of system in each of the experiments. For the N-body simulation system, linear prediction is used as a comparison, e.g., an equivariant graph neural network (EGNN) and an equivariant graph hierarchy-based neural network (EGHN). The accuracy of the prediction is measured in MSE.

[0124] Motion capture dataset: the CMU motion capture database is a large collection of motion capture recordings for various tasks (such as walking, running, and dancing) performed by human subjects. The embodiment focuses on recorded walking motion data of a single subject (subject ♯ 35). The data is in the form of 313D trajectories, each tracking a single joint of the human body. For the training dataset, the same splitting strategy is adopted as Han et al. (2022), with 1100 frame pairs for training, 600 for validation, and another 600 for tests. The interval between each pair is 30 frames in both scenarios. In this task nodes are the joints and edges are defined as the neighbor of nodes. The embodiment again uses the same graph topologies Aglobaland Alocal.

[0125] For the number of clusters K for dense pooling, the embodiment empirically finds that K = 4 yields promising results. The embodiment sets 2-layers and 14-channels for hidden layers of CGE-MPNN in the proposed method. The pooling ratio for the sparse pooling is 0.5. In this experiment, HCGE-MPNN is compared with Euclidean group equivariant models such as EGNN and EGHN as well as CGE-MPNN. The prediction accuracy is measured in MSE.

[0126] Results of dense pooling: multi N-body datatset: Table 3 below is the result of N- body dataset experiments. The embodiment’s sparse-pooling mode performs better or competitive to CGE-MPNN. However, the dense-pooling mode performs worse even than CGE- MPNN. The model with the both modes can suffer from overfitting during training even with small hyperparameters such as message-passing layers or the channel dimension of CGE- MPNN. This could happen since the Clifford representation of the canonical vectors retains abundant information compared to the original vectors, which has huge impact on overfitting when the training data is small.Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0127] Motion capture dataset: Table 2 is the result of motion capture dataset. Both of the pooling modes outperform all the baselines and the dense-pooling mode achieves the best performance among the others. In general, the dense-pooling mode pools nodes to create small graphs with smaller diameters to facilitate propagation of graph information while keeping the number of message-passing layers small. Therefore, the result indicates the advantages of the dense-pooling method, since the graph topology of the data is sparse and its diameter is about 15, which can require the flat message passing methods to have deep message passing layers to capture long-range dependency involving taking risks of over-smoothing or over-squashing. Although the sparse-pooling method also outperforms baseline methods, this mode might not facilitate the message propagation in a manner of dense pooling since it can drop nodes and might not increase the connectivity of the pooled graph.

[0128] Table 3: Prediction error (×10-2) of dense and sparse pooling modes on a multi N- body dataset. The “Multiple System” contains J = 5 different systems. For each column, (M, N / M) indicates that each system contains M complexes of average size N / M . Results averaged across 3 runs. Single System (J=1) Multiple Systems (J=5) (3,3) (5,5) (3,3) (5,5) EGNN 12.69 (0.19) 15.37 (0.13) 13.33 (0.12) 15.48 (0.16) EGHN 11.58 (0.10) 14.42 (0.80) 12.80 (0.56) 14.86 (0.30) CGE-MPNN 9.44 (0.41) 8.18 (0.14) 9.60 (0.21) 7.18 (0.36) HCGE-MPNN (dense) 11.63 (1.45) 10.95 (1.39) 11.79 (1.28) 9.34 (0.64) HCGE-MPNN (sparse) 8.75 (0.24) 8.21 (0.13) 9.14 (0.40) 7.29 (0.61) Table 3

[0129] Ablation study: the ablation study of HCGE-MPNN is performed with the motion capture dataset. The study for each of the dense and sparse modes is conducted and the scores are reported in Table 4 below. For the embodiment’s dense pooling method, the number of clusters was changed to 3 and 7, both of which yielded worse performance. The hidden channels were also modified from 14 to 8, and HCGE-MPNN with 14 hidden channels performed better. Omitting the connectivity loss, in which the topology of pooled graphs are regularized to ensure being sparse, can degrade performance, showing the advantage of having the connectivity loss in the training objective. The advantage of having sparse pooling for this experiment was also investigated. Modifying the hidden channel from 14 to 8 and pooling ratio from 0.5 to 0.2 and 1.0 can degrade performance. Setting pooling ratio to be 1.0 can be equivalent to CGE-MPNN model with the skip connection. Experiments were not performed with larger hidden-channelsAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) and larger hidden layers to keep the maximum computational budget same across all the ablation models.

[0130] Hierarchical graph neural networks (pooling and unpooling): hierarchy is common in various domains and it is typically employed in the concept of information coarsening, where multi-resolution representations of information are obtained at different levels of abstraction along the hierarchy. Coarsening provides simple, yet effective methods to extract features, and the idea can be key in, for example, image-processing and has recently played a vital role in unstructured-data processing. These studies focus mainly on generic data, and more importantly, they do not respect the symmetry of geometric data stemmed from physical systems.

[0131] Graph Neural Networks (GNNs) have achieved a substantial improvement over various tasks, from node-level tasks such as node classification to graph-level tasks such as graph regression and classification. In order to obtain an effective graph representation, many designs of graph hierarchical pooling, an operation to iteratively coarsen graphs into smaller size, have been proposed. These methods typically involve learnable score functions to compute likelihood of nodes to be pooled. Such formulation includes sparse method, which computes a score for each node and reduces the graph by keeping only a fraction of top scoring ones and dropping the rest, and dense method that computes for each node a soft-assignment to a fixed number of clusters. Although the methods can effectively obtain representation of graphs, the methods do not respect symmetry of physical systems, which limits their generalization on real world geometric data such as ones in the context of multi-scale modelling. Some exceptions include models formulated in an equivariant encoder-decoder form with skip-connections. The method of the embodiment is also aligned with the encoder-decoder form, but instead respects Clifford group action, that involves higher-order action to geometrical data.

[0132] FIG.10 provides a visualization of the output of HCGE-MPNN and ground-truth for multi N-body dataset. Graphs 1002, 1004, 1006, 1008, 1010, and 1012 represent the 3D coordinates of the prediction and ground truth particles composing (3, 3, 1)-system. Edges correspond to the frame of each sub-system and each sub-system is fully connected. As shown by the proximity and overlap of the two multi body sets, the predicted body (indicated by ൈ nodes) closely (e.g., accurately) predicts the ground truth body (indicates by + nodes).

[0133] FIG.11 provides a visualization of the output of HCGE-MPNN and ground-truth for motion capture dataset. Graphs 1102, 1104, 1106, 1108, 1110, and 1112 represent the 3D coordinates of the prediction and ground truth joints representing a body. As shown by the proximity and overlap of the two bodies, the predicted body (indicated by ൈ nodes) closely (e.g., accurately) predicts the ground truth body (indicates by + nodes).Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO)

[0134] Referring to FIG.12, a processing system 1200 can include one or more processors 1202, memory 1204, one or more input / output devices 1206, one or more sensors 1208, one or more user interfaces 1210, and one or more actuators 1212. Processing system 1200 can be representative of each computing system disclosed herein.

[0135] Processors 1202 can include one or more distinct processors, each having one or more cores. Each of the distinct processors can have the same or different structure. Processors 1202 can include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuitry (e.g., application specific integrated circuits (ASICs)), digital signal processors (DSPs), and the like. Processors 1202 can be mounted to a common substrate or to multiple different substrates.

[0136] Processors 1202 are configured to perform a certain function, method, or operation (e.g., are configured to provide for performance of a function, method, or operation) at least when one of the one or more of the distinct processors is capable of performing operations embodying the function, method, or operation. Processors 1202 can perform operations embodying the function, method, or operation by, for example, executing code (e.g., interpreting scripts) stored on memory 1204 and / or trafficking data through one or more ASICs. Processors 1202, and thus processing system 1200, can be configured to perform, automatically, any and all functions, methods, and operations disclosed herein. Therefore, processing system 1200 can be configured to implement any of (e.g., all of) the protocols, devices, mechanisms, systems, and methods described herein.

[0137] For example, when the present disclosure states that a method or device performs task “X” (or that task “X” is performed), such a statement should be understood to disclose that processing system 1200 can be configured to perform task “X”. Processing system 1200 is configured to perform a function, method, or operation at least when processors 1202 are configured to do the same.

[0138] Memory 1204 can include volatile memory, non-volatile memory, and any other medium capable of storing data. Each of the volatile memory, non-volatile memory, and any other type of memory can include multiple different memory devices, located at multiple distinct locations and each having a different structure. Memory 1204 can include remotely hosted (e.g., cloud) storage.

[0139] Examples of memory 1204 include a non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any kind of optical storage disk such as a DVD, a Blu- Ray® disc, magnetic storage, holographic storage, a HDD, a SSD, any medium that can be used to store program code in the form of instructions or data structures, and the like. Any and all of the methods, functions, and operations described herein can be fully embodied in the form ofAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) tangible and / or non-transitory machine-readable code (e.g., interpretable scripts) saved in memory 1204.

[0140] Input-output devices 1206 can include any component for trafficking data such as ports, antennas (i.e., transceivers), printed conductive paths, and the like. Input-output devices 1206 can enable wired communication via USB®, DisplayPort®, HDMI®, Ethernet, and the like. Input-output devices 1206 can enable electronic, optical, magnetic, and holographic, communication with suitable memory 1206. Input-output devices 1206 can enable wireless communication via WiFi®, Bluetooth®, cellular (e.g., LTE®, CDMA®, GSM®, WiMax®, NFC®), GPS, and the like. Input-output devices 1206 can include wired and / or wireless communication pathways.

[0141] Sensors 1208 can capture physical measurements of environment and report the same to processors 1202. User interface 1210 can include displays, physical buttons, speakers, microphones, keyboards, and the like. Actuators 1212 can enable processors 1202 to control mechanical forces.

[0142] Processing system 1200 can be distributed. For example, some components of processing system 1200 can reside in a remote hosted network service (e.g., a cloud computing environment) while other components of processing system 1200 can reside in a local computing system. Processing system 1200 can have a modular design where certain modules include a plurality of the features / functions shown in FIG.12. For example, I / O modules can include volatile memory and one or more processors. As another example, individual processor modules can include read-only-memory and / or local caches.

[0143] While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the invention is also to be considered illustrative or exemplary and not restrictive as the invention is defined by the claims. It will be understood that changes and modifications can be made, by those of ordinary skill in the art, within the scope of the following claims, which can include any combination of features from different embodiments described above.

[0144] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B andAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and / or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) CLAIMS What is claimed is:

1. A computer-implemented machine learning method for modelling multi-scale interactions, comprising: determining a geometric vectorization based on a mapping of input data associated with a multi-scale system; generating, using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system, wherein the neural network comprises pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms; and mapping the one or more predicted vectors to the multi-scale interactions of the multi- scale system.

2. The computer-implemented method according to claim 1, wherein the one or more predicted vectors comprises a multi-vector, and wherein mapping the one or more predicted vectors to the multi-scale interactions comprises mapping the multi-vector to a scalar component that is invariant to rotation and translation, a vector component, and a bivector component of the molecular system.

3. The computer-implemented method according to claim 2, wherein the multi-scale system represents a molecular system, wherein the scalar component represents an atomic mass of an atom, wherein the vector component represents information in different channels of a position or a momentum of the atom, and wherein the bivector component represents an angle between the atom and another atom or area.

4. The computer-implemented method according to claim 2, wherein the multi-scale system represents a messenger ribonucleic acid (mRNA) system of atoms, wherein the atoms of the mRNA system of atoms is represented in a configuration as components of the multi-vector, wherein the scalar component represents an atomic energy of an atom of the mRNA system of atoms, and wherein the vector component represents a force or an acceleration associated with the atom.

5. The computer-implemented method according to claim 4, wherein the one or more predicted vectors comprise a representation of the atom, and wherein mapping the one or more predicted vectors to the multi-scale interactions of the multi-scale system further comprises integrating the representation of the atom into a plurality of atoms of the system of atoms.

6. The computer-implemented method according to claim 2, wherein the multi-scale system represents a low earth orbit satellite system, wherein the scalar component represents that massAttorney Docket No.820681 (Client Ref. NLE-1358-24-WO) of a satellite of the satellite system, and wherein the vector component represents the coordinate of the satellite.

7. The computer-implemented method according to claim 2, further comprising: prior to generating the one or more predicted vectors, creating the neural network to describe the molecular system, wherein a use of the multi-vector indicates flexibility to map various information either in a specific blade of the multi-blade projection or in an additional channel of the vector component.

8. The computer-implemented method according to any of the preceding claims, wherein the mapping of the input data is based on Clifford algebra, and wherein generating the one or more predicted vectors is based on the mapping of the input data.

9. The computer-implemented method according to any of the preceding claims, further comprising: collecting a training dataset comprising training input data and training output data; and training the neural network using the training dataset to generate one or more training predicted vectors, wherein the one or more training predicted vectors indicate a final state of a simulation or a step in a multi-step simulation.

10. The computer-implemented method according to claim 9, wherein training the neural network is based on using a supervised neural potential learning method or a generative process of a diffusion model.

11. The computer-implemented method according to any of the preceding claims, wherein the neural network further comprises a geometric algebra or Clifford based neural network, and wherein the geometric algebra or Clifford based neural network is extended with dense or sparse pooling of the pooling and unpooling mechanisms.

12. The computer-implemented method according to claim 11, wherein the neural network is a message passing neural network.

13. The computer-implemented method according to any of the preceding claims, wherein the equivariance of the geometric primitives comprises translation and rotation of the geometric primitives.

14. A computer system for modelling multi-scale interactions, the system comprising one or more hardware processors, which, alone or in combination, are configured to provide for execution of the following steps: determining a geometric vectorization based on a mapping of input data associated with a multi-scale system;Attorney Docket No.820681 (Client Ref. NLE-1358-24-WO) generating, using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system, wherein the neural network comprises pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms; and mapping the one or more predicted vectors to the multi-scale interactions of the multi- scale system.

15. A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, alone or in combination, provide for execution of a method for modelling multi-scale interactions comprising the following steps: determining a geometric vectorization based on a mapping of input data associated with a multi-scale system; generating, using a neural network and the mapped input data, one or more predicted vectors of the multi-scale system, wherein the neural network comprises pooling and unpooling mechanisms that conserve equivariance to geometric primitives of the geometric vectorization by using a multi-blade projection that defines pooling or clustering of the pooling and unpooling mechanisms; and mapping the one or more predicted vectors to the multi-scale interactions of the multi- scale system.

Images