Information processing method and information processing device

Abstract

In this information processing method using a neural network for a structure represented as a set of nodes arranged in space, a computer executes processing including: receiving input of a state of each node; calculating, on the basis of states between the nodes, a frame representing a coordinate axis for each node; and extracting, using the frame, information having predetermined symmetry of the structure from each node.

Description

Information processing method and information processing apparatus

[0001] This disclosure relates to an information processing method and an information processing apparatus.

[0002] Conventionally, techniques for processing information related to molecular or crystal structures using neural networks are known (for example, Reference 1 (Yi-Lun Liao, Tess Smidt, "Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs", 36th Conference on Neural Information Processing Systems (NeurIPS 2022), [Retrieved September 9, 2024], Internet <URL: https: / / openreview.net / forum?id=_efamP7PSjg>), Reference 2 (Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, Yaron Lipman, "Frame Averaging for Invariant and Equivariant Network Design", Conference paper at ICLR 2022), [Retrieved September 9, 2020], Internet <URL: https: / / openreview.net / forum?id=zIUyj55nXR>), Reference 3 (Alexandre Duval, Victor Schmidt, Alex Hernandez Garcia, Santiago Miret, Fragkiskos D. Malliaros, Yoshua Bengio, David Rolnick, "FAENet: frame averaging equivariant GNN for materials modeling", Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA. PMLR 202, 2023, [Retrieved September 9, 2020], Internet <URL: https: / / dl.acm.org / doi / 10.5555 / 3618408.(see also Document 3 (Publication No. 3618769) and Document 4 (Keqiang Yan, Cong Fu, Xiaofeng Qian, Xiaoning Qian, Shuiwang Ji, "Complete and efficient graph transformers for crystal material property prediction", Published as a conference paper at ICLR 2024, [searched on September 9, 2024], Internet <URL: https: / / openreview.net / pdf / d25b06fbca76b8ea41b195b65471fab55cc08945.pdf>)).

[0003] Documents 1 to 4 disclose processing information related to physical properties or forces of a molecular structure or a crystal structure using a machine learning model such as a neural network model. When processing information related to physical properties or forces of a molecular structure or a crystal structure using a machine learning model, in the machine learning model, not only distance information between atoms constituting the molecular structure or the crystal structure but also position information such as a relative position vector representing the relative positions between atoms and direction information can be incorporated to improve its performance.

[0004] Note that even if a rotation operation or a translation operation is performed on a molecular structure or a crystal structure, the physical properties and the like of the molecular structure or the crystal structure remain unchanged, so it has invariance. Also, when a rotation operation or a translation operation is performed on a molecular structure or a crystal structure, the forces and the like acting on the molecular structure or the crystal structure change in the same way as the rotation operation or the translation operation, so it has covariance.

[0005] However, in the computational processing within a machine learning model, when a rotation operation or a translation operation is performed on a molecular structure or a crystal structure, there is no unique method for incorporating into the machine learning model the position information of atoms or the direction information representing the positional relationship between atoms while ensuring the above-mentioned invariance or covariance.

[0006] In this regard, one of the existing approaches disclosed in the above-mentioned references 2-4 is the concept of a "frame." By handling positional information of atoms or directional information representing the positional relationships between atoms on a frame, which is a coordinate system determined by certain rules for molecular or crystal structures, invariance or similar variance is considered in calculations performed within the machine learning model.

[0007] In the techniques disclosed in the above-mentioned references 2-4, various calculations within the neural network model are performed using a predetermined fixed frame for each molecular or crystal structure. However, it is believed that some kind of interaction exists between the multiple atoms constituting the molecular or crystal structure. Therefore, there is a problem that simply using a fixed frame cannot take into account any interactions that exist between multiple atoms. Furthermore, there is no known technique for setting frames for nodes that take into account the interactions that exist between the nodes constituting a structure, not limited to molecular or crystal structures.

[0008] This disclosure aims to perform calculations using a neural network that take into account the interactions between nodes when performing calculations on multiple nodes that constitute a predetermined structure.

[0009] To achieve the above objective, the information processing method relating to this disclosure is an information processing method using a neural network for a structure represented as a set of nodes arranged in space, wherein a computer performs the following steps: receiving input of the state of each node; calculating a frame representing the coordinate axes at each node based on the states between each node; and extracting feature quantities having a predetermined symmetry of the structure from the nodes using the frame.

[0010] Furthermore, the information processing device of the present disclosure is an information processing device that uses a neural network for a structure represented as a set of nodes arranged in space, and comprises: a receiving unit that receives input of the state of each node; a calculation unit that calculates a frame representing the coordinate axes at each node based on the states between each node; and an extraction unit that extracts information having a predetermined symmetry of the structure from the nodes using the frame.

[0011] According to the information processing method and information processing apparatus of this disclosure, when performing calculations concerning multiple nodes constituting a predetermined structure, it is possible to perform calculations that take into account the interactions that exist between the nodes.

[0012] This is a diagram illustrating embodiments of the present disclosure. This is a diagram illustrating invariants and equivariants in molecular or crystal structures. This is a diagram illustrating a frame. This is a diagram illustrating a neural network model that processes information about molecular or crystal structures. This is a diagram illustrating the differences between the prior art and embodiments of the present disclosure. This is a diagram illustrating embodiments of the present disclosure. This is a diagram illustrating a transformer. This is a block diagram showing the hardware configuration of a trained model generation device according to this embodiment. This is a block diagram showing the schematic configuration of the trained model generation system of this embodiment. This is a block diagram showing the hardware configuration of an information processing device according to this embodiment. This is a block diagram showing the schematic configuration of the information processing system of this embodiment. This is a diagram illustrating a trained model generation processing routine executed by the trained model generation device of this embodiment. This is a diagram illustrating an information processing routine executed by the information processing device of this embodiment. This is a diagram illustrating an information processing routine executed by the information processing device of this embodiment.

[0013] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In this embodiment, an information processing device according to the present disclosure will be used as an example. In each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensions and proportions in the drawings are exaggerated for illustrative purposes and may differ from the actual proportions.

[0014] <Outline of this Embodiment> Figure 1 is a diagram illustrating an embodiment of the present disclosure. As shown in Figure 1, we consider a case where structural information representing a molecular structure or crystal structure is input to a neural network M, which is an example of a machine learning model, and the output from the neural network M is used. In this case, the output from the neural network M is used, for example, in the following tasks. In this embodiment, the case in which the target structure is a molecular structure or a crystal structure is described as an example, but it is not limited to this. In this embodiment, the physical properties of a structure composed of atoms are inferred using information that has symmetry in the structure.

[0015] (1. Prediction of physical properties independent of coordinate system) For example, the formation energy, total energy, or band gap of a molecular or crystal structure are physical properties that are independent of the coordinate system of the molecular or crystal structure. Therefore, when predicting invariants that are invariant with respect to rotation or translation of a molecular or crystal structure (hereinafter also simply referred to as invariants), the output from a neural network M can be used. For example, by training a neural network M so that its output is a physical property, it becomes possible to perform a task that predicts that physical property.

[0016] Furthermore, classifications of whether a molecular or crystal structure is harmful to the human body, whether a crystal structure is superconducting, or the space group of a crystal structure are classifications based on physical properties that do not depend on the coordinate system of the molecular or crystal structure (referred to as "invariant classifications" in the table described later). For this reason, when predicting these categories for the entire molecular or crystal structure or for each atom, the output from a neural network M can be used. For example, by training the neural network M so that its output is a space group category, it becomes possible to perform a task that predicts the category of that space group.

[0017] (2. Prediction of physical properties dependent on coordinate system) For example, the force acting on each atom constituting a molecular structure or crystal structure is an equivariate quantity with respect to rotation or translation (hereinafter also simply referred to as an equivariate). Therefore, when predicting equivariates with respect to rotation or translation for a molecular structure or crystal structure, the output from a neural network M can be used. For example, by training the neural network M so that its output is the force acting on each atom, it becomes possible to perform a task that predicts the force acting on each atom.

[0018] (3. Extraction of information to be input to other neural networks) For example, if the neural network M shown in Figure 1 is a neural network for information extraction, the output from the neural network M can be used as input to other neural networks as a state vector (or feature vector) representing the overall structure of the molecular or crystal structure or the state of each atom. Alternatively, the output from the neural network M can be used as input to other neural networks as a state vector (or feature vector) representing geometric information about each atom constituting the entire molecular or crystal structure (e.g., forces acting on the atoms).

[0019] Tables 1 and 2 below show examples of invariants, invariant classifications, and equivariants in molecular or crystalline structures.

[0020]

[0021]

[0022] The "Physical Properties (or Physical Characteristics)" column in Tables 1 and 2 above represents the name of the specific physical property. The "Invariant or Equivalent" column indicates whether the physical property belongs to the category of invariant, invariant, or equivariant. The "Molecular Structure" column indicates whether the physical property is an invariant, invariant, or equivariant when the target structure is a molecular structure. The "Crystal Structure" column indicates whether the physical property is an invariant, invariant, or equivariant when the target structure is a crystal structure. The "Set of Multiple Molecular Structures or Crystal Structures" column represents a set of molecular or crystal structures, and indicates whether the physical property is an invariant, invariant, or equivariant when the target structure is a "set of multiple molecular structures," a "set of at least one molecular structure and at least one crystal structure," or a "set of multiple crystal structures." Note that the classifications of invariants, invariant classifications, and equivariants in Tables 1 and 2 above are based on general interpretations, and other classifications may also exist.

[0023] Figure 2 is a diagram illustrating invariants and equivariants in a molecular or crystal structure. For example, as shown in Figure 2, a rotational operation R and a translational operation t are performed on a certain molecular or crystal structure, and the position vector p of each atom is given. ｉ is the position vector p' ｉ The change to is represented by the following formula. Note that i is an index used to identify the atoms that make up the molecular or crystalline structure.

[0024]

[0025] As shown in Figure 2, when a molecular or crystal structure rotates or translates, the position vectors of each atom change, but the essential properties of the material represented by the molecular or crystal structure remain unchanged. As mentioned above, for example, physical properties such as the energy of a molecular or crystal structure have invariance, meaning they do not change even when the molecular or crystal structure rotates. On the other hand, forces acting on a molecular or crystal structure have the same invariance, meaning they change with the rotation of the molecular or crystal structure.

[0026] As described above, in the computational processing within a machine learning model, when a rotation operation or a translation operation is performed on a molecular structure or a crystal structure, it is difficult to guarantee the invariance or equivariance as described above. Regarding this point, as described above, a technique of setting a "frame" for a molecular structure or a crystal structure is known.

[0027] FIG. 3 is a diagram for explaining a frame. As shown in FIG. 3, when a frame E representing a coordinate system is set for a target molecular structure or crystal structure S, when the molecular structure or crystal structure S rotates, the frame E also rotates, so that the position coordinates of the atoms constituting the molecular structure or crystal structure S become constant. Therefore, by setting a frame for the molecular structure or crystal structure S, the orientation of the molecular structure or crystal structure S is normalized. Also, when the molecular structure or crystal structure S rotates, the coordinate axes are reset for the molecular structure or crystal structure S according to a certain rule. By using a frame, it becomes possible to estimate invariant quantities and equivariant quantities.

[0028] For example, the frame transformation due to rotation is E Ｔ (or, E －１ ), consider the case where the function realized inside the neural network is represented by f and the set of position vectors representing the position coordinates of the atoms is represented by P.

[0029] In this case, consider passing the product of the frame transformation E Ｔ due to rotation and the set P of position vectors of the atoms, i.e., E Ｔ P, to the function f inside the neural network. In this case, f(E Ｔ P) is output from the function f inside the neural network. The product of the frame transformation E Ｔ due to rotation and the position vector P, i.e., E Ｔ P, becomes useful information when estimating invariant quantities because the frame transformation E Ｔ has been performed. For example, inside the function f, the coordinates E Ｔ after frame transformationThe relative positions between atoms and relative positions from the center of mass, based on P, are invariants under rotation and translation, and therefore provide useful information. Furthermore, when estimating these variables, the inverse transform E of the frame is obtained from the output f(E) of the internal function f of the neural network. Ｔ By multiplying by P), Ef(E Ｔ P) is calculated. This Ef(E Ｔ Since P) reflects the information in the original coordinate system, it provides useful information when estimating the same variable (e.g., force).

[0030] Figure 4 illustrates a neural network model for processing information about molecular or crystal structures. It is assumed that a sequence of state vectors x representing the states of N atoms within a molecular or crystal structure, and a sequence of position vectors p representing the three-dimensional positions of those N atoms, are expressed by the following equations. Note that in a crystal structure, although the number of atoms N is infinite, the representation is summarized using a unit cell, which consists of a finite sequence of state vectors x representing the states of atoms within a unit cell, a sequence of position vectors p representing the three-dimensional positional information of those atoms, and a lattice vector representing the unit repeating direction within the three-dimensional space of the unit cell. The state and position of each atom in the entire crystal structure are then represented by the periodic repeating of the unit cell.

[0031]

[0032] In this case, the state vector representing the state of an atom is updated in each layer of the neural network. Specifically, as shown in Figure 4, for example, if the molecular structure or crystal structure is composed of sodium (Na) and chlorine (Cl), the state vectors of each of these multiple sodium (Na) atoms and each of the multiple chlorine (Cl) atoms are repeatedly updated by each layer of the neural network (labeled "Interatomic message passing" in Figure 4). Note that "AE" shown in Figure 4 is a neural network known as AtomEmbedding.

[0033] In the neural network shown in Figure 4, when the state vectors of multiple sodium (Na) and multiple chlorine (Cl) are input to the neural network, the input state vectors are transformed by the AE (Automated Encoding) system.

[0034] Specifically, the state vector x of each of the multiple sodium Na ０ １ , x ０ ２ , x ０ ３ , x ０ ４ and the state vector x of each of the multiple chlorine atoms Cl ０ ５ , x ０ ６ , x ０ ７ , x ０ ８ And are output from AE. And these state vector x ０ １ , x ０ ２ , x ０ ３ , x ０ ４ , x ０ ５ , x ０ ６ , x ０ ７ , x ０ ８ However, this is input to the next layer, "Interatomic message passing," and the state vector x ０ １ , x ０ ２ , x ０ ３ , x ０ ４ , x ０ ５ , x ０ ６ , x ０ ７ , x ０ ８ is the state vector x １ １ , x １ ２ , x１ ３ , x １ ４ , x １ ５ , x １ ６ , x １ ７ , x １ ８ is updated to. These updates are repeated L times, and finally, the state vector x Ｌ １ , x Ｌ ２ , x Ｌ ３ , x Ｌ ４ , x Ｌ ５ , x Ｌ ６ , x Ｌ ７ , x Ｌ ８ is updated to.

[0035] Thus, the state vector x ｉ initially is a vector symbolically representing the atomic species of each atom (e.g., sodium or chlorine, etc.), but by repeating each layer of the state update inside the neural network, it is gradually transformed to represent the state of the atoms within the given molecular structure or crystal structure. The update of the state vector x as described above is represented by the following formula (A).

[0036] (A)

[0037] In the above formula, X is a matrix having the state vectors x of each of the N atoms as column vectors. Also, in the above formula, P is a matrix having the position vectors p of each of the N atoms as column vectors. Also, in the above formula, x ｉ ' is the state vector of atom i after update. In the above formula, f ｉ is a function realized inside the neural network and is a function that outputs the state vector x ｉ ' of atom i. Also, in the above formula, f ｉ←ｊThis is a function implemented within a neural network, and the state vector x of the atom i of interest. ｉ and the state vector x of other atoms j ｊ This is a function that outputs values ​​related to the interaction between [the two elements]. It also includes weights w. ｉｊ This represents the weight of the interaction between the atom of interest i and other atoms j. Within the neural network, the state vectors of multiple atoms are repeatedly updated according to equation (A) above.

[0038] Note that the function f ｉ Inside, the absolute value of the relative position vector |p| represents the distance between atom i and atom j. ｊ -p ｉ The information indicated by | is used. The absolute value of the relative position vector is |p ｊ -p ｉ The information represented by | is invariant under rotation and translation of molecular or crystal structures. Therefore, the function f ｉ The absolute value of the relative position vector |p| inside ｊ -p ｉ By using the | information, it becomes possible to describe interatomic interactions in a translationally and rotationally invariant manner.

[0039] The function f in equation (A) above ｉ←ｊ In contrast, when a coordinate transformation using frames is introduced, the above equation (A) can be expressed by the following equation (B).

[0040] (B)

[0041] E in formula (B) above Ｔ This is a matrix representing the coordinate transformation by the frame. The relative position vector E between two atoms i and j on the coordinate system represented by the frame. Ｔ p ｊ -E Ｔ p ｉ Since it is rotationally and translationally invariant, the function f inside the neural network model ｉ This can effectively capture more three-dimensional positional information of atoms constituting the molecular or crystalline structure. Note that E ＴP is information obtained by projecting the position vector of the atom onto frame E, therefore E Ｔ P is also referred to as projected information.

[0042] Figure 5 is a diagram illustrating the differences between the prior art and embodiments of the present disclosure. As shown in T1 of Figure 5, conventionally, when processing information about molecular structure or crystal structure, a single fixed frame E ｆｉｘ A predefined frame E was set for the molecular structure or crystal structure. Therefore, for atoms A1, A2, A3, and A4 that constitute the molecular structure or crystal structure, ｆｉｘ It was set to [this].

[0043] However, as mentioned above, it is believed that some kind of interaction exists between the multiple atoms that make up the molecular or crystalline structure. Therefore, a problem with simply using a fixed frame is that it is not possible to consider any interactions that exist between multiple atoms. Also, depending on the task being addressed, using a fixed frame may not be appropriate in some cases.

[0044] Therefore, in the embodiments of this disclosure, as shown in T2 of Figure 5, an individual frame E of atom i suitable for the atomic state or interatomic interactions resulting from the molecular structure or crystal structure. ｉ The frame is set. Specifically, since the interactions between atoms differ depending on the atoms constituting the molecular or crystal structure, the frame is set adaptively according to the atoms constituting the molecular or crystal structure. For example, in the embodiment of this disclosure, as shown in T2 of Figure 5, frame E1 is set for atom A1, frame E2 is set for atom A2, frame E3 is set for atom A3, and frame E4 is set for atom A4. For example, frame E1 set for atom A1 is calculated according to the weights representing the interaction strength of the state between the atom of interest A1 and the other atoms A2, A3, and A4. This makes it possible to perform calculations that take into account the interactions that exist between atoms when performing calculations on multiple atoms constituting the molecular or crystal structure.

[0045] Figure 6 is a diagram illustrating an embodiment of the present disclosure. Consider the case where the interaction function for updating the state vector of each atom is represented by equation (B) above. In the embodiment of the present disclosure, weight w represents the interaction strength regarding the state between the atom of interest i and several other atoms j1, j2, j3, j4, j5, j6 surrounding atom i. ｉｊ The weights are calculated within the neural network. ｉｊ Frame E reflects this. ｉ Set this for atom i.

[0046] Specifically, as shown in Figure 6, the interaction with the atom i of interest is weak, and the weight w ｉｊ We disregard the surrounding atoms with small values ​​(for example, atoms j1 and j2 in Figure 6) and set a frame for the atom i of interest. On the other hand, as shown in Figure 6, atoms that have a strong interaction with the atom i of interest and have weight w ｉｊ Surrounding atoms with a large weight (e.g., j3, j4 in Figure 6) are given importance, and a frame is set for the atom i of interest. Although surrounding atoms j1 and j2 in Figure 6 are located near the atom i of interest, their weight w between them and the atom i of interest is also considered. ｉｊ Because it is small, it is disregarded when calculating the frame of the atom i of interest.

[0047] In this embodiment, the weight w between the atom i of interest and multiple other atoms j is used. ｉｊ As a way to achieve this, we repurpose the attention weights of a known neural network called Transformer. This allows us to obtain the weights w between the atom of interest i and several other atoms j. ｉｊ This is learned automatically within the Transformer.

[0048] Figure 7 is a diagram illustrating the transformer. As shown in Figure 7, in this embodiment, the expected value of the state vector x is calculated in the multi-head attention layer MA located inside the known transformer TF. The transformer TF is composed of an encoder TF1 and a decoder TF2. The Nx written near LA in the encoder TF1 indicates that the layer LA constituting the encoder TF1 is repeated N times. The state vector x is gradually updated by this repetition of layer LA. Note that each element constituting the transformer TF shown in Figure 7 (e.g., Add&Normal, etc.) is already known and therefore its explanation is omitted.

[0049] Specifically, the following equation is executed in the multi-head attention layer MA within the transformer TF.

[0050] (C) (D) (E)

[0051] q in each of the above formulas ｉ This represents a query used in Transformer, and k ｉ This represents a key used in Transformers, and v ｉ This represents the value used in the Transformer. Note that i is an index for identifying atoms. Note that W Ｑ , W Ｋ , W Ｖ Each of these is a weight matrix used in Transformers. Ｋ represents the dimensions of the query and key, d Ｖ This represents the dimension of value.

[0052] Weight w in equation (D) above ｉｊ is a weight representing the interaction strength between the atom i of interest and several other atoms j. Also, a in equation (D) above ｉｊk and j are coefficients used in the transformer model. k and j are indices used to identify atoms.

[0053] x' in the above equation (E) ｉ b represents the state vector of the atom i of interest. Also, b in the above equation (E) ｉｊ This is a coefficient used in the transformer model. In this embodiment, b in equation (E) above is calculated according to the following equation (F). ｉｊ Calculate.

[0054] (F)

[0055] As shown in formula (F) above, in this embodiment, the relative position vector p between the atom i of interest and another atom j is ｊ -p ｉ And the frame E calculated for the atom i of interest ｉ (For example, a three-axis vector e ｉ，１ , e ｉ，２ , e ｉ，３ Projection information representing the value calculated by the product of ) is input to the function f inside the transformer model, b ｉｊ It is calculated as follows.

[0056] b calculated in this embodiment ｉｊ This is the frame E calculated for the atom i of interest. ｉ The value corresponds to this. Therefore, this b ｉｊ By incorporating this into the internal calculations of the transformer, it becomes possible to perform calculations that take into account the interactions between atoms that make up the molecular or crystal structure.

[0057] <Details of this Embodiment> The outline of this embodiment is as described above. The details of this embodiment will be described below. In addition, matters that have already been described in the outline above will be explained again below.

[0058] [1.1 System that takes molecular or crystal structure as input] (1. Overall system configuration) The system of this embodiment receives structural data representing at least one molecular or crystal structure. The system then inputs the input information, including information derived from the molecular or crystal structure, into a neural network to calculate and output information such as the state of the molecular or crystal structure, physical properties, or the structure updated to a stable form. In addition to the state of a single structural data, the system may also receive structural data of multiple molecular or crystal structures and calculate and output the states or relationships between those substances.

[0059] Molecular structure data includes at least information about the position and type of each atom in the structure. Furthermore, molecular structure data may also include information such as the empirical formula or the bonding state between each atomic pair. Molecular structure data typically represents the atomic arrangement in three-dimensional space, and the coordinates are given in Cartesian coordinates or oblique coordinate systems, etc.

[0060] Crystal structure data represents the periodically repeating arrangement of atoms in space and is usually expressed as lattice vectors representing the shape of a unit cell and information about the position and type of atoms within the unit cell. Alternatively, a crystal structure representation that expresses information equivalent to the lattice vectors and the position and type of atoms within the unit cell may be used as crystal structure data. The unit cell structure usually represents the arrangement of atoms in three-dimensional space, with atomic positions given as three-dimensional vectors representing Cartesian coordinates or unit cell coordinates, and lattice vectors given as three three-dimensional vectors representing unit translation quantities. However, when dealing with two-dimensional materials, the structure may be represented by the arrangement of atoms in two-dimensional space, atomic positions as two-dimensional vectors, and lattice vectors as two two-dimensional vectors representing unit translation quantities. Furthermore, information such as the empirical formula or space group may be included in addition to the structural information of the unit cell. In the following, data representing at least one of molecular structure data and crystal structure data will also be referred to as structural information.

[0061] The input information to the neural network may include information other than that derived from the input structural information, depending on the task. For example, it may include the descriptive text attached to the structural information, the name of the substance, and environmental information such as the temperature, orientation, or pressure at which the substance is placed.

[0062] The output information from the neural network can include, depending on the task, physical properties such as band gap or formation energy, the direction or magnitude of forces acting on atoms, the structure updated to a stable form, the chemical reaction state between multiple structures, or the relationship between the input description and the structure. Furthermore, one or more feature vectors abstractly representing the "input information," including structural information, may be output as input to other machine learning models such as external neural networks, SVMs, or t-SNEs. Here, the feature vectors may also include multidimensional tensor quantities.

[0063] Neural network training is performed by calculating the gradient dL / dY between the neural network's output (e.g., represented by a tensor Y) and the loss function L defined according to the task, and consequently the gradient dL / dΘ between the neural network's parameters Θ and the loss function L, using backpropagation. Then, the parameters Θ are optimized using optimization techniques such as stochastic gradient descent to minimize the loss function L for the training data. If, during the backpropagation process, a module within the neural network contains discrete operations and its gradient diverges or vanishes, an approximate gradient obtained by approximating that module with a continuous function may be partially used.

[0064] The gradient of the output with respect to the loss function, dL / dY, can be given in various ways. A typical example is the "supervised learning" method, where the predicted output Y of the neural network is given by the ground truth Y. ＊ Prepare a loss function L(Y, Y) defined between the predicted value and the correct value, such as the mean squared error or cross-entropy error. ＊ It is calculated as the gradient dL / dY of ). Note that a method called the diffusion model also adds noise N to the ground truth value. ｔ Diffusion process Y, which involves adding steps ｔ = Y＊ +N １ +...N ｔ In contrast, the noisy state Y ｔ From a state of less Y ｔ－１ This can be considered a supervised learning method for predicting the output Y. In the "self-supervised learning" method (sometimes considered an "unsupervised learning" method), the loss function evaluates how far the predicted output Y is from the most plausible state by using laws (such as physical laws) or properties that can be assumed for the predicted output Y. For example, when Y represents an updated structure, the instability of the structure Y can be calculated based on physical laws and given as the loss function L(Y). Furthermore, there is a quantity O(Y) that can be calculated from Y based on physical laws, etc. (e.g., X-ray diffraction pattern), and the correct value of O obtained from simulations or observations. ＊ Loss function L(O(Y), O(Y), O(Y), such as mean squared error or cross-entropy error, defined between the two. ＊It can also be obtained as the gradient dL / dY of ). Furthermore, the relationships between multiple input structures or additional information can be used as a training signal. For example, when multiple pairs of structural information and their X-ray diffraction patterns are given as input information, a neural network can be trained that maps structural information and X-ray diffraction pattern data to the same feature space by outputting the feature vectors of both the structural information and the X-ray diffraction pattern, and defining a loss that brings the distance between the feature vectors of corresponding structures and X-ray diffraction patterns closer, while increasing the distance between the feature vectors of other structures and X-ray diffraction patterns. In addition to X-ray diffraction patterns, text information such as explanatory texts associated with the structure can also be used as information linked to the structural information. Similarly, by using category information assigned to the input structure (e.g., metallic or nonmetallic) to output feature vectors for multiple structures, a loss function can be defined that reduces the distance between feature vectors of structures belonging to the same category while increasing the distance between feature vectors of structures belonging to different categories. This allows for the learning of abstract feature vectors and feature spaces based on the categories of structures, which can then be used for searching for similar structures based on the distance between feature vectors or for visualizing the feature space using t-SNE, etc. In the "reinforcement learning" method, the input information is considered as the state s in reinforcement learning, the neural network as the policy function π, and the output Y as the deterministic action prediction a = π(s) or the stochastic action distribution π(a|s). The reward function r defined for these can be included in the loss function as L = -r, etc., with its sign inverted. By using reinforcement learning, even if the reward function r is not differentiable with respect to a or π(a|s), a gradient that increases the reward function r (decreases the loss function) can be calculated and used as the gradient dL / dY. Furthermore, the gradient dL / dY may be calculated by a combination of multiple methods such as supervised learning, self-supervised learning, unsupervised learning, or reinforcement learning.

[0065] (1.2 State Variable Update Operation) Within the neural network described above, we consider an operation to update state variables related to the input structure using input information that includes information derived from the input structure. In the operation under consideration, the input structure is represented by a set of N nodes, each node being a unit formed by grouping at least one atom in the structure. The information that each node possesses is provided by at least a sequence of state vectors representing the state of each node and a sequence of position vectors representing the spatial coordinates of each node. The sequence of state vectors and the sequence of position vectors are expressed by the following equations.

[0066]

[0067] A neural network is a learnable function f that, at any point within its structure, uses at least some or all of the matrix X representing the state of the node at that time and the matrix P representing the position of the node. ｉ←ｊ This results in some or all of the state vectors x in matrix X. ｉ Update it as follows:

[0068]

[0069] Here, the function f ｉ←ｊ This calculates the effect that node j has on the state of node i as an abstract state vector, and w ｉｊ w is a scalar variable that represents its weight. ｉｊ Furthermore, the values ​​are to be dynamically calculated in response to the input information using matrices X and P, or variables calculated elsewhere in the neural network. Within the neural network, one or more such interaction operations between nodes are included, and by combining a linear transformation Wx + b (W represents a matrix and b represents a vector) for each state vector x, a known batch normalization or known layer normalization, a known nonlinear function such as a Rectified Linear Unit (ReLU) function or a sigmoid function, and an integration process such as mean pooling or max pooling, the variables related to the input information are repeatedly updated and transformed into a predicted value Y according to the task.

[0070] (1.3 Node Configuration) Nodes may directly represent individual atoms within a molecular or crystal structure. In particular, the initial state vector x of a node given as input to the network ｉ As such, a learnable atomic embedding vector corresponding to the atomic number of atom i is used. The atomic embedding vector can be realized using known techniques and is expressed by the following equation (3). Below, ATOMICNUMBER(i) represents the atomic number of atom i, and AtomEmbedding represents the calculation of the vector using atomic embedding. AtomEmbedding is realized using known techniques.

[0071]

[0072] Alternatively, the state vector may be a vector formed by concatenating embedding vectors corresponding to the group or period of atom i in the periodic table. In this case, for example, the state vector x is represented by the following equation (4). Below, GROUP(i) represents the group of atom i, and GroupEmbedding represents the calculation of the vector by embedding the group of atom i. Also, below, PERIOD(i) represents the period of atom i, and PeriodEmbedding represents the calculation of the vector by embedding the period of atom i. GroupEmbedding and PeriodEmbedding are implemented using known techniques.

[0073]

[0074] Alternatively, the atomic species corresponding to atom i may be represented by a mixture of multiple atomic species based on site occupancy, and the mixing ratio of each atomic species n is o ｎ When given by , the state vector may be a linear sum of the atomic embedding vectors. In this case, for example, the state vector x is expressed by the following equation (5).

[0075]

[0076] Furthermore, a node may represent multiple atoms in a structure. In this case, the same atom may be included in multiple nodes, and the number of atoms included in a node does not have to be constant. Within a neural network, the node configuration (e.g., the number of nodes, the atom assignment to each node, or the position coordinates of each node) does not need to be constant, and the node configuration may change through node merging or splitting along the way. For node merging, in addition to merging processes such as average pooling and maximum pooling, known techniques such as merging or splitting based on attention mechanisms can also be used.

[0077] For example, conventional technologies use a node configuration where nodes correspond one-to-one with atoms at the time of input to the neural network. However, they also use a network configuration that converts multiple atom nodes into "neural atom" nodes along the way, and then returns to a node configuration with individual atoms.

[0078] (1.4 State update by interaction operation using attention mechanism) The interaction operation represented by equation (2) above can be realized by the attention mechanism in known transformer-type neural networks. In the attention mechanism in transformers, the state vector sequence X is transformed into a matrix called the query Q, key K, and value V. Note that the following φ Ｑ ､φ Ｋ ､φ Ｖ This is a function used in Transformers.

[0079]

[0080] The matrix Q above is the following vector q ｉ It has as a column vector vector q. ｉ This is a query vector used in Transformers.

[0081]

[0082] Furthermore, the above matrix K is the following vector k ｉ It has as a column vector vector k. ｉThis is a key vector used in Transformers.

[0083]

[0084] Furthermore, the above matrix V is the following vector v ｉ It has as a column vector. Vector v ｉ This is a value vector used in Transformers.

[0085]

[0086] Note that N is the total number of nodes, and d Ｋ represents the number of rows in matrices Q and K, and d Ｖ represents the number of rows in matrix V. Here, each transformation φ is usually a linear transformation φ Ｑ (X) = W Ｑ X or φ Ｑ (X) = W Ｑ X + b Ｑ These are given to matrices Q, K, and V respectively, but other forms may also be used. Note that W Ｑ b is the query weight matrix used in the transformer. Ｑ is an arbitrary vector.

[0087] Then, using matrices Q, K, and V, we give the following interaction function between nodes. The following Attention i This is a function that represents the attention mechanism of the Transformers.

[0088]

[0089] Here, the state vector x of each node i ｉ The weight vector used to update w ｉ = [w ｉ１ ,w ｉ２ , ..., lol ｉＮ ] is expressed by the following equation (11).

[0090]

[0091] In equation (11) above, u(x) is a function that takes an arbitrary real-valued vector as input. Typically, softmax(x), a known function that normalizes the vector output so that each element is non-negative and the sum is 1, is used. Alternatively, sigmoid(x), a known function that normalizes each element of the vector output to a value between 0 and 1, or the following function u(x), a known function that normalizes the sum to 1, may be used.

[0092]

[0093] Alternatively, one may use an identity function u(x) = x that does not undergo any transformation. In equation (11) above, c is a scalar coefficient, and the following values ​​are usually used.

[0094]

[0095] Note that q in formula (11) above Ｔ ｉ The part K can be directly x without going through matrix Q or the transformation φ(Q) or φ(V) to matrix K. Ｔ ｉ It is sometimes represented as WK, etc.

[0096] a in formula (11) above ｉ = [a ｉ１ , a ｉ２ , ..., a ｉＮ ] Ｔ Each a ｉｊ or b in formula (10) above ｉｊ This primarily serves as a known relative position encoding, abstractly representing information such as the relative position between two nodes i and j using scalar and vector values, respectively. These may be calculated from structural or state variables such as matrix P or matrix X, and may include learnable parameters. Furthermore, information other than matrix P or matrix X, such as interatomic bond states related to the input structure, may also be used. In addition, a ｉｊ It is also possible to manually set the value for a, for example, when the softmax function or the sigmod function is used as the function u, ｉｊBy assigning a large negative value to a specific j, the inflow of information from j to i can be blocked. This type of masking is useful in known autoregressive networks that use neural networks to output atoms one by one and predict the structure.

[0097] Furthermore, the operation represented by equation (10) above is an expression that encompasses both operations generally called self-attention and operations called cross-attention. For example, when the node configuration represents a single molecular or crystal structure, equation (10) above can be considered as a self-attention operation representing the interaction between nodes within a single structure. On the other hand, when the node configuration represents multiple structures, a ｉｊ By manipulating the value of , the interaction weights when node i and node j belong to the same structure are changed to w ｉｊ By forcing it to equal 0, the above equation (10) can be considered as a mutual attention operation representing the action from one structure to another.

[0098] [2 Adaptive Frame] (2.1 Frame Overview) For each node coordinate of the input structure, an arbitrary rotation matrix R(R Ｔ Consider the case where the operation expressed in the following equation (12) is performed by a rigid body transformation using an arbitrary square matrix (R = I) and a translation vector t.

[0099]

[0100] Alternatively, if the following matrix representation (12-1) is used, the rigid body transformation is expressed by the following equation (13).

[0101] (12-1)

[0102]

[0103] Even when a transformation such as the above formula (12) or formula (13) is performed, the state update operation by formula (1) is performed regardless of whether a coordinate transformation is performed or not, f ｉ (X, P') = f ｉ A function f that is invariant with respect to (X, P). ｉSuppose we want to find [something]. A frame is a geometric coordinate transformation technique that is useful in such situations.

[0104] Frames are inherent in the matrix P representing the structure, and axes [e1, e2, ..., e] are extracted according to certain rules. Ｋ ] (Matrix E ∈ R ｄＰ×Ｋ It refers to the function f (represented by...). ｉ In this case, the matrix P representing the input coordinates is used directly as the state vector x ｉ Instead of using it to update, the projection information E onto space by axis E is expressed as equation (14) below. † By using P, a transformation that is invariant to rotation can be obtained. Here, E † This is usually the transpose of E, and when E is a square matrix, it is the inverse matrix E －１ Alternatively, you may use this method. Furthermore, if the quantity is inherently invariant under rotation and translation, such as distance, it can be calculated directly from matrix P and used.

[0105] For example, when the input coordinates rotate P' = RP, the axis also rotates E' = RE, and E' † P' = (RE) † RP = E † R Ｔ RP = E † P becomes E † It can be seen that P is rotationally invariant coordinate information.

[0106] Furthermore, if E is invariant with respect to the translation t of P, then f ｉ － Within this, by using information based on relative position (including the distance and angle between two points), f ｉ － This can be made rotationally and translationally invariant. Because,

[0107]

[0108] At that time,

[0109]

[0110] And so,

[0111]

[0112] This is because the components cancel each other out within their relative positions.

[0113] To determine E in a translationally invariant manner, it can be determined based on the coordinates obtained by removing the mean from matrix P, or on the relative positions of two points in matrix P.

[0114] (2.2 Conventional frame method) Conventional frame methods use a fixed (static) global frame E that is independent of the node state X and independent of node i for each input structure's position matrix P.

[0115] (2.2.1 Frame Averaging) When the way of selecting frame E is not uniquely determined, and there is a finite set E = {E} as a candidate for frame, １ , E ２ , ..., E ｜Ｅ｜ When obtained as}, invariance can be guaranteed by calculating for all frame candidates and averaging the results, as shown in equation (15) below. This is called frame averaging. Frame averaging is a known technique (see, for example, reference 3).

[0116]

[0117] Furthermore, to reduce the computational load during training, stochastic frame averaging, which randomly selects one frame from the candidate frames and executes it during training, has also been proposed in the past (see, for example, reference 3).

[0118] (2.2.2 Frames using PCA for molecular structures) As a method for calculating frames, for molecular structures, principal component analysis (PCA) is performed on the matrix P representing the atomic coordinates of the molecule, and a frame E = [e １ , e ２ , e ３ Methods for determining ] are known (see, for example, reference 3).

[0119] Specifically, the dimension d of the matrix P that represents the coordinates. Ｐ If the result is 3, calculate the 3x3 covariance matrix Σ for matrix P, and then find its eigenvalues ​​λ. １ , λ ２ , λ ３and the eigenvector e whose length is normalized to 1 １ , e ２ , e ３ is calculated. Here, assuming there is no degeneracy of eigenvalues and the magnitude relationship of eigenvalues is λ １ > λ ２ > λ ３ and using e １ , e ２ , e ３ even if the order of is uniquely arranged, there remains an arbitrariness in the signs of the eigenvectors. Therefore, the frame is E = [±e １ , ±e ２ , ±e ３ and there are 2 ３ = 8 candidates. If the rotation matrix R representing the target rotation excludes reflection, except for the case where det(E) = -1, the candidates in this case will be 4. Regardless of the sign of det(E), when frame averaging or probabilistic frame averaging based on Equation (15) is performed for all 8 frame candidates, the function f ｉ will be a function invariant to the enantiomers of structure P. On the other hand, if 4 frame candidates are targeted when det(E) is determined to be either positive or negative, the function f ｉ will distinguish the differences between the enantiomers of structure P and return different values.

[0120] (2.2.3 Frame Using PCA for Crystal Structure) For a crystal structure, a method is known to define a frame E = [e1, e2, e3] consisting of orthonormal vectors by similarly performing PCA on the atomic coordinates within the unit cell of the crystal structure represented by the unit cell (see, for example, Reference 2).

[0121] However, when the PCA has a high symmetry in the matrix P representing the structure, eigenvalue degeneracy of the covariance matrix occurs, and there is a problem that rotational arbitrariness remains with respect to the frame axes. In particular, crystal structures often have such symmetry, and in particular, in the cubic crystal system, there is a problem that rotational arbitrariness remains in all three axes. In addition, for crystal structures, generally there is also arbitrariness in the way of taking the unit cell itself, and the above method has a problem that different results are calculated for the same structure that is just a change in the representation of the unit cell.

[0122] (2.2.4 Frame Using Lattice Vectors for Crystal Structure) For crystal structures, a frame method using lattice vectors has been proposed (see, for example, Reference 4). This considers the linear sum with integer coefficients represented by the following formula (16) for three lattice vectors l １ , l ２ , l ３ representing the translational directions of the unit cell.

[0123]

[0124] Then, from the combinations of the coefficients n １ , n ２ , n ３ of the above formula (16), a set of three linearly independent vectors is selected as the frame E = [e1, e2, e3] in ascending order of the norm ||e|| ２ . When there are multiple candidates for e with the same length, there is no rule for which one to choose.

[0125] (2.3 Adaptive Frame Linked to Interaction) The frames proposed so far have been directly defined for the input structure P. On the other hand, considering the use for which the frame is used, it is for effectively using geometric features such as the relative positions between nodes when updating the states of nodes by the interaction between nodes represented by the above formula (2).

[0126] At this time, in the interaction model such as the above formula (2), if there is a node j for which the weight w ｉｊ becomes 0, that node is x ｉIt does not affect the state update. Therefore, within this operation, the existence of such node j should be ignored, and when defining the frame, it is considered that defining the frame for a structure in which the coordinates of such node have been excluded from P will allow for the definition of a coordinate system that better captures the effect of this function.

[0127] Therefore, in the embodiments of this disclosure, when updating the state of each node i, the weight of the inter-node interaction w ｉ Frame E is such that the coordinates of nodes with larger weights are given more importance, while those with smaller weights are disregarded or ignored. ｉ This is calculated for each node i as follows:

[0128]

[0129] In this way, local frames E are adaptively calculated according to the interaction weights. ｉ Using this, the state vector x ｉ This frame E allows for more effective extraction of the geometric features necessary for updating. ｉ Using this, the state update operation in equation (2) above can be rewritten as follows.

[0130]

[0131] Local frame E ｉ Projection information E † ｉ By using P, rotation-invariant features can be extracted, and features that are inherently rotation-invariant, such as distance, can also be calculated from the input coordinates P and used.

[0132] (3) Method for constructing an adaptive frame) In the above formula (17) frame （ｉ） (speaking) ｉ Here are some implementation examples of the function P). The purpose of this function is to generate a fixed number (K) axis vectors [e １ , e ２ , ..., e Ｋ ] (matrix E ｉ The task is to determine and output one or more pairs of (represented by ). Furthermore, the following steps may be commonly used within this function.

[0133] (Non-negative weighting) Weight vector w ｉ Each component w ｉｊ It assumes a non-negative scalar value, but if given as an arbitrary real number, a transformation will be performed to make the weights non-negative. ｉ ←φ(w ｉ ) can be applied beforehand. As an example of the transformation, take the absolute value for each element lol ｉ ←|w ｉｊ | are some examples.

[0134] (Relative position coordinates) Instead of absolute position coordinates P, relative position coordinates P are used that are invariant to translational operations on P. ｉ Convert to and use. P ｉ As an example, it is possible to use relative position coordinates centered on node i, which are expressed by the following equations (19) to (21).

[0135]

[0136] Note that in the above equation, p ｉ Instead of subtracting, the mean coordinate p of P is expressed by the following formula. － You may use relative position coordinates obtained by subtracting from each node's coordinate p.

[0137]

[0138] Alternatively, in the above equation, p ｉ Instead of subtracting, the weighted average coordinate p is expressed by the following formula. － ｗｉ You may use relative position coordinates obtained by subtracting from each node's coordinate p.

[0139]

[0140] These translation-invariant relative position coordinates are collectively called P ｉ Represented by E as P ｉ By determining based on E ｉ This is invariant under translational operations on P.

[0141] (Exclusion of target node i) P ｉ or w ｉ You may delete the row or column corresponding to node i from among them. In particular p ｉ Relative position coordinates P centered on this coordinate ｉNow, let's consider the relative coordinate p corresponding to i. ｉｊ Since it becomes a zero vector, it is excluded.

[0142] (Adding random noise to weights) In the frame selection algorithm, the weights w ｉ This includes operations such as selecting an axis based on the magnitude of each value. When multiple nodes have the same weight, it may not be possible to determine the magnitude relationship, so the weight w ｉ You may perform an operation to randomly order each element by assigning a small random number to it. The random number is assigned to each w ｉｊ Both addition and multiplication are possible. Such operations are equivalent to applying stochastic frame averaging in existing studies. Furthermore, such operations on weights are also effective for the eigenvalue degeneracy problem in weighted PCA.

[0143] (Calculation of multiple frame candidates) If multiple frame candidates can be calculated based on factors such as the relative size or the arbitrariness of the positive and negative signs, in addition to probabilistically selecting one by methods such as adding random noise to the weights as described above, the results of state updates from multiple candidate frames may be averaged, as shown in equation (15) above. This is equivalent to applying frame averaging from existing research.

[0144] (Order of multiple frame axes) In the frame determination method described below in the embodiments of this disclosure, the frame axis [e １ , e ２ , ..., e Ｋ A certain order is set for ], but these orders can be rearranged according to certain rules. For example, the order of the axes obtained by the following method can be reversed to [e Ｋ ,e Ｋ－１ ,...,e1] may be the final output.

[0145] (3.1 Weighted PCA Frame) (Step 3.1.1 Calculation of Weighted Covariance Matrix) Relative position coordinates P centered on node i ｉ We calculate the weighted covariance matrix. The weighted covariance matrix is ​​expressed by the following formula.

[0146]

[0147] Note that the weighted covariance matrix Σ is used here. ｉ 1 / Σ for the whole ｊ lol ｉｊ Multiplying by a coefficient such as 1 / N does not change the final result, so the explanation of the difference is omitted. Also, relative position coordinate P ｉ Alternatively, normalized mutual relative position coordinates may be used, in which each relative position vector is normalized to a length of 1.

[0148] (Step 3.1.2 Obtaining Eigenvectors) Weighted Covariance Matrix Σ ｉ d Ｐ The number of eigenvalues ​​λ １ , λ ２ , ..., λ ｄＰ and the corresponding eigenvector e １ , e ２ , ..., e ｄＰ We obtain the following. Note that dP is P or Σ ｉ The spatial dimension is ||e||. Also, each eigenvector e has length ||e||. ２ Since there is an element of arbitrariness, it is assumed that the length is normalized to a constant value such as 1.

[0149] (Step 3.1.2 Rearranging Eigenvectors) Rearrange the eigenvectors according to the relative magnitudes of the eigenvalues. The order of the eigenvalues ​​(ascending or descending) does not matter. Coordinate P ｉ When there is high symmetry, the eigenvalue λ １ , λ ２ , ..., λ ｄＰ Multiple eigenvalues ​​with the same value may appear within the matrix. In this case, the "random noise addition to weights" described above can be used to randomly determine the order relationship between multiple eigenvalues ​​with equal values. １ , λ ２ , ..., λ ｄＰ Random noise can be directly added to the data to perform the ordering.

[0150] (Step 3.1.2 Output of the frame) Reordered eigenvector e １ , e ２ , ..., e ｄＰ Select K eigenvectors from and, while preserving their order, Eｉ = [±e １ , ±e ２ , ···, ±e Ｋ is output. The number of axes K can be set to any value as an integer within the range of 1 ≤ K ≤ d Ｐ . Since there is an arbitrariness in the sign for each eigenvector e, when selecting K eigenvectors, 2 Ｋ types of frame candidates are conceivable. In particular, when K = d Ｐ , 2 ｄＰ types of candidates are conceivable, but at this time, depending on the type of rotation targeted, the candidates may be narrowed down according to the sign of det(E ｉ ). For example, when excluding reflection from the target, that is, when the function f - ｉ←ｊ in formula (18) is to distinguish enantiomers, only E[[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​After determining each e in step 1 and step 2(c) above, e ← e / ||e|| ２ The length may be normalized as follows. In this case, the equation in step 2(c) can be simplified as follows.

[0157]

[0158] Step 2(a) Evaluation function w' ｉｊ The function φ(p) inside ｉｊ ,e ｍ ) is vector p ｉｊ and e ｍ This is a function that returns a non-negative scalar value such that it is 0 when the two points are parallel and takes its maximum value when they are orthogonal. For example, p ｉｊ and e ｍ The angle θ between them ｉｊｍ The cosine function value of cosθ ｉｊｍ = (p ｉｊ / ||p ｉｊ || ２ ) Ｔ (e ｍ / ||e ｍ || ２ ) using φ(p ｉｊ , e ｍ ) = 1 - |cosθ ｉｊｍ It is given as |, etc.

[0159] (Step 3 Output of frame) The obtained [e １ , e ２ , ..., e Ｋ ] to frame E ｉ Output as follows. Note that the number of axes K is 1 ≤ K ≤ d Ｐ It can be set to any integer within the range K = d. Ｐ and d Ｐ If = 3, then the final axis e of the previous step. Ｋ The determination is made using the cross product operation without going through steps 2(a)-(c), Ｋ = e １ ×e ２ and e Ｋ = -e １ ×e ２ You may give any of the following as candidates. Or, e Ｋ = ±e １ ×e ２You may also give these two options as candidates.

[0160] (3.3 Non-orthogonal maximum frame) (Step 1 Axis selection according to the maximum weight) Weight vector w ｉ Each element lol ｉｊ From among them, select node j with the maximum weight, P ｉ The relative position coordinate p corresponding to j within this coordinate system ｉｊ to e １ I will choose it as such.

[0161] (Step 2: Selection of axes according to each weight) Below, the k-th axis e ｋ The decision operation is repeated sequentially for k = 2, 3, ..., K. (a) Select the node j that maximizes the evaluation function expressed by the following equation.

[0162]

[0163] (b) P ｉ The relative position coordinate p corresponding to j within this coordinate system ｉｊ to e ｋ Select as follows. After determining each e in step 1 and step 2(c) above, e ← e / ||e|| ２ The length may be normalized as follows. Also, the evaluation function w' in step 2(a) ｉｊ The function φ(p) inside ｉｊ , e ｍ ) is defined similarly to the case of the orthogonal maximum frame.

[0164] (Step 3 Output of frame) The obtained [e １ , e ２ , ..., e Ｋ ] to frame E ｉ It outputs as follows. Unlike the weighted PCA frame and the orthogonal maximum frame that determine orthogonal axes, the number of axes K can be set to any integer value of 1 or greater without any upper limit.

[0165] (3.4 Greedy Frame) (Step 1 Node Selection Based on Weights) Weight Vector w ｉ The elements lol ｉｊ From among them, select K nodes j in descending order of weight. １ ,j ２ , ..., j Ｋ Choose this option.

[0166] (Step 2: Axis selection according to the selected node) Relative position coordinates [p] corresponding to the selected node ｉｊ１ , p ｉｊ２ , ..., p ｉｊＫ ] to P ｉ Obtained from among them, this is the frame axis [e １ , e ２ , ..., e Ｋ Select as ] and frame E ｉ Outputs the following. Here, each axis vector e is e ← e / ||e|| ２ The length may be normalized as follows. Unlike the weighted PCA frame and the orthogonal maximum frame which determine orthogonal axes, the number of axes K can be set to any integer value of 1 or more without upper limit.

[0167] (3.5 Mixed Frames) Multiple frame selection methods as described above may be combined. For example, M types of frame selection methods may be adopted, and K from the mth method. ｍ Individual frame axis E ｉ （ｍ） = [e （ｍ） １ , e （ｍ） ２ , ..., e （ｍ） Ｋｍ When obtaining ], a frame in which those axes are aligned may be used. This frame is represented by the following formula.

[0168]

[0169] Furthermore, if there are clearly overlapping axes among the combinations, they may be consolidated into one. For example, in the three methods of orthogonal maximal frames, non-orthogonal maximal frames, and greedy frames, e is common. １ Weight as such lol ｉｊ The relative position coordinates p of node j where the value is maximized. ｉｊ The one that is chosen.

[0170] (4. Design of features between nodes) The key to this method is that in the state update based on the interaction represented by equation (2) above, the projection information E is as shown in equation (18) above. † ｉ This involves performing calculations that include some or all of the information of P.

[0171] Here, frame E ｉ Whether each axis vector e within the graph is normalized to a length of 1, or whether they are orthogonal to each other, can be arbitrarily set during the design process.

[0172] Furthermore, the relative position coordinates P defined above ｉ Projection information E using † ｉ P ｉ Projection information E † ｉ This includes the calculation of P. Because the relative position coordinates P ｉ The reference coordinate p(p ｉ It can be expressed as in equation (21A) using (or mean coordinates, etc.) and calculated as in equation (21b).

[0173] (21A) (21B)

[0174] Furthermore, projection information E † ｉ For P, the function l(||p) is the length of each coordinate. ｉ || ２ A diagonal matrix D with ) on its diagonals. ｌ（Ｐ） = diag(l(||p １ || ２ ), l(||p ２ || ２ ), ..., l(||p Ｎ || ２ Using E † ｉ PD ｌ（Ｐ） In that case, projection information E † ｉ Let P be included. Using l(r) = 1 / r, PD ｌ（Ｐ） This is the normalized length of each coordinate vector p. Similarly, E † ｉ P ｉ For the diagonal matrix D ｌ（Ｐｉ） When multiplied by, it can be expressed as shown in the following equation (21C), and thus the projection information E † ｉ This includes P.

[0175]

[0176] When using the attention mechanism of the transformer, this information is represented by the internode feature b in equation (10) above. ｉｊ This is used in the calculation of E. Below, these projection information E † ｉ Specific examples of rotation-invariant and translation-invariant features containing P or P are given.

[0177] In common, a G-dimensional Gaussian basis function Γ is used as a function to convert scalar values ​​into vectors of arbitrary dimensions. The Gaussian basis function Γ is defined by the parameters of the mean and standard deviation of G Gaussian bases (represented by the G-dimensional vectors μ and σ, respectively), and the d-th dimension component is expressed as follows.

[0178]

[0179] Here, μ and σ may be given as constant parameters, or they may be learned as learnable parameters along with the other parameters.

[0180] (4.1 Relative position information) Coordinates p of node i ｉ Relative position coordinates P centered on this coordinate ｉ And, frame E where each axis vector e is normalized to length 1. ｉ Using and, projection information E † ｉ P ｉ Calculate E † ｉ P ｉ Projection information E corresponding to node j within the node † ｉ P ｉｊ This is the relative position vector p between node i and node j. ｉｊ This is the result of transforming it into rotation-invariant frame coordinates. E is a K-dimensional vector. † ｉ P ｉｊ The k-th component of is transformed into a vector using the Gaussian basis function Γ, and then linearly transformed and summed up, resulting in the quantity b in equation (23). ｉｊ This can be used as a translation- and rotation-invariant internode feature. Here, W １ , ..., WＫ These are learnable matrix parameters.

[0181]

[0182] (4.2 Angle Information) Coordinates p of node i ｉ Relative position coordinates P centered on this coordinate ｉ And a diagonal matrix D whose diagonals are the reciprocals of the lengths of each relative position. ｌ（Ｐｉ） And, frame E where each axis vector e is normalized to length 1. ｉ Using E † ｉ P ｉ D ｌ（Ｐｉ） Calculate E † ｉ P ｉ D ｌ（Ｐｉ） Projection information E corresponding to node j within the node † ｉ p ｉｊ / ||p ｉｊ || ２ This is the relative position vector between node i and node j and each axis e of the frame. ｋ This represents the value of the cosine function of the angle. These are each transformed into vectors using the Gaussian basis function Γ, and then linearly transformed and summed up to obtain the quantity b in the following equation (24). ｉｊ This can be used as a translation- and rotation-invariant internode feature. Here, W １ , ..., W Ｋ These are learnable matrix parameters.

[0183]

[0184] (4.3 Distance Information) The distance between nodes is inherently a quantity invariant under translation and rotation. Therefore, the length of the relative position vector ||p| is calculated directly from P. ｊ -p ｉ || ２ The quantity obtained by transforming this into a vector using the Gaussian basis function Γ, and then performing a linear transformation, is the following equation (25) b ｉｊ These can be used as translation-invariant and rotation-invariant internode features, where W is a learnable matrix parameter.

[0185]

[0186] (4.4 Combinations) The above three types of feature quantities b ｉｊ These can also be added together in any combination. In this case, the matrix parameter W or the Gaussian basis parameters μ and σ may be prepared individually for each type.

[0187] [Information Processing Device of the Embodiment] Figure 8A is a block diagram showing the hardware configuration of the trained model generation device 14A according to the embodiment. As shown in Figure 8A, the trained model generation device 14A includes a CPU (Central Processing Unit) 42A, a GPU (Graphics Processing Unit) 43A, a memory 44A, a GPU memory 45A, a storage device 46A, an input / output interface (Interface) 48A, a storage medium reader 50A, and a communication interface 52A. Each component is connected to the others so as to be able to communicate with each other via a bus 54A.

[0188] The storage device 46A stores a trained model generation program for executing the processes described later. The CPU 42A and GPU 43A, which are examples of processors, are central processing units that execute various programs and control various components. Specifically, the CPU 42A reads a program from the storage device 46A and executes the program using memory 44A as a workspace. The CPU 42A performs the various calculation processes described above according to the program stored in the storage device 46A. The GPU 43A also reads a program from the storage device 46A and executes the program using GPU memory 45A as a workspace. The GPU 43A performs the various calculation processes described above according to the program stored in the storage device 46A.

[0189] More specifically, the GPU 43A and GPU memory 45A primarily efficiently perform parallel processing of various numerical calculations within the model described in the trained model generation program. The GPU 43A reads the GPU computation program from the program read from the storage device 46A and executes the program using the GPU memory 45A as its workspace.

[0190] Memory 44A and GPU memory 45A are composed of RAM (Random Access Memory) and temporarily store programs and data as a working area. Storage device 46A is composed of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0191] The I / F 48A is an interface for inputting data from and outputting data to external devices. It may also be connected to various input devices, such as keyboards and mice, and output devices, such as displays and printers, for outputting various types of information. A touch panel display may be used as an output device and function as an input device.

[0192] The storage medium reader 50A reads data stored on various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0193] Communication I / F 52A is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® are used.

[0194] The trained model generation device 14A of this embodiment generates a trained model for calculating information about multiple atoms constituting a molecular structure or crystal structure using the method described above.

[0195] The functional configuration of the pre-trained model generation system 10A of this embodiment will now be described. As shown in Figure 8B, the pre-trained model generation system 10A of this embodiment includes an input device 12A, a pre-trained model generation device 14A, and an output device 16A.

[0196] The input device 12A is a device for inputting various types of information to the trained model generation device 14A. The output device 16A is a device for outputting various types of information calculated by the trained model generation device 14A.

[0197] Functionally, the trained model generation device 14A includes a learning unit 18A. Furthermore, a data storage unit 30A and a trained model storage unit 32A are provided in a predetermined storage area of ​​the trained model generation device 14A. Each functional configuration is realized by the CPU 42A reading each program stored in the storage device 46A, expanding it into memory 44A, and executing it. At this time, some calculation processing programs are sent to the GPU 43A for efficient parallel processing.

[0198] First, the trained model generator 14A trains a transformer, which is an example of a neural network. In the following, at least one atom from among the multiple atoms constituting the molecular structure or crystal structure will be treated as a single node.

[0199] The data storage unit 30A stores training data representing molecular or crystal structures. The training data represents molecular or crystal structures and corresponds to the machine learning algorithm used to train the transformer.

[0200] For example, when training a transformer using unsupervised machine learning or self-supervised learning, molecular structure data or crystal structure data without correct labels is stored in the data storage unit 30A as training data. When training a transformer using supervised machine learning, molecular structure data or crystal structure data with correct labels is stored in the data storage unit 30A as training data. When training a transformer using reinforcement learning, data representing simulation results, etc., is stored in the data storage unit 30A as training data.

[0201] Furthermore, the data storage unit 30A stores various types of information necessary for the trained model generation device 14A to perform predetermined information processing.

[0202] When the learning unit 18 receives an instruction signal to train the Transformer, it reads out the training data stored in the data storage unit 30.

[0203] The learning unit 18 reads the training data stored in the data storage unit 30. Then, based on the training data, the learning unit 18 trains the Transformer using a predetermined machine learning algorithm.

[0204] Specifically, the learning unit 18 obtains a trained transformer by training the transformer TF shown in Figure 7. Then, the learning unit 18 stores the trained transformer in the trained model storage unit 32.

[0205] Furthermore, the state vector x is updated in the learned attention mechanism within the multi-head attention layer MA1 of the encoder TF1 of the transformer TF shown in Figure 7, or within the masked multi-head attention layer MA2 of the decoder TF2. Note that each of the above processes is configured to be executable within the multi-head attention layer. The encoder TF1 or decoder TF2 outputs an invariant, an invariant classification, an equivariant, or a feature, depending on the task. An invariant, an invariant classification, an equivariant, or a feature is an example of information about multiple atoms. Information about multiple atoms is an example of information with a predetermined symmetry in the structure. "Determined symmetry" is mathematically defined as a group; in the case of rotational and translational symmetry in N-dimensional space, it is the Euclidean group E(N) (N is a natural number, and so on), and for rotational symmetry, it is the orthogonal group O(N), etc. (Special is added to the beginning of the term if det=1). For example, the symmetry used in this embodiment is SE(3). "Having symmetry" means either invariant, equivariant, or both with respect to the transformations generated by the above group.

[0206] [Information Processing Device of an Embodiment] Figure 9A is a block diagram showing the hardware configuration of the information processing device 14B according to an embodiment. As shown in Figure 9A, the information processing device 14B includes a CPU (Central Processing Unit) 42B, a GPU (Graphics Processing Unit) 43B, a memory 44B, a GPU memory 45B, a storage device 46B, an input / output interface (Interface) 48B, a storage medium reader 50B, and a communication interface 52B. Each component is connected to the others so as to be able to communicate with each other via a bus 54B.

[0207] The storage device 46B stores information processing programs for executing the processes described later. The CPU 42B and GPU 43B, which are examples of processors, are central processing units that execute various programs and control various components. Specifically, the CPU 42B reads a program from the storage device 46B and executes the program using memory 44B as a workspace. The CPU 42B performs the various calculation processes described above according to the program stored in the storage device 46B. The GPU 43B also reads a program from the storage device 46B and executes the program using GPU memory 45B as a workspace. The GPU 43B performs the various calculation processes described above according to the program stored in the storage device 46B.

[0208] More specifically, the GPU 43B and GPU memory 45B primarily efficiently perform parallel processing of various numerical calculations within models described in the information processing program. The GPU 43B reads the GPU calculation program from the program read from the storage device 46B and executes the program using the GPU memory 45B as its workspace.

[0209] Memory 44B and GPU memory 45B are composed of RAM (Random Access Memory) and temporarily store programs and data as a working area. Storage device 46B is composed of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0210] The I / F 48B is an interface for inputting data from and outputting data to external devices. It may also be connected to various input devices, such as keyboards and mice, and output devices, such as displays and printers, for outputting various types of information. A touch panel display may be used as an output device and function as an input device.

[0211] The storage medium reader 50B reads data stored on various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0212] Communication I / F 52B is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® are used.

[0213] The information processing device 14B of this embodiment calculates information about a plurality of atoms constituting a molecular structure or crystal structure using the method described above.

[0214] The functional configuration of the information processing system 10B of this embodiment will now be described. As shown in Figure 9B, the information processing system 10B of this embodiment includes an input device 12B, an information processing device 14B, and an output device 16B.

[0215] The input device 12B is a device for inputting various types of information to the information processing device 14B. The output device 16B is a device for outputting various types of information calculated by the information processing device 14B.

[0216] Functionally, the information processing device 14B includes a processing unit 20B. The processing unit 20B is an example of the receiving unit, calculation unit, and extraction unit of this disclosure. In addition, a data storage unit 30B and a trained model storage unit 32B are provided in a predetermined storage area of ​​the information processing device 14B. Each functional configuration is realized by the CPU 42B reading each program stored in the storage device 46B, expanding it into memory 14B, and executing it.

[0217] When the information processing device 14B receives structural information representing a molecular structure or crystal structure, it calculates the state vector x of at least one atom constituting the molecular structure or crystal structure.

[0218] The data storage unit 30B stores various types of information necessary for the information processing device 14B to perform predetermined information processing. The trained model storage unit 32B stores trained transformers generated by the trained model generation device 14A.

[0219] When the processing unit 20B receives an instruction signal to calculate the state vector x of the nodes constituting the target molecular structure or crystal structure, it reads out the trained encoder and trained decoder, or both, from the trained transformers stored in the trained model storage unit 32B. The following explanation uses the case where the trained encoder is used to update the state vector x as an example, but is not limited to this. The state vector x may be updated using the trained decoder, or both the trained encoder and trained decoder may be used to update the state vector x.

[0220] Then, the processing unit 20B receives the state vector x of each of the multiple nodes as input to the trained encoder.

[0221] A trained encoder accepts the state of each node representing one or more atoms among the multiple atoms constituting a molecular or crystal structure. Specifically, the trained encoder accepts the initial value of the state vector x for each of the multiple nodes.

[0222] Next, the trained encoder performs the processing described above in its multi-head attention layer. Specifically, for each of the multiple nodes, the trained encoder assigns a weight w representing the interaction strength between the state of node i of interest and the states of multiple other nodes j. ｉｊ Based on this, frame E represents the coordinate axes at node i of interest. ｉ The system then calculates the frame E calculated for each of the nodes, and the trained encoder then performs a process to calculate information about multiple atoms.

[0223] More specifically, first, the trained encoder takes the state vector x of each of the multiple nodes. ｉ And, position vector p representing the spatial position of each of the multiple nodes. ｉ We accept it.

[0224] Next, the trained encoder outputs the state vector x of each of the multiple nodes. ｉ Based on this, weights w represent the interaction strength between the node of interest i and multiple other nodes j. ｉｊ A weight vector w whose components are ｉ To calculate this, a weight vector w is calculated for each node i of interest. ｉ The following is calculated. Specifically, the trained encoder calculates the weight vector w for each node i of interest according to the above equation (D). ｉ This is calculated.

[0225] Next, the trained encoder calculates the state vector x of node i for each of the multiple nodes. ｉ and the state vector x of multiple other nodes j ｊ A weight vector w representing the interaction strength between and ｉ Based on this, frame E represents the coordinate axes at node i of interest. ｉ This is calculated. As a result, frame E is set for each of the multiple nodes. The trained encoder calculates the frame for each node by performing the process described above (3. Method for constructing adaptive frames).

[0226] (Weighted PCA Frames) For example, a trained encoder calculates a frame for each node by performing the process described in (3.1 Weighted PCA Frames) above. Specifically, when calculating a frame for each node, the trained encoder calculates the relative position vector p between the node of interest i and the other nodes j for each of the multiple nodes. ｊ -p ｉ Alternatively, a relative coordinate matrix P whose components are vectors whose length has been normalized to 1. ｉ And the weights w between the node of interest i and several other nodes j. ｉｊ A weight vector w whose components are ｉ Weight matrix diag(w) ｉ Based on this, by performing Principal Component Analysis (PCA), a frame E represented by one or more axis vectors is obtained. ｉ Calculate.

[0227] (Orthogonal Maximum Frame) Alternatively, for example, a trained encoder may calculate the frame for each node by performing the process described in (3.2 Orthogonal Maximum Frame) above.

[0228] In this case, the trained encoder may calculate mutually orthogonal axis vectors as frames by performing the following calculation process.

[0229] For example, when a trained encoder calculates a frame for each node, it uses the weights w between the node of interest i and several other nodes j. ｉｊ From among them, select the other node j1 corresponding to the weight with the largest value. Then, the trained encoder outputs the position vector p of the selected other node j1. ｊ１ node p that is attracting attention ｉ The relative position vector p represents the difference between and . ｊ１ -p ｉ The normalized vector corresponding to the first axis vector e １ Set it as follows.

[0230] Next, the trained encoder takes the first axis vector e １Select another node j2 that minimizes the evaluation function between it and the first axis vector e. １ Select another node j2 such that the absolute value of the inner product between it and the selected node j2 becomes smaller and the weight becomes larger. Then, the trained encoder calculates the position vector p of the selected other node j2. ｊ２ The position vector p of node i of interest ｉ The relative position vector p represents the difference between and . ｊ２ -p ｉ Select the corresponding vector and set the selected vector to the first axis vector e １ By modifying it to be orthogonal, the first axis vector e １ The second axis vector e is orthogonal to it. ２ Set it.

[0231] Then, the trained encoder takes the first axis vector e １ and the second axis vector e ２ A third axis vector e that is orthogonal to it. ３ By setting this, frame E is represented by one or more axis vectors. ｉ Calculate the first axis vector e. For example, by performing the calculation of the cross product, １ and the second axis vector e ２ A third axis vector e that is orthogonal to it. ３ It is possible to identify this.

[0232] (Non-orthogonal maximum frame) Alternatively, for example, a trained encoder may calculate the frame for each node by performing the process described in (3.3 Non-orthogonal maximum frame) above.

[0233] In this case, the trained encoder may calculate non-orthogonal axis vectors as frames by performing the following calculation process.

[0234] For example, when a trained encoder calculates a frame for each node, it uses the weights w between the node of interest i and several other nodes j. ｉｊFrom among them, select the other node j1 corresponding to the weight with the largest value. Then, the trained encoder outputs the position vector p of the selected other node j1. ｊ１ The position vector p of node i of interest ｉ The relative position vector p represents the difference between and . ｊ１ -p ｉ The normalized vector corresponding to the first axis vector e １ Set it as follows.

[0235] Next, the trained encoder takes the first axis vector e １ Select another node j2 such that the absolute value of the inner product between it and the selected node j2 becomes smaller and the weight becomes larger. Then, the trained encoder calculates the position vector p of the selected other node j2. ｊ２ node p that is attracting attention ｉ The relative position vector p represents the difference between and . ｊ２ -p ｉ Select the corresponding vector and set the selected vector to the second axis vector e ２ Set it as follows.

[0236] Next, the trained encoder takes the first axis vector e １ The absolute value of the dot product between and becomes smaller, and the second axis vector e ２ Select another node j3 such that the absolute value of the inner product between it and the other node becomes smaller and the weight becomes larger, and then select the position vector p of the selected other node j3. ｊ３ node p that is attracting attention ｉ The relative position vector p represents the difference between and . ｊ３ -p ｉ Select the corresponding vector, and set the selected vector to the third axis vector e ３ Set it as follows.

[0237] Alternatively, for example, a trained encoder may compute a per-node frame by performing the process described in (3.4 Greedy Frames) or (3.5 Mixed Frames) above. When performing (3.4 Greedy Frames) above, the trained encoder, when computed a per-node frame, selects multiple other nodes for each of the multiple nodes, in descending order of weight from the weights between the node of interest and multiple other nodes. The trained encoder then computes a frame represented by one or more axis vectors by setting each of the vectors corresponding to the relative position vectors representing the difference between the position vectors of the selected multiple other nodes and the node of interest as one or more axis vectors.

[0238] Next, the trained encoder calculates the frame E for each of the multiple nodes, for the node i of interest. ｉ For this, the position p of node i of interest ｉ and the position p of multiple other nodes j ｊ The relative position vector p represents the difference between the two. ｊ -p ｉ By projecting this, we obtain relationship information between the node of interest i and several other nodes j, which is b. ｉｊ The following is calculated. For example, a trained encoder calculates the relational information b according to the above formula (F). ｉｊ Calculate.

[0239] Then, the trained encoder calculates relational information b for each of the multiple nodes, with respect to the node i of interest. ｉｊ And the weight lol ｉｊ Based on this, the state vector x of node i of interest ｉ The process of updating is repeated. For example, a trained encoder updates the state vector x of node i of interest according to equation (E) above. ｉ Repeat the process of updating.

[0240] Then, the trained encoder updates the node state vector x for each of the multiple nodes. ｉThe process is executed to output '. For example, the trained encoder outputs the updated node state vector x of equation (E) above. ｉ Output '.

[0241] As mentioned above, the trained encoder has a layer configuration that allows it to perform each of the above processes.

[0242] <Operation of the Pre-trained Model Generator 14A> Next, the operation of the pre-trained model generator 14A in this embodiment will be explained with reference to the figure. When the pre-trained model generator 14A receives training data, it stores it in the data storage unit 30. Then, when the pre-trained model generator 14A receives an instruction signal to start the training process, it executes the pre-trained model generation processing routine shown in Figure 10.

[0243] <Trained Model Generation Processing Routine> In step S100, the learning unit 18A acquires multiple training data stored in the data storage unit 30A.

[0244] In step S102, the learning unit 18A trains the Transformer using a known machine learning algorithm based on the training data acquired in step S100.

[0245] In step S104, the learning unit 18A acquires the learned transformers.

[0246] In step S106, the learning unit 18A stores the learned transformer acquired in step S104 into the learned model storage unit 32A.

[0247] Next, when the information processing device 14B receives structural information of the target molecular structure or crystal structure, it executes the information processing routine shown in Figure 11.

[0248] <Information Processing Routine> In step S200, the processing unit 20B receives the input structural information.

[0249] In step S202, the processing unit 20B sets the initial value of the state vector for each node of the molecular structure or crystal structure. For example, the processing unit 20B sets the initial value of the state vector x according to equations (3) to (5) above.

[0250] In step S204, the processing unit 20B reads out the encoder portion of the trained transformer stored in the trained model storage unit 32B.

[0251] In step S206, the processing unit 20B inputs the node-specific state vector x initialized in step S202 to the learned encoder read in step S204.

[0252] In step S208, the processing unit 20B causes the learned encoder to perform processing. The processing in step S208 is performed by the subroutine shown in Figure 12.

[0253] In step S300 of Figure 12, the trained encoder receives the state vector x of each of the multiple nodes.

[0254] In step S302 of Figure 12, the trained encoder uses the state vector x of each of the multiple nodes. ｉ Based on this, weights w represent the interaction strength between the node of interest i and multiple other nodes j. ｉｊ A weight vector w whose components are ｉ Calculate.

[0255] In step S304 of Figure 12, the trained encoder calculates the state vector x of node i of interest for each of the multiple nodes. ｉ and the state vector x of multiple other nodes j ｊ A weight vector w representing the interaction strength between and ｉ Based on this, frame E represents the coordinate axes at node i of interest. ｉ This calculates the frame E for each node. For example, by performing each of the processes described above, the frame E for each node is calculated. ｉ Calculate.

[0256] In step S306 of Figure 12, the trained encoder calculates the frame E for each of the multiple nodes for the node i of interest. ｉ For this, the position p of node i of interest ｉ and the position p of multiple other nodes j ｊ The relative position vector p represents the difference between the two. ｊ -p ｉ By projecting this, we obtain relationship information between the node of interest i and several other nodes j, which is b. ｉｊ The following is calculated. For example, a trained encoder calculates the relational information b according to the above formula (F). ｉｊ Calculate.

[0257] In step S306 of Figure 12, the trained encoder calculates relational information b for each of the multiple nodes, with respect to the node i of interest. ｉｊ And the weight lol ｉｊ Based on this, the state vector x of node i of interest ｉ The process of updating is repeated. For example, a trained encoder updates the state vector x of node i of interest according to equation (E) above. ｉ The process of updating is repeated. Then, the trained encoder updates the node state vector x for each of the multiple nodes. ｉ The process is executed to output '. For example, the trained encoder outputs the updated node state vector x of equation (E) above. ｉ Output '.

[0258] Next, in step S210 of Figure 11, the processing unit 20B obtains the updated state vector x' output in step S208.

[0259] Then, in step S212, the processing unit 20B outputs the updated state vector x'.

[0260] As described above, the updated state vector x' corresponds to the physical properties of the molecular or crystal structure, the forces acting on the molecular or crystal structure, or input to another neural network. As described above, the updated state vector x' reflects information about frame E set for each node. Furthermore, frame E set for each node is a coordinate system that takes into account the interactions that exist between atoms. Therefore, when performing calculations on multiple atoms constituting a molecular or crystal structure, it is possible to perform calculations that take into account the interactions that exist between atoms.

[0261] As explained above, when the information processing device 14B calculates information about multiple atoms constituting a molecular structure or crystal structure using a neural network, the neural network receives the state of each node representing one or more atoms among the multiple atoms constituting the molecular structure or crystal structure. Next, for each of the multiple nodes, the neural network calculates a frame representing the coordinate axes of the node of interest based on weights representing the interaction strength between the state of the node of interest and the states of the other nodes. Then, the neural network performs a process to calculate information about the multiple atoms based on the frames calculated for each of the multiple nodes. This makes it possible to perform calculations that take into account the interactions that exist between atoms when performing calculations about multiple atoms constituting a molecular structure or crystal structure.

[0262] When using machine learning models such as neural networks to predict physical properties or forces in molecular or crystal structures, it is expected that performance can be improved by incorporating not only interatomic distance information but also positional and directional information such as relative position vectors within the machine learning model. However, it is not obvious how to incorporate atomic positional or directional information while ensuring that the machine learning model remains invariant or identically variable to rotational or translational operations of the input structure's coordinate system.

[0263] In the embodiments of this disclosure, instead of using a predetermined fixed frame for each input structure, the frame is adaptively determined according to the target task or the inference status of atomic states within the machine learning model, and the machine learning model is trained accordingly. By adaptively learning the frame according to the inference state of each atom within each prediction task or machine learning model, rather than directly defining a fixed frame for the input structure, a frame that follows the operation described by the machine learning model can be realized, thereby improving the accuracy of prediction tasks. Furthermore, by using this in various neural networks that process molecular structures or crystal structures as input, more advanced information about the structure can be transmitted to subsequent networks, and improved performance can be expected in tasks such as structure generation.

[0264] Furthermore, the technology disclosed herein is not limited to the embodiments described above, and various modifications and applications are possible without departing from the gist of this disclosure.

[0265] [Modification] In the above embodiment, the use of a transformer as an example of a neural network was described, but it is not limited to this. A layer capable of performing the above-described processing may be constructed using a neural network different from a transformer, and the above-described processing may be performed using such a layer.

[0266] Furthermore, the molecular structure or crystal structure of the above embodiment may be, for example, an enantiomer. In this case, since an enantiomer corresponds to a reflection of a certain molecular structure or crystal structure in a mirror, it is possible to design the neural network to distinguish between enantiomers or not, depending on the intended application. For example, if frame E is a square matrix, and frame E is calculated such that the sign of det(E) is always constant, the neural network will distinguish between enantiomers. Conversely, if frame E is calculated such that the sign of det(E) changes randomly, the neural network will not distinguish between enantiomers. Similarly, by converting the above-mentioned signed projection information to absolute values, it is possible to avoid distinguishing between enantiomers.

[0267] Furthermore, when estimating these variables, the neural network calculates and updates rotational and translationally invariant feature vectors for each atom using interatomic relative distances and projection information from frames. From these, spatial geometric quantities such as forces are regressed and output based on an invariant coordinate system, and then estimated by converting the frames inversely back to the orientation of the input coordinate system. Here, the frames used in the inverse conversion of the output values ​​are the frames E calculated for each atom. ｉ Alternatively, a conventional fixed frame may be used. Furthermore, instead of inverse frame transformation, in the output layer, for example, v ｉ = Σ ｊ lol ｉｊ (p ｊ -p ｉ v is a weighted sum of geometric quantities such as relative position vectors calculated from the input structure, as shown above. ｉ By outputting this, it is possible to estimate the same variables in accordance with the orientation of the input structure. Here, the weight w ｉｊ This is calculated based on the state vectors of atoms, which are computed and updated within the neural network.

[0268] Furthermore, although the above embodiment describes the case where the structure represented as a set of nodes arranged in space is a molecular structure or a crystal structure, it is not limited to these. For example, each node of the target structure may consist of an atom, molecule, gene, human, mobile body, robot, tangible object, fluid, and physical entity in information space, or a combination of two or more of these. In this case, the above information processing device performs information processing using a neural network on the structure represented as a set of nodes arranged in space. Specifically, the information processing device receives input of the state of each node, calculates a frame representing the coordinate axes at each node based on the state between each node, and extracts information having a predetermined symmetry of the structure from the nodes using the frame. In this case, the frame is calculated by principal component analysis or the like. For example, the frame is calculated by the method exemplified in 3.1 to 3.5 above. The frame may be an orthogonal or non-orthogonal system. Having symmetry means either being invariant or equivariant with respect to symmetry. Furthermore, symmetry includes any or a combination of rotation, translation, and inversion in space.

[0269] Furthermore, as mentioned above, the above embodiment was described using the example of calculating the state vector x using a trained encoder, but it is not limited to this. The state vector x may be calculated using a trained decoder, or both a trained encoder and a trained decoder may be used to calculate the state vector x.

[0270] Furthermore, in the above embodiment, each process executed by the CPU or GPU after reading the software (program) may be executed by various processors other than the CPU or GPU. Examples of such processors include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacturing, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electrical circuits that have a circuit configuration specifically designed to execute a particular process, such as ASICs (Application Specific Integrated Circuits). Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU or GPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0271] Furthermore, although the above embodiment describes a configuration in which each program is pre-stored (installed) in a storage device, the invention is not limited to this. The program may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, Blu-ray disc, or USB memory. Alternatively, the program may be provided in a form that is downloaded from an external device via a network.

[0272] In this embodiment, a simulation experiment is conducted regarding the proposed method described above. In this embodiment, the task of predicting physical properties is simulated when the proposed method is applied to a lightweight neural network based on GeoFormer disclosed in the following reference. Specifically, in this embodiment, a task similar to the one shown in Table 1 of the following reference is simulated. The following reference uses the QM9 dataset.

[0273] References: "Geometric Transformer with Interatomic Positional Encoding", <https: / / proceedings.neurips.cc / paper_files / paper / 2023 / hash / aee2f03ecb2b2c1ea55a43946b651cfd-Abstract-Conference.html>

[0274] Table 3 below shows the simulation results of this embodiment. In the table below, μ represents the dipole moment of the molecule, and its unit is [mD]. G represents the free energy of the molecule, and its unit is [meV]. The values ​​shown in the table below are the mean absolute errors between the correct values ​​and the predicted values ​​of the physical properties. The "GeoFormer (lightweight version, no frame)" section below shows the errors when calculating the dipole moment and free energy, which are examples of molecular physical properties, using the lightweight version of GeoFormer, a neural network disclosed in the above reference. The "PCA (existing fixed frame)" section below shows the errors when calculating the dipole moment and free energy, which are examples of molecular physical properties, using PCA to set a fixed frame for the lightweight version of GeoFormer. The "Orthogonal Maximum (proposed)" section below shows the errors when calculating the dipole moment and free energy, which are examples of molecular physical properties, using the proposed frame (a frame formed by orthogonal axes) for the lightweight version of GeoFormer. Furthermore, the "Non-orthogonal Maximum (Proposed)" section below represents the error when calculating the dipole moment and free energy, which are examples of molecular properties, using the proposed frame (a frame formed by non-orthogonal axes) in the lightweight version of GeoFormer.

[0275] As shown in the table below, the proposed method of this embodiment has a smaller error and can estimate material properties with greater accuracy than conventional methods.

[0276] (Note) The following is a note regarding the nature of this disclosure.

[0277] (Note 1) An information processing method using a neural network for a structure represented as a set of nodes arranged in space, wherein the computer performs the following steps: receiving input of the state of each node; calculating a frame representing the coordinate axes at each node based on the states between each node; and extracting information having a predetermined symmetry of the structure from the nodes using the frame. (Note 2) The information processing method according to Note 1, wherein each node consists of an atom, molecule, gene, human, mobile body, robot, tangible object, fluid object, and physical entity in information space, or a combination of two or more of these. (Note 3) The information processing method according to Note 1, wherein the frame is calculated by principal component analysis. (Note 4) The information processing method according to Note 1, wherein the frame uses an orthogonal system. (Note 5) The information processing method according to Note 1, wherein the frame uses a non-orthogonal system. (Note 6) The information processing method according to Note 1, wherein having the symmetry means either being invariant or equivariant with respect to the symmetry. (Note 7) The information processing method according to Note 1, wherein the symmetry includes any or a combination of rotation, translation, and inversion in the space. (Note 8) The information processing method according to Note 1, wherein the physical properties of a structure composed of atoms are inferred using the information having the symmetry.(Note 9) The information processing method described in Note 8, wherein the physical properties include at least one of the following: formation energy, total energy, zero-point energy, enthalpy, free energy, internal energy, specific heat, total magnetization, absolute magnetization, HOMO (Highest Occupied Molecular Orbital) energy level, LUMO (Lowest Unoccupied Molecular Orbital) energy level, HOMO-LUMO gap, band gap, superconducting transition temperature, bulk modulus, shear modulus, energy at which a set of molecular or crystal structures bond, physical properties representing the classification of magnetism, whether or not it is a ferroelectric, whether or not it is a pyroelectric, whether or not it is a piezoelectric, whether or not it is superconducting, whether or not a set of molecular or crystal structures undergoes a predetermined chemical reaction, force acting on atoms, variable of atomic position to a stable state structure, dipole moment, and atomic position in each structure when a set of molecular or crystal structures bond. (Note 10) The information processing method according to Note 1, wherein the structure is a molecular structure or a crystal structure, and when calculating information about a plurality of atoms constituting the molecular structure or crystal structure using the neural network, the neural network receives the state of each node representing one or more atoms among the plurality of atoms constituting the molecular structure or crystal structure, the neural network calculates a frame representing the coordinate axes of the node of interest for each of the plurality of nodes based on weights representing the interaction strength between the state of the node of interest and the states of the plurality of other nodes, and the neural network calculates the symmetry information based on the frames calculated for each of the plurality of nodes.(Note 11) The information processing method according to Note 10, wherein the information having symmetry is the state of the node, the neural network further receives the spatial position of each of the plurality of nodes, calculates weights representing the interaction strength between the node of interest and the plurality of other nodes based on the state of each of the plurality of nodes, calculates relational information between the node of interest and the plurality of other nodes by projecting relative positional information representing the difference between the position of the node of interest and the positions of the plurality of other nodes onto the frame calculated for the node of interest for each of the plurality of nodes, repeatedly updates the state of the node of interest for each of the plurality of nodes based on the relational information and weights calculated for the node of interest, and outputs the updated state of the node for each of the plurality of nodes. (Note 12) The information processing method according to Note 11, wherein the state for each of the plurality of nodes is a state vector representing the state of the node, the spatial position of each of the plurality of nodes is a position vector representing the position of the node, the weight representing the interaction strength regarding the state between the node of interest and the plurality of other nodes is a weight vector whose components are the weights between the node of interest and the plurality of other nodes, the neural network calculates relational information between the node of interest and the plurality of other nodes by projecting a relative position vector representing the difference between the position vector of the node of interest and the position vectors of the plurality of other nodes onto the frame set for the node of interest for each of the plurality of nodes, repeatedly updates the state vector of the node of interest for each of the plurality of nodes based on the relational information and weights calculated for the node of interest, and outputs the updated state vector of the node of interest for each of the plurality of nodes.(Note 13) The information processing method according to Note 12, wherein the neural network calculates the frame represented by one or more axis vectors by performing principal component analysis on each of the multiple nodes, based on a relative coordinate matrix whose components are the relative position vectors between the node of interest and multiple other nodes, and a weight matrix whose components are the weight vectors between the node of interest and multiple other nodes. (Note 14) The information processing method according to Note 12, wherein the neural network calculates the frame represented by one or more axis vectors by performing principal component analysis on each of the multiple nodes, based on a relative coordinate matrix whose components are vectors whose components are the lengths of the relative position vectors between the node of interest and multiple other nodes normalized to 1, and a weight matrix whose components are the weight vectors between the node of interest and multiple other nodes. (Note 15) The information processing method according to Note 12, wherein when the neural network calculates a frame for each node, it selects another node corresponding to the maximum weight among the weights between the node of interest and a plurality of other nodes, sets a normalized vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the node of interest as the first axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest, sets a second axis vector orthogonal to the first axis vector by modifying the selected vector to be orthogonal to the first axis vector, and calculates the frame represented by one or more axis vectors by setting a third axis vector orthogonal to the first axis vector and the second axis vector.(Note 16) The information processing method according to Note 12, wherein when the neural network calculates a frame for each node, it selects another node corresponding to the maximum weight among the weights between the node of interest and a plurality of other nodes, sets a normalized vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest as the first axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the node of interest, sets the selected vector as the second axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the absolute value of the dot product with the second axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest, and sets the selected vector as the third axis vector. (Note 17) The information processing method according to Note 12, wherein the neural network, when calculating a frame for each node, selects a plurality of other nodes in descending order of weight from the weights between the node of interest and a plurality of other nodes for each of the plurality of nodes, and sets each of the vectors corresponding to the relative position vector representing the difference between the position vectors of the selected plurality of other nodes and the node of interest as one or more axis vectors, thereby calculating the frame represented by the one or more axis vectors. (Note 18) An information processing device using a neural network for a structure represented as a set of nodes arranged in space, comprising: a receiving unit that receives input of the state of each node; a calculation unit that calculates a frame representing the coordinate axes at each node based on the states between each node; and an extraction unit that extracts information having a predetermined symmetry of the structure from the nodes using the frame.(Note 19) A program for causing a computer to perform information processing using a neural network on a structure represented as a set of nodes arranged in space, the program receiving input of the state of each node, calculating a frame representing the coordinate axes at each node based on the states between each node, and using the frame to extract information having a predetermined symmetry of the structure from the nodes, and causing the computer to perform such processing.

[0278] The disclosure of Japanese Patent Application No. 2024-169463, filed on 27 September 2024, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. An information processing method using a neural network for a structure represented as a set of nodes arranged in space, wherein the computer performs the following steps: receiving input of the state of each node; calculating a frame representing the coordinate axes at each node based on the states between each node; and extracting information having a predetermined symmetry of the structure from the nodes using the frame.

2. The information processing method according to claim 1, wherein each node consists of an atom, molecule, gene, human, mobile body, robot, tangible object, fluid, and physical entity in information space, or a combination of two or more of these.

3. The information processing method according to claim 1, wherein the frame is calculated by principal component analysis.

4. The information processing method according to claim 1, wherein the frame uses an orthogonal system.

5. The information processing method according to claim 1, wherein the frame uses a non-orthogonal system.

6. The information processing method according to claim 1, wherein having the symmetry means either being invariant or equivariant with respect to the symmetry.

7. The information processing method according to claim 1, wherein the symmetry includes any or a combination of rotation, translation, and inversion in space.

8. The information processing method according to claim 1, wherein the physical properties of a structure composed of atoms are inferred using the symmetry information.

9. The information processing method according to claim 8, wherein the physical properties include at least one of the following: formation energy, total energy, zero-point energy, enthalpy, free energy, internal energy, specific heat, total magnetization, absolute magnetization, HOMO (Highest Occupied Molecular Orbital) energy level, LUMO (Lowest Unoccupied Molecular Orbital) energy level, HOMO-LUMO gap, band gap, superconducting transition temperature, bulk modulus, shear modulus, energy at which a set of molecular or crystal structures bond, physical properties representing the classification of magnetism, whether or not it is a ferroelectric, whether or not it is a pyroelectric, whether or not it is a piezoelectric, whether or not it is superconducting, whether or not a set of molecular or crystal structures undergoes a predetermined chemical reaction, force acting on atoms, variable of atomic position to a stable state structure, dipole moment, and atomic position in each structure when a set of molecular or crystal structures bond.

10. The information processing method according to claim 1, wherein the structure is a molecular structure or a crystal structure, and when calculating information about a plurality of atoms constituting the molecular structure or crystal structure using the neural network, the neural network receives the state of each node representing one or more atoms among the plurality of atoms constituting the molecular structure or crystal structure, the neural network calculates a frame representing the coordinate axes of the node of interest for each of the plurality of nodes based on weights representing the interaction strength between the state of the node of interest and the states of the plurality of other nodes, and the neural network calculates the symmetry information based on the frames calculated for each of the plurality of nodes.

11. The information processing method according to claim 10, wherein the symmetrical information is the state of the node, the neural network further receives the spatial position of each of the plurality of nodes, calculates weights representing the interaction strength between the node of interest and the plurality of other nodes based on the state of each of the plurality of nodes, calculates relational information between the node of interest and the plurality of other nodes by projecting relative positional information representing the difference between the position of the node of interest and the positions of the plurality of other nodes onto the frame calculated for the node of interest for each of the plurality of nodes, repeatedly updates the state of the node of interest for each of the plurality of nodes based on the relational information and weights calculated for the node of interest, and outputs the updated state of the node for each of the plurality of nodes.

12. The information processing method according to claim 11, wherein the state for each of the plurality of nodes is a state vector representing the state of the node, the spatial position of each of the plurality of nodes is a position vector representing the position of the node, the weight representing the interaction strength regarding the state between the node of interest and the plurality of other nodes is a weight vector whose components are the weights between the node of interest and the plurality of other nodes, the neural network calculates relational information between the node of interest and the plurality of other nodes by projecting a relative position vector representing the difference between the position vector of the node of interest and the position vectors of the plurality of other nodes onto the frame set for the node of interest for each of the plurality of nodes, repeatedly updates the state vector of the node of interest for each of the plurality of nodes based on the relational information and weights calculated for the node of interest, and outputs the updated state vector of the node of interest for each of the plurality of nodes.

13. The information processing method according to claim 12, wherein the neural network, when calculating the frame for each node, calculates the frame represented by one or more axis vectors by performing principal component analysis for each of the plurality of nodes based on a relative coordinate matrix whose components are the relative position vectors between the node of interest and a plurality of other nodes, and a weight matrix whose components are weight vectors whose components are the weights between the node of interest and a plurality of other nodes.

14. The information processing method according to claim 12, wherein the neural network calculates the frame represented by one or more axis vectors by performing principal component analysis for each of the plurality of nodes, based on a relative coordinate matrix whose components are vectors normalized to 1 in length of the relative position vector between the node of interest and the plurality of other nodes, and a weight matrix whose components are weight vectors whose components are the weights between the node of interest and the plurality of other nodes.

15. The information processing method according to claim 12, wherein when the neural network calculates a frame for each node, it selects another node corresponding to the maximum weight among the weights between the node of interest and a plurality of other nodes, sets a normalized vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the node of interest as the first axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest, sets a second axis vector orthogonal to the first axis vector by modifying the selected vector to be orthogonal to the first axis vector, and calculates the frame represented by one or more axis vectors by setting a third axis vector orthogonal to the first axis vector and the second axis vector.

16. The information processing method according to claim 12, wherein when the neural network calculates a frame for each node, it selects another node corresponding to the maximum weight among the weights between the node of interest and a plurality of other nodes, sets a normalized vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest as the first axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the node of interest, sets the selected vector as the second axis vector, selects another node such that the absolute value of the dot product with the first axis vector becomes small and the absolute value of the dot product with the second axis vector becomes small and the weight becomes large, selects a vector corresponding to the relative position vector representing the difference between the position vector of the selected other node and the position vector of the node of interest, and sets the selected vector as the third axis vector.

17. The information processing method according to claim 12, wherein the neural network, when calculating a frame for each node, selects a plurality of other nodes in descending order of weight from the weights between the node of interest and a plurality of other nodes for each of the plurality of nodes, and calculates the frame represented by the one or more axis vectors by setting each of the vectors corresponding to the relative position vector representing the difference between the position vectors of the selected plurality of other nodes and the node of interest as one or more axis vectors.

18. An information processing device using a neural network for a structure represented as a set of nodes arranged in space, comprising: a receiving unit that receives input of the state of each node; a calculation unit that calculates a frame representing the coordinate axes at each node based on the states between each node; and an extraction unit that extracts information having a predetermined symmetry of the structure from the nodes using the frame.

Images