A self-supervised graph neural network pre-training method based on contrastive learning
Through comparative learning and self-supervised pre-training technology, the problem of insufficient generalization performance of graph neural networks in the lack of labeled data scenarios is solved, and a highly generalized model is built, which is suitable for a variety of downstream graph mining tasks.
Patent Information
- Application Number
- CN202110983302.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-08-25
AI Technical Summary
The existing graph neural networks have insufficient generalization performance in drug molecular data scenarios without labeling, and cannot effectively stimulate their learning ability.
Using a self-supervised graph neural network pre-training method based on contrast learning, the positive and negative samples are constructed by substructure decomposition of compound molecules, and the graph convolutional neural network is pre-trained, and modules such as attention mechanism and multi-level gating cycle units are used. , build a self-supervised learning model and use the InfoNCE loss function to optimize the generalization performance of the model.
It realizes learning general rules from large-scale label-free data, obtains a highly generalized self-supervised learning model, and can demonstrate fitting capabilities in different downstream tasks, avoiding training a brand new model for each task.
Smart Images

Figure CN114038517B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning, and more specifically, to a self-supervised graph neural network pre-training method based on contrastive learning. Background Art
[0002] Drug research and development is a multidisciplinary, long-term, and high-investment system engineering project. It faces problems such as high R&D costs, long cycles, and high failure rates. Therefore, people have begun to use artificial intelligence technology to assist in drug research and development. In recent years, graph neural networks, as an emerging technology in deep learning, have shown excellent performance on graph data. Compound molecules are a natural graph data, so they have opened up a new path for studying deep learning in assisting drug development.
[0003] Nowadays, graph neural networks based on supervised learning have achieved great success in the past few years. In order to learn powerful expression capabilities, they rely on a large amount of manually labeled graph data for optimization. Large-scale labeled graph data, especially labeled data based on the field of medicinal chemistry, is usually difficult to obtain, and the annotation of these data often requires expert knowledge in the field of biochemistry. In most cases, it is difficult for us to obtain a large amount of labeled data, so it is difficult for graph neural networks based on supervised learning to exert their powerful learning capabilities. How to use large-scale unlabeled molecular data for pre-training so that graph neural networks can learn potential features and information is a hot topic and difficulty in research.
[0004] In the existing technology, Chinese invention patent CN112862093A discloses "a graph neural network training method and device", which was published on May 28, 2021. The above method includes: obtaining an unlabeled graph dataset and a first labeled graph dataset; using the unlabeled graph dataset as a training sample, training a preset graph neural network model, adjusting the parameters of the graph neural network model, and obtaining a first graph neural network model; using the first labeled graph dataset as a training sample, training the first graph neural network model, adjusting the parameters of the first graph neural network model, and obtaining a second graph neural network model; using the second labeled graph dataset of the graph neural network's application scenario as a training sample, training the second graph neural network model, adjusting the parameters of the second graph neural network model, and obtaining a graph neural network applied to the application scenario; in this invention, the scheme improves the efficiency of graph neural network training when training the graph neural network, but its generalization performance is insufficient, and it can only be targeted at specific scenarios, and cannot stimulate the learning ability of the graph neural network. Summary of the invention
[0005] In order to solve the technical defect of insufficient generalization performance of existing graph neural network technology in model training for scenarios lacking labeled drug molecule data, the present invention provides a self-supervised graph neural network pre-training method based on contrastive learning.
[0006] In order to achieve the above invention purpose, the technical solution adopted is:
[0007] A self-supervised graph neural network pre-training method based on contrastive learning, comprising the following steps:
[0008] S1: Preprocess the compound molecules in the public database and screen out organic molecules;
[0009] S2: Decompose and extract the structure of the screened organic molecules, use the obtained substructures as identifiers, and build a corpus of substructures;
[0010] S3: Consider the decomposed substructure as a super node and construct the corresponding subgraph data. The subgraph data and the original molecular graph data constitute a positive sample pair. Ten subgraph data are randomly selected to form a negative sample pair with the original molecular graph data.
[0011] S4: Construct a graph convolutional neural network based on the attention mechanism, a multi-layer perceptron module based on multi-level gated recurrent units and transformation of full-image features to form a self-supervised learning model;
[0012] S5: Input all positive and negative sample pair data into the self-supervised learning model for pre-training and save.
[0013] In the above scheme, by performing substructure decomposition on unlabeled compound molecules to construct positive and negative samples for pre-training, the general rules of the data can be learned from large-scale unlabeled data, and the obtained self-supervised learning model has good generalization. When it is necessary to solve specific downstream tasks, the self-supervised learning model can be directly used for pre-training, thereby avoiding training a completely new model for each downstream task, and solving the problem of insufficient generalization performance caused by deep learning model training in scenarios with unlabeled drug molecules; a substructure-level pre-training strategy is proposed, so that the self-supervised learning model can capture the structural rules of molecular graph data, thereby giving it the ability to fit downstream graph mining tasks of unrestricted types.
[0014] Preferably, in step S1, the process of preprocessing the zinc compound molecules in the public database includes: downloading a number of compounds from the zinc compound database, first removing heavy elements, and then filtering monoatomic compounds and inert gases to obtain effective compound molecules.
[0015] In the above scheme, the database uses SMILES strings to represent molecules. First, elements with higher atomic numbers and larger relative atomic masses, i.e. heavy elements, are removed. Then, single atomic compounds and inert gases such as H2, C, O2, CO2, N2, etc. are filtered out. The obtained valid compound molecules are used for substructure decomposition and training corpus construction.
[0016] Preferably, in step S2, the process of performing substructure decomposition on the compound molecule and constructing a training corpus includes:
[0017] The effective compound molecules are sequentially converted from SMILES to molecular structure, decomposed into substructures, and deserialized into substructures;
[0018] According to the mapping relationship of the above hash table, the substructure of the original molecule is deserialized using Rdkit, that is, represented by the SMILES string of the substructure. The SMILES is used to construct the substructure corpus and saved in a local file.
[0019] Preferably, the conversion operation from SMILES to molecular structure is performed by converting through the Python tool library Rdkit and obtaining the atomic number, ring structure information, and functional group information corresponding to each atom;
[0020] The substructure decomposition operation uses a hash table to record the atom numbers corresponding to the ring structure information and functional group information. The remaining common atom numbers are also recorded in the hash table. The key of the hash table of each molecule is the new number of the subgraph, starting from 1. The corresponding values are the atom numbers corresponding to the ring structure of the original molecule, the atom numbers corresponding to the functional groups, and the common atom numbers.
[0021] In step S3, according to the original molecule, the hash table corresponding to the substructure is read, and the subgraph data is constructed according to the hash table. The substructure is regarded as a super node, and the above original molecule graph data and substructure graph data are used as positive sample pairs, and 10 subgraphs are randomly selected from all subgraph data to form negative sample pairs with the atomic graph.
[0022] Preferably, the number of nodes in the subgraph data is the maximum key value N of the hash table, and the subgraph is represented by an N×N adjacency matrix. The connection relationship of the subgraph is the same as the connection relationship of the substructure in the original molecular graph, 0 represents no edge connection, and 1 represents edge connection; according to the number of atoms R in the original molecule, it is represented by an R×R adjacency matrix, and the connection relationship is the same as the chemical bond connection relationship of the molecule, 0 represents no edge connection, and 1 represents edge connection; the ratio of positive and negative sample pairs is 1:10.
[0023] Preferably, the node features of the original molecular graph data include: atom type, charge number, free radical electron number, chirality type, hybrid orbital type, number of connected hydrogens, degree, aromaticity; except for the charge number and free radical electron number which are encoded as integers, each feature is represented by a one-hot encoding;
[0024] The subgraph data after substructure decomposition uses the embedding vector of the substructure SMILES string corresponding to each node as the feature of the node. The corpus composed of the substructure SMILES of all molecules, and its corresponding embedding vector is trained together with the model.
[0025] Preferably, in step S4, the graph convolutional neural network based on the attention mechanism includes a plurality of single-layer attention graph convolution modules connected in sequence, and the single-layer attention graph convolution module network includes a message passing submodule, an information aggregation submodule and an update submodule, the output end of the message passing submodule is electrically connected to the input end of the information aggregation submodule, and the output end of the information aggregation submodule is electrically connected to the input end of the update submodule;
[0026] The message passing submodule calculates the attention weights of the features of the N first-order neighbor nodes for any central node based on the adjacency matrix of the input graph data, and normalizes the weights; the information aggregation submodule performs weighted summation of the features of the N first-order neighbors according to the normalized attention weights obtained by the message passing module to obtain a new feature vector for subsequent central node feature updates; the update submodule performs an addition operation based on the new feature vector and the current central node feature vector, and outputs a new feature vector for the central node;
[0027] The process of constructing a multi-level gated recurrent unit includes: inputting the original molecular graph data and the substructure graph data into a multi-layer graph convolutional neural network based on the attention mechanism, performing feature averaging operations on all nodes of each graph data for the output of each convolution layer, and obtaining a global embedding representation of the graph data; obtaining the global embedding representation of the graph data corresponding to all layer blocks and inputting it into the gated recurrent unit, and outputting a multi-level graph data embedding vector representation;
[0028] The process of constructing a multi-layer perceptron module for transforming the whole-image features includes: inputting the embedded vector representation of the multi-level graph data output by the graph convolutional neural network based on the attention mechanism into the multi-layer perceptron module and outputting the embedded vector representation of the whole image.
[0029] Preferably, the graph convolutional neural network based on the attention mechanism further includes a residual module, which is configured to be connected behind each layer of the graph attention convolution module, specifically the first level normalization layer, the first activation layer, the first graph convolution layer, the residual layer, and the above layer blocks are repeated and connected in sequence;
[0030] The residual module is used to add the output graph data features and input graph data features after being processed by the multi-layer graph convolutional neural network to output a feature map for the input of the next layer of convolutional module.
[0031] Preferably, in step S4, the self-supervised learning model is trained using a back-propagation algorithm, and the objective function during training is based on the definition of maximizing mutual information. A lower bound of the mutual information InfoNCE is used as the loss function to estimate the distance between the embedding vectors of the positive and negative sample pairs, respectively. The distance is calculated based on the inner product formula of the vector, and the generalization performance of the model is optimized by minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
[0032] In the above scheme, the formula of InfoNCE loss function is as follows:
[0033]
[0034] Here, f refers to a graph neural network with trainable parameters, x refers to the original molecular graph data, also commonly referred to as the anchor data point, and x + Refers to data similar or equal to x (molecular substructure data), x j refers to the jth negative sample constructed, and N refers to the number of negative samples. The purpose of contrastive learning is to shorten the distance between similar samples and increase the distance between dissimilar samples. Here, the distance between positive and negative samples and anchor data points is measured by the inner product of the vector.
[0035] Minimizing the InfoNCE loss function is equivalent to maximizing the lower bound of the mutual information between positive samples and anchor data, so that the graph neural network can learn local node features and substructure features.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] The present invention provides a self-supervised graph neural network pre-training method based on contrastive learning. By performing substructure decomposition on unlabeled compound molecules to construct positive and negative samples for pre-training, the general rules of data can be learned from large-scale unlabeled data. The obtained self-supervised learning model has good generalization. When it is necessary to solve specific downstream tasks, the self-supervised learning model can be directly used for pre-training, thereby avoiding training a completely new model for each downstream task, and solving the problem of insufficient generalization performance caused by deep learning model training in scenarios with lack of labeled drug molecules; a substructure-level pre-training strategy is proposed, so that the self-supervised learning model can capture the structural rules of molecular graph data, thereby giving it the fitting ability for downstream graph mining tasks of unrestricted types. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a flow chart of the method of the present invention;
[0039] Figure 2 It is a flow chart of the self-supervised graph neural network pre-training method based on contrastive learning of the present invention;
[0040] Figure 3 A schematic diagram of a process of molecular pretreatment and molecular substructure decomposition in one embodiment of the present invention;
[0041] Figure 4 Schematic diagram of the structure of a single-layer graph attention convolution module in one embodiment of the present invention;
[0042] Figure 5 It is a schematic diagram of the structure of an attention convolutional neural network in one embodiment of the present invention;
[0043] Figure 6 A schematic diagram of the structure of a gated recurrent unit in one embodiment of the present invention;
[0044] Figure 7 A schematic diagram of the structure of a multi-level gated recurrent unit readout model in one embodiment of the present invention;
[0045] Figure 8 A schematic diagram of the structure of a multi-layer perceptron model in one embodiment of the present invention;
[0046] Fig. 9 This is a process for initializing node features of original molecular graph data and substructure subgraph data in one embodiment of the present invention. DETAILED DESCRIPTION
[0047] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0048] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0049] Example 1
[0050] like Figure 1 As shown, a self-supervised graph neural network pre-training method based on contrastive learning includes the following steps:
[0051] S1: Preprocess the compound molecules in the public database and screen out organic molecules;
[0052] S2: Decompose and extract the structure of the screened organic molecules, use the obtained substructures as identifiers, and build a corpus of substructures;
[0053] S3: Consider the decomposed substructure as a super node and construct the corresponding subgraph data. The subgraph data and the original molecular graph data constitute a positive sample pair. Ten subgraph data are randomly selected to form a negative sample pair with the original molecular graph data.
[0054] S4: Construct a graph convolutional neural network based on the attention mechanism, a multi-layer perceptron module based on multi-level gated recurrent units and transformation of full-image features to form a self-supervised learning model;
[0055] S5: All positive and negative sample pairs are input into the self-supervised learning model for pre-training and saved to facilitate fine-tuning of downstream tasks.
[0056] In the above scheme, by performing substructure decomposition on unlabeled compound molecules to construct positive and negative samples for pre-training, the general rules of the data can be learned from large-scale unlabeled data, and the obtained self-supervised learning model has good generalization. When it is necessary to solve specific downstream tasks, the self-supervised learning model can be directly used for pre-training, thereby avoiding training a completely new model for each downstream task, and solving the problem of insufficient generalization performance caused by deep learning model training in scenarios with unlabeled drug molecules; a substructure-level pre-training strategy is proposed, so that the self-supervised learning model can capture the structural rules of molecular graph data, thereby giving it the ability to fit downstream graph mining tasks of unrestricted types.
[0057] Preferably, in step S1, the process of preprocessing the zinc compound molecules in the public database includes: downloading a number of compounds from the zinc compound database, first removing heavy elements, and then filtering monoatomic compounds and inert gases to obtain effective compound molecules.
[0058] In the above scheme, the database uses SMILES strings to represent molecules. First, elements with higher atomic numbers and larger relative atomic masses, i.e. heavy elements, are removed. Then, single atomic compounds and inert gases such as H2, C, O2, CO2, N2, etc. are filtered out. The obtained valid compound molecules are used for substructure decomposition and training corpus construction.
[0059] Preferably, in step S2, the process of performing substructure decomposition on the compound molecule and constructing a training corpus includes:
[0060] The effective compound molecules are sequentially converted from SMILES to molecular structure, decomposed into substructures, and deserialized into substructures;
[0061] According to the mapping relationship of the above hash table, the substructure of the original molecule is deserialized using Rdkit, that is, represented by the SMILES string of the substructure. The SMILES is used to construct the substructure corpus and saved in a local file.
[0062] Preferably, the conversion operation from SMILES to molecular structure is performed by converting through the Python tool library Rdkit and obtaining the atomic number, ring structure information, and functional group information corresponding to each atom;
[0063] The substructure decomposition operation uses a hash table to record the atom numbers corresponding to the ring structure information and functional group information. The remaining common atom numbers are also recorded in the hash table. The key of the hash table of each molecule is the new number of the subgraph, starting from 1. The corresponding values are the atom numbers corresponding to the ring structure of the original molecule, the atom numbers corresponding to the functional groups, and the common atom numbers.
[0064] In step S3, according to the original molecule, the hash table corresponding to the substructure is read, and the subgraph data is constructed according to the hash table. The substructure is regarded as a super node, and the above original molecule graph data and substructure graph data are used as positive sample pairs, and 10 subgraphs are randomly selected from all subgraph data to form negative sample pairs with the atomic graph.
[0065] Preferably, the number of nodes in the subgraph data is the maximum key value N of the hash table, and the subgraph is represented by an N×N adjacency matrix. The connection relationship of the subgraph is the same as the connection relationship of the substructure in the original molecular graph, 0 represents no edge connection, and 1 represents edge connection; according to the number of atoms R in the original molecule, it is represented by an R×R adjacency matrix, and the connection relationship is the same as the chemical bond connection relationship of the molecule, 0 represents no edge connection, and 1 represents edge connection; the ratio of positive and negative sample pairs is 1:10.
[0066] Preferably, the node features of the original molecular graph data include: atom type, charge number, free radical electron number, chirality type, hybrid orbital type, number of connected hydrogens, degree, aromaticity; except for the charge number and free radical electron number which are encoded as integers, each feature is represented by a one-hot encoding;
[0067] The subgraph data after substructure decomposition uses the embedding vector of the substructure SMILES string corresponding to each node as the feature of the node. The corpus composed of the substructure SMILES of all molecules, and its corresponding embedding vector is trained together with the model.
[0068] Preferably, in step S4, the graph convolutional neural network based on the attention mechanism includes a plurality of single-layer attention graph convolution modules connected in sequence, and the single-layer attention graph convolution module network includes a message passing submodule, an information aggregation submodule and an update submodule, the output end of the message passing submodule is electrically connected to the input end of the information aggregation submodule, and the output end of the information aggregation submodule is electrically connected to the input end of the update submodule;
[0069] The message passing submodule calculates the attention weights of the features of the N first-order neighbor nodes for any central node based on the adjacency matrix of the input graph data, and normalizes the weights; the information aggregation submodule performs weighted summation of the features of the N first-order neighbors according to the normalized attention weights obtained by the message passing module to obtain a new feature vector for subsequent central node feature updates; the update submodule performs an addition operation based on the new feature vector and the current central node feature vector, and outputs a new feature vector for the central node;
[0070] The process of constructing a multi-level gated recurrent unit includes: inputting the original molecular graph data and the substructure graph data into a multi-layer graph convolutional neural network based on the attention mechanism, performing feature averaging operations on all nodes of each graph data for the output of each convolution layer, and obtaining a global embedding representation of the graph data; obtaining the global embedding representation of the graph data corresponding to all layer blocks and inputting it into the gated recurrent unit, and outputting a multi-level graph data embedding vector representation;
[0071] The process of constructing a multi-layer perceptron module for transforming the whole-image features includes: inputting the embedded vector representation of the multi-level graph data output by the graph convolutional neural network based on the attention mechanism into the multi-layer perceptron module and outputting the embedded vector representation of the whole image.
[0072] Preferably, the graph convolutional neural network based on the attention mechanism further includes a residual module, which is configured to be connected behind each layer of the graph attention convolution module, specifically the first level normalization layer, the first activation layer, the first graph convolution layer, the residual layer, and the above layer blocks are repeated and connected in sequence;
[0073] The residual module is used to add the output graph data features and input graph data features after being processed by the multi-layer graph convolutional neural network to output a feature map for the input of the next layer of convolutional module.
[0074] Preferably, in step S4, the self-supervised learning model is trained using a back-propagation algorithm, and the objective function during training is based on the definition of maximizing mutual information. A lower bound of the mutual information InfoNCE is used as the loss function to estimate the distance between the embedding vectors of the positive and negative sample pairs, respectively. The distance is calculated based on the inner product formula of the vector, and the generalization performance of the model is optimized by minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
[0075] In the above scheme, the formula of InfoNCE loss function is as follows:
[0076]
[0077] Here, f refers to a graph neural network with trainable parameters, x refers to the original data (the original molecular graph data, also commonly referred to as the anchor data point), and x + Refers to data similar or equal to x (molecular substructure data), x j refers to the jth negative sample constructed, and N refers to the number of negative samples. The purpose of contrastive learning is to shorten the distance between similar samples and increase the distance between dissimilar samples. Here, the distance between positive and negative samples and anchor data points is measured by the inner product of the vector.
[0078] Minimizing the InfoNCE loss function is equivalent to maximizing the lower bound of the mutual information between positive samples and anchor data, so that the graph neural network can learn local node features and substructure features.
[0079] Example 2
[0080] like Figure 3 As shown in the figure, the process of obtaining substructure subgraphs through substructure decomposition of molecules is as follows Figure 3 shown.
[0081] In this example, benzoic acid is used as an example to briefly illustrate the steps of substructure decomposition. Benzoic acid is represented by a SMIELS string (C1=CC=CC=C1C(=O)O) in the ZINC database and converted into the format of a molecular structure by the Python open source chemical computing toolkit RdKit. First, the atomic numbers corresponding to all rings and functional groups in the molecule are obtained through Rdkit as overall super nodes and stored in a hash table. Similarly, all bonds in the molecule are treated as overall super nodes and stored in a hash table. Then, the connection relationship between the super nodes is recorded, and nodes where more than three super nodes intersect are agreed to be intermediate nodes and stored in a hash table, and connected edges are added at the same time. Furthermore, the SMIELS string corresponding to the super node, that is, the substructure SMILES string, is recorded during the decomposition process.
[0082] For example, Figure 3 The substructure decomposition process of benzoic acid is to treat the benzene ring group as a supernode and record its corresponding substructure SMILES (CCCCCC). The same operation is performed on other supernodes, and the supernodes after substructure decomposition are connected according to the connection relationship of the original molecule. Finally, the corresponding substructure diagram of benzoic acid is obtained.
[0083] The above operations are performed on all compound molecules obtained by ZINC data screening, and all substructure graph data and all non-repeating substructure SMILES are recorded and saved locally for subsequent acquisition of word embedding vectors.
[0084] Example 3
[0085] like Figure 2As shown, in order to construct positive and negative sample pairs, corresponding self-supervised training tasks are designed. In the preprocessing and substructure decomposition steps, the substructure graph data and the original molecular graph data constitute a positive sample pair, and accordingly, ten subgraph data are randomly selected from the substructure graph data set and the original molecular graph data to form a negative sample pair, satisfying the positive and negative sample ratio of 1 to 10. Specifically, one original molecular graph data and its corresponding substructure graph data constitute a positive sample pair, and 10 substructure graph data are randomly selected from the original molecular graph data to constitute 10 negative sample pairs for the input of the subsequent graph attention convolutional neural network.
[0086] Example 4
[0087] like Figure 4 As shown, in this example, the single-layer graph convolution network module based on the attention mechanism consists of a message passing module, an information aggregation module, and an update module. The message passing module calculates the attention weights for the features of any central node, assuming that it has N first-order neighbors, and normalizes the weights according to the adjacency matrix of the input graph data. The information aggregation module performs weighted summation of the features of the N first-order neighbors according to the normalized attention weights obtained by the message passing module to obtain a new feature vector for subsequent updates of the central node features. The update module performs an addition operation based on the new feature vector and the current central node feature vector, and outputs a new feature vector for the central node. The above three constitute a complete single-layer attention graph convolution module, which is used to output updated graph data.
[0088] Accordingly, if Figure 4 As shown in the figure, the calculation formula corresponding to the single-layer graph convolutional network module based on the attention mechanism is:
[0089]
[0090]
[0091]
[0092]
[0093] Here N i It refers to all first-order neighbor nodes of graph node i. The message passing module is shown in Formula 1. For any node pair, neighbor node j and center node i, firstly, the node feature h is learned by the learnable parameter W. i and h jThe diversity of features is increased through linear transformation, and after concatenation, the attention score of neighbor node j relative to the center node i is output through a learnable linear transformation. Similarly, all neighbor nodes of the center node i perform the above operation to obtain the attention scores of multiple neighbor nodes j relative to the center node i. The information aggregation module is shown in formulas 2 and 3. All attention scores are activated by the LeakyReLU function and normalized, that is, the sum of all attention scores is 1, and α is obtained. ij That is, the attention weight of neighbor node j relative to the central node i, and then all neighbor nodes are weighted summed according to the attention weight, and the message feature h′ to be updated is output i The update module is shown in Formula 4. It updates the input feature h of the central node i and the message feature h′ to be updated i Perform sum operation and output the new features of the central node i As the input of the single-layer graph convolutional network module based on the attention mechanism, the next round of update iteration is carried out.
[0094] Similarly, all nodes in the graph can be identified as central nodes, so all node features on the graph data can be updated and used for the next round of update iterations.
[0095] Example 5
[0096] like Figure 5 As shown, in this example, the above-mentioned multiple single-layer attention graph convolution modules are connected in sequence to form the multi-layer graph convolution neural network based on the attention mechanism. Exemplarily, the multi-layer graph convolution neural network based on the attention mechanism contains 3 single-layer attention convolution modules, and for any central node of the graph data input to the network, the structural information on the graph data is captured by aggregating the features of its third-order neighbor nodes and updating its own features.
[0097] Furthermore, if Figure 5 As shown, the residual module, that is, the addition operation, is connected after each layer of the graph attention convolution module, specifically the first-level normalization layer, the first activation layer, the first graph convolution layer, the residual layer, repeating the above layer blocks and connecting them in sequence, for adding the output graph data features and the input graph data features after being processed by the multi-layer graph convolutional neural network to output the feature map for the input of the next layer of convolution module.
[0098] Specifically, Figure 5As shown in the figure, the convolution operation process of the multi-layer graph convolutional neural network based on the attention mechanism is to repeatedly perform the BN-ReLU activation function-convolution operation, so that the single-layer attention graph convolution module can first perform normalization processing before receiving the output graph data of the previous layer, so that the feature graph data distribution is pulled back to the approximate normal distribution, and the input data falls at a position where the nonlinear activation function is relatively sensitive to the input, and then the convolution feature extraction operation is performed, so it has better performance. In the network model, for the shallow layer l and the deep layer L, according to the above residual module, that is, the addition operation, the forward propagation calculation can be obtained through the recursive relationship as follows:
[0099]
[0100] where x l and x L are the inputs of the lth and Lth layers respectively, F(x i ,ω i ) is the residual function calculation of the residual branch, x i is the input of the i-th layer, ω i is the convolution operation on the residual branch of the i-th layer. Assuming the loss function is loss, the back propagation formula is obtained by derivation of the loss function:
[0101]
[0102] From the above, we can get the loss function loss for the shallow layer x l The gradient calculation of is determined by two terms: the former is the loss function for the deep layer x L The latter is a value that is not easily offset, which effectively avoids the problem of gradient vanishing when the weight is very small and speeds up the convergence of the model. This also shows that in the fine-tuning module, the result of adding the identity branch and the residual branch is kept "clean", which becomes a direct channel for information transmission and can accelerate the forward and backward propagation of the model structure.
[0103] Example 6
[0104] See also Figure 6 and Figure 7 , a schematic diagram of the structure of the single gated recurrent unit and a schematic diagram of the structure of the expanded multi-level gated recurrent unit readout model.
[0105] The original molecular graph data and substructure graph data are input into the multi-layer graph convolutional neural network based on the attention mechanism. For the output of each convolution layer, the feature average operation is performed on all nodes of each graph data to obtain the global embedding representation of the graph data. The global embedding representation of the graph data corresponding to all layer blocks is obtained and input into the gated recurrent unit, and the output is a multi-level graph data embedding vector representation.
[0106] For example, Figure 2 As shown in the structural diagram, the feature embedding vector of the whole graph is obtained by averaging the feature outputs of the three layers of nodes in the multi-layer graph convolutional neural network, and is represented by symbols g and 1 , g 2 , g 3 Indicates that the hierarchical hidden information is represented by G n , which represents the hierarchical information of the previous n-1 layers implied by the input to the i-th layer, and is used to update the hierarchical information of the current i-th layer. The calculation formula of a single gated recurrent unit is as follows:
[0107] u n =σ(W u g n +Y u G n ) (1)
[0108] r n =σ(W r g n +Y r G n ) (2)
[0109]
[0110]
[0111] Where W u and Y u represents the linear transformation parameter of the update gate, W r and Y r represents the linear transformation parameter of the reset gate, W g and Y g Represents the linear transformation parameters of the hidden state to be updated, and ☉ represents the multiplication of the corresponding elements of the matrix.
[0112] For example, the feature embedding vector g of the whole graph corresponding to the nth layer in the multi-layer graph convolutional neural network is n , the first n-1 layers of the layer hidden information G n , calculate the update coefficient u of the current layer's hierarchical hidden information according to formula 1 n , calculate the forgetting coefficient r of the current layer's hidden information according to formula 2 n , when r n Close to 0, resetting the gate will forget most of the first n-1 layers of information. n and r n After the coefficient, the hidden state of the layer to be updated is calculated according to Formula 3 Used for the calculation of new hierarchical hidden information.
[0113] Furthermore, Formula 4 calculates the hierarchical latent information G of the first n-1 layersn and the first n-1 layers of the hierarchical hidden information G n To output the hierarchical hidden state information of the first n layers.
[0114] Furthermore, the full-graph features of the last layer of the multi-layer graph convolutional neural network are input into the multi-level gated recurrent unit readout model, and finally a multi-level full-graph representation of the graph is obtained. This representation implies multi-level structural information, and the structural information implied by each single-layer graph convolution module is adaptively calculated through learning. The auxiliary model learns structured feature representations for multi-level graph data, which in turn enhances the representation ability of the graph embedding vector and improves the performance of the model.
[0115] Example 7
[0116] like Figure 8 and Fig. 9 As shown, the structure diagram of the multi-layer perceptron model is shown;
[0117] Exemplarily, the multi-level global representation output of the graph after the multi-level gated recurrent unit reads out the model is linearly transformed by a two-layer perceptron to obtain an updated multi-level full graph representation.
[0118] Furthermore, in this embodiment, the graph convolutional neural network module based on the attention mechanism, based on the multi-level gated recurrent unit module, and the multi-layer perceptron module are connected in sequence to constitute a self-supervised learning model based on the graph convolutional neural network; the estimation model is trained using the back propagation algorithm, and the objective function during training is based on the definition of maximizing mutual information. InfoNCE is used as the loss function to estimate the distance between the embedded vectors of the positive and negative sample pairs respectively, and the distance is calculated based on the inner product formula of the vector. The generalization performance of the model is optimized by minimizing the distance of the positive sample pairs and maximizing the distance of the negative sample pairs, so that the model can fully learn the substructure information of the molecular graph data.
[0119] Among them, the formula of InfoNCE loss function is as follows:
[0120]
[0121] Here, f refers to a graph neural network with trainable parameters, x refers to the original data (the original molecular graph data, also commonly referred to as the anchor data point), and x + Refers to data similar or equal to x (molecular substructure data), x j refers to the jth negative sample constructed, and N refers to the number of negative samples. The purpose of contrastive learning is to shorten the distance between similar samples and increase the distance between dissimilar samples. Here, the distance between positive and negative samples and anchor data points is measured by the inner product of the vector.
[0122] Minimizing the InfoNCE loss function is equivalent to maximizing the lower bound of the mutual information between positive samples and anchor data, so that the graph neural network can learn local node features and substructure features.
[0123] The present application discloses a self-supervised graph neural network pre-training method based on contrastive learning. During training, the ZINC data set of 10 million compound molecules is used, and data pre-processing operations are performed, including filtering heavy elements, inert gases and other non-organic molecules. Then, the original molecules are decomposed into substructures to construct positive and negative sample pairs. All constructed positive and negative sample pairs are input into the model, and the model is trained using the back-propagation algorithm and the aforementioned objective function.
[0124] Before the positive and negative sample graph data are input into the model, the node features of each graph data need to be initialized. The features used are shown in Fig. 9 .
[0125] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A self-supervised graph neural network pre-training method based on contrastive learning, characterized in that: The following steps are involved: S1: Preprocess the compound molecules in the public database and screen out organic molecules; S2: Decompose and extract the structure of the screened organic molecules, use the obtained substructures as identifiers, and build a corpus of substructures; S3: Consider the decomposed substructure as a super node and construct the corresponding subgraph data. The subgraph data and the original molecular graph data constitute a positive sample pair. Randomly select a number of subgraph data and the original molecular graph data to form a negative sample pair. S4: constructing a graph convolutional neural network based on an attention mechanism, a multi-layer perceptron module based on a multi-level gated recurrent unit and a transformation for full-image features, and forming a self-supervised learning model; in step S4, the graph convolutional neural network based on the attention mechanism includes a plurality of single-layer attention graph convolution modules connected in sequence, and the single-layer attention graph convolution module network includes a message passing submodule, an information aggregation submodule and an update submodule, and the output end of the message passing submodule is electrically connected to the input end of the information aggregation submodule, and the output end of the information aggregation submodule is electrically connected to the input end of the update submodule; The message passing submodule calculates the attention weights of the features of any central node, assuming that it has N first-order neighbor nodes, and normalizes the weights according to the adjacency matrix of the input graph data; the information aggregation submodule performs weighted summation of the features of the N first-order neighbor nodes according to the normalized attention weights obtained by the message passing submodule to obtain a new feature vector for subsequent updates of the central node features; The update submodule performs an addition operation based on the new feature vector and the feature vector of the current central node, and outputs a new feature vector of the central node; The process of constructing a multi-level gated recurrent unit includes: inputting the original molecular graph data and sub-graph data into a multi-layer graph convolutional neural network based on an attention mechanism, performing a feature averaging operation on all nodes of each graph data for the output of each convolutional layer, and obtaining a global embedding representation of the graph data; obtaining the global embedding representation of the graph data corresponding to all layer blocks and inputting it into the gated recurrent unit, and outputting a multi-level graph data embedding vector representation; The process of constructing a multi-layer perceptron module for transforming the whole-image features includes: inputting the embedded vector representation of the multi-level graph data output by the graph convolutional neural network based on the attention mechanism into the multi-layer perceptron module and outputting the embedded vector representation of the whole image; S5: Input all positive and negative sample pair data into the self-supervised learning model for pre-training and save.
2. According to claim 1, a self-supervised graph neural network pre-training method based on contrastive learning is characterized in that: In step S1, the process of preprocessing the compound molecules in the public database includes: downloading a number of compounds from the public compound database zinc, first removing heavy elements, and then filtering monoatomic compounds and inert gases to obtain valid compound molecules.
3. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 2, characterized in that: In step S2, the process of structurally decomposing organic molecules and constructing a corpus of substructures includes: The effective compound molecules are sequentially converted from SMILES to molecular structure, decomposed into substructures, and deserialized into substructures; According to the mapping relationship of the hash table, the substructure of the original molecule is deserialized using Rdkit, that is, represented by the SMILES string of the substructure. The SMILES is used to construct the substructure corpus and saved in a local file.
4. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 3, characterized in that: The conversion operation from SMILES to molecules is performed through the Python tool library Rdkit to obtain the atom number, ring structure information, and functional group information corresponding to each original molecule; The substructure decomposition operation uses a hash table to record the atom numbers corresponding to the ring structure information and functional group information. The remaining common atom numbers are also recorded in the hash table. The key of the hash table of each molecule is the new number of the subgraph, starting from 1. The corresponding values are the atom numbers corresponding to the ring structure of the original molecule, the atom numbers corresponding to the functional groups, and the common atom numbers.
5. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 4, characterized in that: In step S3, according to the original molecule, the hash table corresponding to the substructure is read, and the subgraph data is constructed according to the hash table. The substructure is regarded as a super node, and the original molecule graph data and the subgraph data are used as positive sample pairs. Ten subgraphs are randomly selected from all the subgraph data to form negative sample pairs with the original molecule graph.
6. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 5, characterized in that: The number of nodes in the subgraph data is the maximum key value N of the hash table, and the subgraph is represented by an N×N adjacency matrix. The connection relationship of the subgraph is the same as the connection relationship of the substructure in the original molecular graph, 0 represents no edge connection, and 1 represents edge connection; according to the number of atoms R in the original molecule, it is represented by an R×R adjacency matrix, and the connection relationship is the same as the chemical bond connection relationship of the molecule, 0 represents no edge connection, and 1 represents edge connection; the ratio of positive and negative sample pairs is 1:
10.
7. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 6, characterized in that: The node features of the original molecular graph data include: atom type, charge number, number of free radical electrons, chirality type, hybrid orbital type, number of connected hydrogens, degree, and aromaticity; except for the charge number and the number of free radical electrons, which are encoded as integers, each feature is represented by a one-hot encoding; The subgraph data after substructure decomposition uses the embedding vector of the substructure SMILES string corresponding to each node as the feature of the node. The corpus composed of the substructure SMILES of all molecules, and its corresponding embedding vector is trained together with the model.
8. The self-supervised graph neural network pre-training method based on contrastive learning according to claim 7, characterized in that: The graph convolutional neural network based on the attention mechanism also includes a residual module, which is set to be connected behind each layer of the graph attention convolution module, specifically the first level normalization layer, the first activation layer, the first graph convolution layer, the residual layer, and the above layer blocks are repeated and connected in sequence; The residual module is used to add the output graph data features and input graph data features after being processed by the multi-layer graph convolutional neural network to output a feature map for the input of the next layer of convolutional module.
9. The self-supervised graph neural network pre-training method based on contrastive learning as claimed in claim 8, characterized in that: In step S4, the self-supervised learning model is trained using a back-propagation algorithm. The objective function during training is based on the definition of maximizing mutual information. A lower bound of the mutual information, InfoNCE, is used as the loss function to estimate the distance between the embedding vectors of the positive and negative sample pairs, respectively. The distance is calculated based on the inner product formula of the vector. The generalization performance of the model is optimized by minimizing the distance between the positive sample pairs and maximizing the distance between the negative sample pairs.
