Multi-level organic chemical reaction prediction method based on double encoders

Abstract

The invention provides a multi-level organic chemistry reaction prediction method based on double encoders, and belongs to the technical field of machine learning assisted organic chemistry. The technical problems that a Transform model lacks attention to intermolecular interaction, coding is insufficient, and generalization ability is not high are solved. According to the technical scheme, the method comprises the following steps: S10, preprocessing a chemical reaction SMILES sequence data set; s20, performing feature extraction on the input chemical reaction by using an atom encoder and a molecular encoder; s30, multi-level feature fusion is applied to atomic and molecular feature representation; s40, weight addition is carried out on the model prediction result, the loss obtained by the target and the loss obtained by comparative learning, and the model is trained; and S50, predicting a chemical product by using the trained model. The method has the advantages that the accuracy of chemical product prediction is improved, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning-assisted organic chemistry, and in particular to a prediction method for multi-level organic chemical reactions based on dual encoders. Background Art

[0002] In recent years, large language modeling such as Transformer has demonstrated excellent performance in the field of organic chemistry prediction. Chemists can use the powerful modeling capabilities of machine learning to predict and discover new types of organic chemical reactions. Compared with traditional recurrent neural networks (RNN) and long short-term memory networks (LSTM), the Transformer model can better capture long-distance dependencies through the self-attention mechanism, and has stronger feature representation and parallel computing capabilities. Although large language models have efficient data processing capabilities and can improve the accuracy of chemical reaction predictions, their performance is highly dependent on the quality and quantity of training data. Therefore, when computing resources and chemical data sets are limited, extracting deeper chemical information to improve the accuracy of model predictions is an important research and improvement direction.

[0003] At present, in the organic chemical reaction prediction method, the chemical formula represented by SMILES is segmented according to atoms, and then the Transformer model is used to obtain the attention weights between atoms, ignoring the interaction between molecules. For example, the model proposed in "Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction" is completely based on the attention mechanism, and the text representation of reactants, reagents and products is used for machine translation problems. However, the model does not pay attention to the information between molecules, which leads to inaccurate prediction of organic chemical products. The present invention uses molecular encoders and atomic encoders to extract the interaction information between atoms and molecules respectively, and performs multi-level feature fusion to extract deeper features. In addition, "Is bert really robust? A strong baseline for natural language attack on text classification and entailment" proposes that the natural language processing model based on machine learning is susceptible to small and often imperceptible interference. Therefore, avoiding the model from being affected by these synonyms and non-standard data and optimizing the extraction of different feature information at the same time are the issues faced by the present invention. Summary of the invention

[0004] The purpose of the present invention is to provide a prediction method for multi-level organic chemical reactions based on dual encoders. The method uses two encoders to construct representations of atoms and molecules in an input chemical formula. The atomic encoder obtains the relationship between atoms through a self-attention mechanism, and the molecular encoder averages the atomic embedding information to obtain the representation of the molecular embedding information, and then obtains the interaction between molecules through the attention mechanism. Then, a multi-level feature fusion module is used to fuse the output results of the atomic encoder and the molecular encoder, and the result is input to the decoder. Then, the prediction result of the model is dynamically weighted and fused with the loss of the target result and the loss of contrastive learning. This process improves the accuracy of the model in predicting organic chemical products and enhances the generalization ability of the model. The present invention will provide an important reference for organic chemists in experimental design and synthetic path planning.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a prediction method of multi-level organic chemical reactions based on dual encoders, comprising the following steps:

[0006] S10, read the organic chemical reaction SMILES training, validation and test datasets in sequence, and process the reactants and reagents as source sentences x = {x1,...,x i ,...,x n} as the target sentence y={y1,...,y i ,...,y n} SMILES expression. Then, regular expressions are used to process and mark the data, and more data samples are obtained through data enhancement.

[0007] S20, using the atomic encoder and the molecular encoder to respectively encode the input source sentence x = {x1, ..., x i ,...,x n} for feature extraction, first input into the atom encoder to convert into the corresponding ID according to the source dictionary, and then mapped into X = {X1,...,X i ,...,X n}, the scaled X plus the position information code E pos become Where pos indicates the position, followed by After L cycles of multi-head attention mechanism, the correlation matrix X between atoms is obtained L , molecular encoder according to the appendix of the manual Figure 2 The embedding codes of the reactants and reagents of each chemical reaction are averaged to obtain the molecular embedding matrix X′, which is then scaled and added with the new position encoding matrix to become After L times of multi-head attention mechanism, the correlation matrix X′ between molecules is obtained L ;

[0008] S30, correlation matrix X between atoms L and the correlation matrix X′ between molecules L The last dimension is concatenated, and the weight matrix λ is obtained through the linear layer and the Softmax activation function. The weight matrix is ​​then used for weighted fusion to obtain the multi-level feature fusion matrix X″, which is then input into the decoder.

[0009] S40, target sentence y={y1,...,y i ,...,y n} is input to the decoder, and the Teacher Forcing technology and Mask mechanism are used to make the decoder process the embedding of the entire sentence at the same time during the training phase, predict each molecular token in parallel until the "end of sentence marker" is generated or the maximum length allowed for prediction is reached, and a probability distribution is generated in the entire vocabulary space. The predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss X loss , and the loss of contrastive learning Add weights and back-propagate the weights of the optimized model;

[0010] S50. Use the trained dual encoder model to predict products for new organic chemical reactions.

[0011] As the prediction method for multi-level organic chemical reactions of dual encoders provided by the present invention, the specific steps of step S10 are as follows:

[0012] S11. Read the organic chemical reaction SMILES dataset, retain the reactions without stereochemical information and atomic mapping, and then follow the standard neural machine translation steps to use regular expressions to process and mark the data. For the normalized dataset, use the regular expression Regex to perform subword segmentation on the normalized sequence. The definition of Regex is as follows:

[0013] Regex="(\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\(|\)|\.|=|#|-|\+|\\\\\ / |:|~|@|\?|>|\*|\$|\%[0-9]{2}|[0-9])"

[0014] For the original data set, RDkit was used to convert the SMILES string into a molecule object. Then, all atoms in the molecule object were traversed. If there was a molAtomMapNumber attribute, the Python code atom.ClearProp("molAtom-MapNumber") was used to remove these mapping numbers to obtain SMILES characters without atom mapping. Finally, the regular expression Regex was used for subword segmentation. The reactants and reagents were combined into the source string, and the two reactants were separated by "." and the reactants and reagents were separated by ". <separated>"Separation;

[0015] S12. Perform data enhancement on the training, test, and validation data sets respectively. During the enhancement process, filter out the non-standardized and repeated data to obtain more data samples.

[0016] As the prediction method for multi-level organic chemical reactions of dual encoders provided by the present invention, the specific steps of step S20 are as follows:

[0017] S21, convert the processed source sentence and target sentence into corresponding IDs according to the source dictionary and target dictionary respectively, and then project them to D dimension through the Embedding layer of the dual encoder to obtain the source embedding X = {X1,...,X i ,...,X n } and target embedding Y = {Y1,...,Y i ,...,Y n }, then scale and add position information code E pos , the final atomic embedding calculation formula is as follows:

[0018]

[0019]

[0020] in To process the source embedding, is the processed target embedding, a is the scaling constant;

[0021] S22. For the Transformer model with dual encoders, Input to the atomic encoder, pass through the multi-head self-attention module, and initialize the weight matrix W through attention Q , W K , W V , and dynamically adjust the weight matrix so that the weight reflects the importance of the sample, and finally obtain the Q, K, V matrices as follows:

[0022]

[0023]

[0024]

[0025] Q is the query matrix, K and V are the key and value matrices respectively, b Q , b k , b v is the bias vector, these parameters are learned by the model during training;

[0026] S23, finally get the attention matrix A j :

[0027]

[0028] where d k Represents the dimension of the key matrix. In the self-attention, the Q, K, and V matrices are linearly changed many times, and then the attention function is executed in parallel. They are connected and projected again to get the final result. The core of multi-head attention is scale dot product attention. Then the multi-head attention is connected in series through dot product attention. The calculation formula is as follows:

[0029]

[0030] Then A j After layer normalization, i.e. LN(A j ), the calculation formula is as follows:

[0031]

[0032] LN(A j ) passes through the Dropout layer to prevent the model from overfitting, and then passes through the feedforward network, which contains two linear transformations with a ReLU activation in the middle, followed by layer normalization and Dropout layers. Finally, steps S22 to S23 are repeated L times to obtain the final atomic correlation matrix X L ;

[0033] S24. The processed source sentences are converted into corresponding IDs according to the source dictionary. The model obtains the output of the corresponding embedding layer according to the ID of each chemical formula, and then Figure 2 , the embedding codes of reactants and reagents in each chemical reaction are averaged, and the dot and special symbols are retained to obtain the molecular embedding representation X′={X′1,...,X′ i ,...,X′ n }, at the same time, recalculate the filling matrix and position encoding matrix, then scale and add the new position encoding E′ pos , the final molecular embedding calculation formula is as follows:

[0034]

[0035] S25, Molecular Embedding The filled matrix is ​​input into the molecular encoder, and steps S22 to S23 are repeated L times to obtain the correlation matrix X′ between molecules L ;

[0036] As the prediction method for multi-level organic chemical reactions of dual encoders provided by the present invention, the specific steps of step S30 are as follows:

[0037] S31, correlation matrix X between atoms L and the correlation matrix X′ between molecules L The last dimension is concatenated, and then the weight matrix λ is obtained through the linear layer and the Softmax activation function. The weight matrix is ​​then used for weighted fusion to obtain the multi-level feature fusion matrix X″, which is then input into the decoder. The relevant calculation formula is as follows:

[0038] L = W·concat(X L ,X′ L ) (10)

[0039] λ=softmax(L,dim=1) (11)

[0040] X″=λX L +(1-λ)X′ L (12)

[0041] Where W is the weight matrix of the linear layer;

[0042] As the prediction method for multi-level organic chemical reactions of dual encoders provided by the present invention, the specific steps of step S40 are as follows:

[0043] S41. The decoder predicts the corresponding target symbol sequence one element at a time. Like the encoder, the decoder also consists of a stack of L identical layers. A third sublayer is inserted, which performs multi-head attention on the output of the encoder stack, using the residual connections around each sublayer, followed by layer normalization. The model's ability to translate word for word from source sentence x to target sentence y is calculated as follows:

[0044]

[0045] Where w(·) is a linear function that outputs the previously generated word y i The probability of i is the i-th source representation, and the i-th decoded hidden state s i Calculated as:

[0046] s i =f(s i-1 ,y i-1 ,c i )(14)

[0047] S42, the predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss, and the calculation formula is as follows:

[0048]

[0049] Among them, L n is the negative log-likelihood of calculating the correct class label, L s It means to calculate the average of negative log probability of all categories. represents the predicted probability of the i-th sample in the true category, M represents the number of samples. In order to prevent the model from overfitting when there are too few training samples, label smoothing is performed on the basis of the multi-classification cross entropy loss function. After adding label smoothing, the true probability distribution becomes ε:

[0050]

[0051] Among them, η is a small hyperparameter, K represents the total number of multi-classification categories, j represents the target label, and the label-smoothed multi-classification cross entropy loss X loss as follows:

[0052] X loss =(1-η-ε)L n +εL s (18)

[0053] S43, N represents the number of samples in a batch. By enhancing the data n times, we get n pairs of positive sample pairs. At this time, nN samples are shared. The contrast loss first selects all positive classes of each sample in the batch so that the embedding similarity between the positive classes of each sample is high. Secondly, the negative classes of all standard samples in the batch are selected so that the embedding similarity between the negative classes of the standard samples is low. The contrast loss Using InfoNCE loss, the calculation formula is as follows:

[0054]

[0055] Among them, z i is the normalized sample of the ith positive class in a batch, z j is the non-normalized sample of the ith positive class in a batch enhanced by n times, sim(·) is the cosine similarity, τ is the temperature parameter, and z k is a normalized sample among the other (nN-1) samples in a batch;

[0056] S44, multi-label smoothed cross entropy loss function to obtain loss X loss Loss with Contrastive Learning Add the weights and back propagate the weights of the optimization model. The calculation formula is as follows:

[0057]

[0058]

[0059] Among them, E α is the current Epoch, E total is the total epoch set, L z is the final loss function;

[0060] As a prediction method for multi-level organic chemical reactions using dual encoders provided by the present invention, in step S50: the dual encoder model is trained based on steps S10 to S40, and the trained dual encoder model is used to predict products of organic chemical reactions.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. Deeper feature extraction: Traditional organic chemical product prediction methods ignore the interactions between molecules. The dual encoder model uses an atom encoder and a word encoder to extract molecular and atomic information respectively. Based on a simple and effective molecular extraction algorithm, the model focuses on both atomic and molecular information.

[0063] 2. Multi-level feature fusion: This paper proposes a fusion layer with automatic weight update, which uses linear splicing to fuse the outputs of atomic and molecular encoders, batch normalization and Softmax activation functions to obtain weight parameters, and then enhances the complementary information between features at different levels based on the fusion of gating units. Unlike simple connection, this design further extracts complementary information.

[0064] 3. Enhance the generalization ability of the model: The dual encoder model adopts an optimized multi-classification cross entropy loss function. On the basis of the smooth cross entropy loss function, contrastive learning loss is added. The two loss functions are weightedly fused by automatically updating the weight parameters. This method can be applied not only to normalized SMILES chemical datasets, but also to non-normalized SMILES datasets, which improves the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention.

[0066] Figure 1 The figure is an overall flow chart of the dual encoder multi-level method for predicting organic chemical products in the present invention.

[0067] Figure 2 The flowchart of the algorithm for molecular feature representation of the present invention.

[0068] Figure 3 This is a model framework diagram of the dual encoder multi-level method for predicting organic chemical products of the present invention.

[0069] Figure 4 An exemplary diagram of the dual encoder multi-level method for organic chemical product prediction of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention;

[0071] Example 1

[0072] See also Figure 1 As shown, the technical solution provided in this embodiment is a prediction method for multi-level organic chemical reactions of dual encoders, comprising the following steps:

[0073] S10, read the organic chemical reaction SMILES training, validation and test datasets in sequence, and process the reactants and reagents as source sentences x = {x1,...,x i ,...,x n } as the target sentence y={y1,...,y i ,...,y n } SMILES expression. Then, regular expressions are used to process and mark the data, and more data samples are obtained through data enhancement.

[0074] S20, using the atomic encoder and the molecular encoder to respectively encode the input source sentence x = {x1, ..., x i ,...,x n } for feature extraction, first input into the atom encoder to convert into the corresponding ID according to the source dictionary, and then mapped into X = {X1,...,X i ,...,X n }, the scaled X plus the position information code E pos become Where pos indicates the position, followed by After L cycles of multi-head attention mechanism, L is 6, the correlation matrix X between atoms is obtained. L , molecular encoder according to the appendix of the manual Figure 2 The embedding codes of the reactants and reagents of each chemical reaction are averaged to obtain the molecular embedding matrix X′, which is then scaled and added with the new position encoding matrix to become After L times of multi-head attention mechanism, the correlation matrix X′ between molecules is obtained L ;

[0075] S30, correlation matrix X between atoms L and the correlation matrix X′ between molecules L The last dimension is concatenated, and the weight matrix λ is obtained through the linear layer and the Softmax activation function. The weight matrix is ​​then used for weighted fusion to obtain the multi-level feature fusion matrix X″, which is then input into the decoder.

[0076] S40, target sentence y={y1,...,y i ,...,y n } is input to the decoder, and the Teacher Forcing technique and Mask mechanism are used to make the decoder process the embedding of the entire sentence at the same time during the training phase, predict each molecular token in parallel until the "end of sentence marker" is generated or the maximum length allowed for prediction is reached, and generate a probability distribution in the entire vocabulary space. The predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss X loss , and the loss of contrastive learning Add weights and back-propagate the weights of the optimized model;

[0077] S50. Use the trained dual encoder model to predict products for new organic chemical reactions.

[0078] The specific steps of step S10 are as follows:

[0079] S11. The original data set used for USPTO-Jin training, validation, and testing in this example comes from a public chemical reaction data set mined from U.S. patent authorizations from 1976 to September 2016. A total of 479,035 reactions without stereochemical information and atomic mapping are retained. First, the chemical reaction SMILES string in the training set is read, and the source data set and the target data set are divided according to ">>". Then, the SMILES string in the source data set is converted into a molecular object using RDkit. Then, all atoms in the molecular object are traversed. If there is a molAtomMapNumber attribute, the Python code atom.ClearPr-op("molAtomMapNumber") is used to remove these mapping numbers to obtain SMILES characters without atomic mapping. Finally, the regular expression Regex is used for subword segmentation. The reactants and reagents are combined into the source string. The reactants are separated by ". " and the reactants and reagents are separated by ". <separated>"Separated by , Regex is defined as follows:

[0080] Regex="(\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\(|\)|\.|=|#|-|\+|\\\\\ / |:|~|@|\?|>|\*|\$|\%[0-9]{2}|[0-9])"

[0081] Because the original USPTO-Jin dataset contains only single products, after removing the atom mapping number, regular expression Regex is used for subword segmentation to obtain the final target dataset. The data processing methods of the validation set and the test set are the same.

[0082] S12. Root alignment data augmentation is performed on the original datasets of USPTO-Jin training, validation, and testing, respectively. During the augmentation process, the data that cannot be standardized and repeated are filtered out. An example of root alignment data augmentation is shown in Table 1.

[0083] Table 1 Root alignment data augmentation steps

[0084]

[0085] The root atoms in Table 1 are in bold. First, a reaction is selected from the data set and an atom is randomly selected as the root atom. [CH:5] and C are selected here to obtain the reactant with the specified root atom. Then, the atom mapping is removed to obtain the final input. Then, from the left to the right of the reactant SMILES, the atom mapping that appears on the product SMILES is searched. Once found, the atom is selected as the root of the reactant. [C:7] is selected here. Without performing atom mapping, the root atom SMILES representation of the product is obtained. Finally, the reactants and products are tokenized.

[0086] The USPTO-Jin dataset is enhanced by one doubling using root alignment data enhancement to obtain the USPTO-Jin_R dataset. The detailed information of the two datasets is shown in Table 2:

[0087] Table 2 Statistics of the two data sets

[0088]

[0089] The specific steps of step S20 are as follows:

[0090] S21. The source sentences and target sentences of the processed USPTO-Jin_R dataset are converted into corresponding IDs according to the source dictionary and the target dictionary respectively. The source dictionary has 304 entries, which are projected to 512 dimensions through the Embedding layer of the dual encoder to obtain the source embedding X = {X1,...,X i ,...,X n } and the target embedding Y = {Y1,...,Y i ,...,Y n }, then scale and add position information code E pos , the final atomic embedding calculation formula is as follows:

[0091]

[0092]

[0093] in To process the source embedding, is the processed target embedding, and the value of a is 22.627416997969522;

[0094] S22. For the Transformer model with dual encoders, Input to the atomic encoder, pass through the multi-head self-attention module, and initialize the weight matrix W through attention Q , W K , W V , and dynamically adjust the weight matrix so that the weight reflects the importance of the sample, and finally obtain the Q, K, V matrices as follows:

[0095]

[0096]

[0097]

[0098] Q is the query matrix, K and V are the key and value matrices respectively, b Q 、b k 、b v is the bias vector, these parameters are learned by the model during training;

[0099] S23, finally get the attention matrix A j :

[0100]

[0101] where d k Represents the dimension of the key matrix. In the self-attention, the Q, K, and V matrices are linearly changed many times, and then the attention function is executed in parallel. They are connected and projected again to get the final result. The core of multi-head attention is scale dot product attention. Then the multi-head attention is connected in series through dot product attention. The calculation formula is as follows:

[0102]

[0103] Then A j After layer normalization, i.e. LN(A j ), the calculation formula is as follows:

[0104]

[0105] LN(A j ) After Dropout to prevent the model from overfitting, it passes through a feedforward network, which contains two linear transformations with a ReLU activation in the middle, followed by layer normalization and Dropout. Finally, steps S22 to S23 are repeated 6 times to obtain the final atomic correlation matrix X L , the batch size is 36, and the maximum length of the SMILS string is 48. L The Tensor value is as follows:

[0106]

[0107] S24. The processed source sentences are converted into corresponding IDs according to the source dictionary. The model obtains the output of the corresponding embedding layer according to the ID of each chemical formula, and then Figure 2 , the embedding codes of reactants and reagents in each chemical reaction are averaged, and the dot and special symbols are retained to obtain the molecular embedding representation X′={X′1,...,X′ i ,...,X′ n }, at the same time, recalculate the filling matrix and position encoding matrix, then scale and add the new position encoding E′ pos , at this time the value of a is also 22.627416997969522, and the final molecular embedding calculation formula is as follows:

[0108]

[0109] S25, Molecular Embedding The filled matrix is ​​input into the molecular encoder, and steps S22 to S23 are repeated 6 times to obtain the correlation matrix X′ between molecules L , specific X′ L The Tensor value is as follows;

[0110]

[0111] The specific steps of step S30 are as follows:

[0112] S31, correlation matrix X between atoms L and the correlation matrix X′ between molecules L Concatenate according to the last dimension, and then pass through the linear layer and Softmax activation function to get the weight matrix λ. The specific value of λ's Tensor is as follows:

[0113]

[0114] Then use the weight matrix for weighted fusion to obtain the multi-level feature fusion matrix X″, which is then input into the decoder. The relevant calculation formula is as follows:

[0115] L = W·concat(X L ,X′ L ) (10)

[0116] λ=softmax(L,dim=1) (11)

[0117] X″=λX L +(1-λ)X′ L (12)

[0118] Among them, W is the weight matrix of the linear layer as follows:

[0119]

[0120] The final Tensor value of X″ is as follows:

[0121]

[0122] The specific steps of step S40 are as follows:

[0123] S41. The decoder predicts the corresponding target symbol sequence one element at a time. Like the encoder, the decoder also consists of a stack of L identical layers. A third sublayer is inserted, which performs multi-head attention on the output of the encoder stack, using the residual connection around each sublayer, followed by layer normalization. The model's ability to translate word for word from source sentence x to target sentence y is calculated as follows:

[0124]

[0125] Where w(·) is a linear function that outputs the previously generated word y i The probability of i is the i-th source representation, and the i-th decoded hidden state s i Calculated as:

[0126] s i =f(s i-1 ,y i-1 ,c i )(14)

[0127] S42, the predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss, and the calculation formula is as follows:

[0128]

[0129]

[0130] Among them, L n is the negative log-likelihood of calculating the correct class label, L s It means to calculate the average of negative log probability of all categories, p yi represents the predicted probability of the i-th sample on the true category, M represents the number of samples, and the number of samples in the USPTO-Jin_R training set is 817,446. In order to prevent the model from overfitting when there are too few training samples, label smoothing is performed on the basis of the multi-classification cross entropy loss function. After adding label smoothing, the true probability distribution becomes ε:

[0131]

[0132] Among them, η is a small hyperparameter, η is used as 0.1, K represents the total number of multi-classification categories, j represents the target label, and the label-smoothed multi-classification cross entropy loss X loss as follows:

[0133] X loss =(1-η-ε)L n +εL s (18)

[0134] At this time

[0135] S43, N represents the number of samples in a batch, N is 36, through data enhancement 1 times, n is 1, and the number of samples of 2 pairs of positive samples is obtained. A batch has 72 samples in total. The contrast loss first selects all positive classes of each sample in the batch, so that the embedding similarity between the positive classes of each sample is high. Secondly, the negative classes of all standard samples in the batch are selected, so that the embedding similarity between the negative classes of standard samples is low. The contrast loss Using InfoNCE loss, the calculation formula is as follows:

[0136]

[0137] Among them, z i is the normalized sample of the ith positive class in a batch, z j is the non-normalized sample with the ith positive class in a batch enhanced by 1 times, sim(·) is the cosine similarity, τ is the temperature parameter, and z k is a normalized sample among the other 71 samples in a batch;

[0138] S44, multi-label smoothed cross entropy loss function to obtain loss X loss Loss with Contrastive Learning Add the weights and back propagate the weights of the optimization model. The calculation formula is as follows:

[0139]

[0140] Among them, E α The current Epoch is 1, E total The total epoch is set to 500, L z is the final loss function;

[0141] In step S50: Based on steps S10 to S40, the dual encoder model is trained. The model uses the adam optimizer, the initial learning rate is set to 5e-4, the learning rate is automatically updated, the dropout is 0.3, and the label_smoothing is 0.1. The trained dual encoder model is used to predict the product of the organic chemical reaction. The prediction accuracy is better than the baseline model and previous work, as shown in Tables 3 and 4:

[0142] Table 3 Test results of the model on two datasets

[0143]

[0144] Table 4 Comparison of related work on USPTO_Jin dataset

[0145]

[0146] Example 2

[0147] USPTO-FULL is a larger chemical reaction dataset consisting of approximately 1,000,000 reactions. Using root alignment data augmentation, the training, validation, and test sets of the USPTO-FULL dataset are augmented 5 times to obtain the diverse USPTO-FULL_5R dataset.

[0148] After S12 processing, a total of 4,803,620 chemical reaction data sets were obtained, including 3,843,150 training sets, 480,355 validation sets, and 480,115 test sets.

[0149] The experimental hyperparameters of this embodiment are set as follows: the batch size is 128, the initial value of the learning rate is 5e-4 and the learning rate scheduler based on the inverse square root is used, the dropout is set to 0.3, and the weight decay is 0.0001. The Adam optimizer is used in the loss function, and the beta parameter of the Adam optimizer is "0.9,0.98".

[0150] In order to evaluate the proposed method, the model evaluation indicators are BLEU and accuracy (ACC), which are calculated as follows:

[0151]

[0152] Among them, the manual translation is represented by s i,j , M is a total of M reference answers, and the translation is represented by c i , E is the total number of translations, d represents the dth phrase, h d (c i ) is the dth phrase in the translation c i The number of occurrences in h d (s i,j ) is the dth phrase in the standard answer S i,j The number of times it appears in .

[0153]

[0154] Among them, BP (Brevity Penalty) is the length penalty factor, Y C Indicates the length of the machine translation, Y S Indicates the length of the reference answer. When there are multiple reference translations, the length closest to the translation is selected. When the translation is longer than the reference translation, the penalty factor is 1, which means no penalty; when the translation is less than or equal to the reference translation, the penalty factor is calculated. The final formula of BLEU is as follows:

[0155]

[0156] Among them, Wn=1 / n, and the accuracy of 4-gram is counted at most. In addition, the beam search used is 5. The test results of the model calculate the accuracy of TOP-1, TOP-3 and TOP-5 respectively. The accuracy calculation formula is as follows.

[0157]

[0158] For each reaction, the model predicts 5 products, and u represents the sum of the model prediction accuracies in TOP-n.

[0159] The trained dual encoder model is used to predict the products of organic chemical reactions. The prediction accuracy is better than that of the baseline model, as shown in Table 5:

[0160] Table 5 Test results of the model on the USPTO-FULL_5R dataset

[0161]

[0162] Example 3

[0163] In Example 3, after removing duplicate and non-normalized reactions using Schwaller et al., a data set of 1,088,170 reactions with only a single product was retained, referred to as the USPTO-S data set, and detailed information is shown in Table 6.

[0164] Table 6 Dataset statistics

[0165]

[0166] The experimental parameters of Example 3 are the same as those of Example 2, and the experimental comparison results are shown in Table 7:

[0167] Table 7 Comparison of related work on USPTO_S dataset

[0168]

[0169] On the USPTO-S dataset, the model proposed in this embodiment has Top-1, Top-3, and Top-5 scores that are 0.47, 0.43, and 0.72 higher than the baseline model, respectively, indicating that the model has learned the atomic feature information and the interactions between molecules, extracted richer information, and improved the accuracy of the model's product prediction.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / separated> < / separated>

Claims

1. A prediction method for multi-level organic chemical reactions based on dual encoders, characterized in that: The following steps are involved: S10, read the organic chemical reaction SMILES training, validation and test datasets in sequence, and process the reactants and reagents as source sentences x = {x1,...,x i ,...,x n } as the target sentence y={y1,...,y i ,...,y n } SMILES expression, then use regular expressions to process and mark the data, and then use data enhancement to obtain more data samples; S20, using the atomic encoder and the molecular encoder to respectively encode the input source sentence x = {x1, ..., x i ,...,x n } for feature extraction, first input into the atom encoder to convert into the corresponding ID according to the source dictionary, and then mapped into X = {X1,...,X i ,...,X n }, the scaled X plus the position information code E pos become Where pos indicates the position, followed by After L cycles of multi-head attention mechanism, the correlation matrix X between atoms is obtained L , the molecular encoder averages the embedding codes of the reactants and reagents of each chemical reaction according to Figure 2 to obtain the molecular embedding matrix X′, and then scales and adds the new position encoding matrix to become After L times of multi-head attention mechanism, the correlation matrix X′ between molecules is obtained L ; S30, correlation matrix X between atoms L and the correlation matrix X′ between molecules L Concatenate according to the last dimension, pass through the linear layer and the Softmax activation function to get the weight matrix λ, then use the weight matrix for weighted fusion to get the multi-level feature fusion matrix X″, and then input it into the decoder; S40, target sentence y={y1,...,y i ,...,y n } is input to the decoder, and the Teacher Forcing technique and Mask mechanism are used to make the decoder process the embedding of the entire sentence at the same time during the training phase, predict each molecular token in parallel until the "end of sentence marker" is generated or the maximum length allowed for prediction is reached, and generate a probability distribution in the entire vocabulary space. The predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss X loss , and the loss of contrastive learning Add weights and back-propagate the weights of the optimized model; S50. Use the trained dual encoder model to predict products for new organic chemical reactions.

2. The prediction method of multi-level organic chemical reactions based on dual encoders according to claim 1, characterized in that: The specific steps of step S10 are as follows: S11. Read the organic chemical reaction SMILES dataset and retain the reactions without stereochemical information and atomic mapping. Then, follow the standard neural machine translation steps and use regular expressions to process and mark the data. For the normalized dataset, use the regular expression Regex to perform subword segmentation on the normalized sequence. The definition of Regex is as follows: Regex="(\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\(|\)|\.|=|#|-|\+|\\\\\ / |:|~|@|\?|>|\*|\$|\%[0-9]{2}|[0-9])" For the original data set, RDkit is used to convert the SMILES string into a molecule object. Then, all atoms in the molecule object are traversed. If there is a molAtomMapNumber attribute, the Python code atom.ClearProp is used to remove these mapping numbers to obtain SMILES characters without atom mapping. Finally, the regular expression Regex is used for subword segmentation. The reactants and reagents are combined into the source string, and the two reactants are separated by ". " and the reactants and reagents are separated by ". <separated> ” separation;< / separated> S12. Perform data enhancement on the training, test, and validation data sets respectively. During the enhancement process, filter out the non-standardized and repeated data to obtain more data samples.

3. The prediction method of multi-level organic chemical reactions based on dual encoders according to claim 1, characterized in that: The specific steps of step S20 are as follows: S21, convert the processed source sentence and target sentence into corresponding IDs according to the source dictionary and target dictionary respectively, and then project them to D dimension through the Embedding layer of the dual encoder to obtain the source embedding X = {X1,...,X i ,...,X n } and target embedding Y = {Y1,...,Y i ,...,Y n }, then scale and add position information code E pos , the final atomic embedding calculation formula is as follows: in is the processed source embedding, is the processed target embedding, a is the scaling constant; S22. For the Transformer model with dual encoders, Input to the atomic encoder, pass through the multi-head self-attention module, and initialize the weight matrix W through attention Q , W K , W V , and dynamically adjust the weight matrix so that the weight reflects the importance of the sample, and finally obtain the Q, K, V matrices as follows: Q is the query matrix, K and V are the key and value matrices respectively, b Q , b k , b v is the bias vector, these parameters are learned by the model during training; S23, finally get the attention matrix A j : where d k Indicates the dimension of the key matrix. In the self-attention, the Q, K, and V matrices are linearly changed many times. K T The K matrix is ​​transposed, and then the attention function is executed in parallel, and they are connected and projected again to get the final result. The core of multi-head attention is scale dot product attention, and then the multi-head attention is connected in series through dot product attention. The calculation formula is as follows: Among them, W O is the weight matrix of the attention mechanism, followed by A j After layer normalization, i.e. LN(A j ), the calculation formula is as follows: LN(A j ) passes through the Dropout layer to prevent the model from overfitting, and then passes through the feedforward network, which contains two linear transformations with a ReLU activation in the middle, and then passes through the layer normalization and the Dropout layer. Finally, steps S22 to S23 are repeated L times to obtain the final atomic correlation matrix X L ; S24. The processed source sentences are converted into corresponding IDs according to the source dictionary. The model obtains the output of the corresponding embedding layer according to the ID of each chemical formula. Then, according to Figure 2, the embedding codes of the reactants and reagents in each chemical reaction are averaged, and the dot and special symbols are retained to obtain the molecular embedding representation X′={X′1,...,X′ i ,...,X′ n }, at the same time, recalculate the filling matrix and position encoding matrix, then scale and add the new position encoding E′ pos , the final molecular embedding calculation formula is as follows: S25, Molecular Embedding The filled matrix is ​​input into the molecular encoder, and steps S22 to S23 are repeated L times to obtain the correlation matrix X′ between molecules L .

4. The prediction method of multi-level organic chemical reactions based on dual encoders according to claim 1, characterized in that: The specific steps of step S30 are as follows: S31, correlation matrix X between atoms L and the correlation matrix X between molecules L ′ is concatenated according to the last dimension, and then the weight matrix λ is obtained through the linear layer and the Softmax activation function. Then the weight matrix is ​​used for weighted fusion to obtain the multi-level feature fusion matrix X″, which is then input into the decoding. The relevant calculation formula is as follows: L=W·concat(X L ,X L ′)(10) λ=softmax(L,dim=1) (11) X″=λX L +(1-λ)X′ L (12) Where W is the weight matrix of the linear layer.

5. The prediction method of multi-level organic chemical reactions based on dual encoders according to claim 1, characterized in that: The specific steps of step S40 are as follows: S41. The decoder predicts the corresponding target symbol sequence one element at a time. Like the encoder, the decoder also consists of a stack of L identical layers. A third sublayer is inserted, which performs multi-head attention on the output of the encoder stack, using the residual connections around each sublayer, followed by layer normalization. The model's ability to translate word for word from source sentence x to target sentence y is calculated as follows: Where w(·) is a linear function that outputs the previously generated word y i The probability of i is the i-th source representation, and the i-th decoded hidden state s i Calculated as: s i =f(s i-1 ,y i-1 ,c i ) (14) S42, the predicted product and the target product are subjected to the multi-label smoothed cross entropy loss function to obtain the loss, and the calculation formula is as follows: Among them, L n is the negative log-likelihood of calculating the correct class label, L s It means to calculate the average of negative log probability of all categories, p yi represents the predicted probability of the i-th sample in the true category, M represents the number of samples. In order to prevent the model from overfitting when there are too few training samples, label smoothing is performed on the basis of the multi-classification cross entropy loss function. After adding label smoothing, the true probability distribution becomes ε: Among them, η is a small hyperparameter, K represents the total number of multi-classification categories, j represents the target label, and the label-smoothed multi-classification cross entropy loss X loss as follows: X loss =(1-n-e)L n +εL s (18) S43, N represents the number of samples in a batch. By enhancing the data n times, we get n pairs of positive sample pairs. At this time, nN samples are shared. The contrast loss first selects all positive classes of each sample in the batch, so that the embedding similarity between the positive classes of each sample is high. Secondly, the negative classes of all standard samples in the batch are selected, so that the embedding similarity between the negative classes of standard samples is low. The contrast loss Using InfoNCE loss, the calculation formula is as follows: Among them, z i is the normalized sample of the ith positive class in a batch, z j is the non-normalized sample of the ith positive class in a batch enhanced by n times, sim(·) is the cosine similarity, τ is the temperature parameter, and z k is a normalized sample among the other (nN-1) samples in a batch; S44, multi-label smoothed cross entropy loss function to obtain loss X loss Loss with Contrastive Learning Add the weights and back propagate the weights of the optimization model. The calculation formula is as follows: Among them, E α is the current Epoch, E total is the total epoch set, L z is the final loss function.

6. The method for predicting multi-level organic chemical reactions based on dual encoders according to claim 1, characterized in that: In the step S50: the dual encoder model is trained based on the steps S10 to S40, and the trained dual encoder model is used to predict the products of the organic chemical reaction.

Images