Molecular property prediction method based on attention mechanism transfer learning

Abstract

The present invention discloses a molecular property prediction method based on attention mechanism transfer learning, which is applied to the technical field of chemical and material property prediction model construction. It includes the following steps: S1. Obtaining unlabeled molecular structure data; S2. Pre-training the unlabeled molecular structure based on the Transformer model to obtain a molecular structure understanding enhancement model; S3. Using the molecular structure understanding enhancement model obtained in S2 as a large model, fine-tuning the model according to limited molecular annotation data to construct a molecular property prediction model, and obtaining the final molecular property prediction model; S4. Using the labeled data to verify the accuracy of the final molecular property prediction model, and obtain data on the molecular property prediction effect. The pre-trained molecular structure understanding model of the present invention can be conveniently applied to prediction tasks in various property fields. Thanks to the transferability of the model, the amount of data and computing power cost required for molecular property prediction can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of building chemical and material property prediction models, and more specifically to a molecular property prediction method based on attention mechanism transfer learning. Background Art

[0002] In recent years, research on molecular property prediction has received widespread attention and has developed rapidly. With the development of computer technology, many new prediction methods and tools have emerged, and artificial intelligence and machine learning techniques have also been widely used in molecular property prediction. Compared with other disciplines, the overall explorable space in molecular chemistry is extremely large. However, predicting molecular properties often requires a large amount of molecular property data. However, obtaining large amounts of annotated data from general experiments and theoretical calculations is relatively difficult and requires a lot of effort or computing resources. Therefore, data in the field of molecular chemistry is small in scale, highly diverse, and difficult to obtain in batches through experiments. This has led to a severe shortage of open source datasets in related fields.

[0003] Random forests (RFs) have achieved some success in molecular property prediction models for regression tasks. However, due to the extreme lack of labeled data, the model's generalization ability is poor, prone to overfitting, and the actual prediction results are unsatisfactory. Similar to the challenges encountered in natural language processing (NLP), where large amounts of unlabeled data, such as molecular sequence information, are available, we are inspired by the recent large-scale pre-training of Transformers-based models in natural language processing. Similarly, large-scale molecular structure understanding enhancement models can be trained using large amounts of unlabeled sequenced molecular information. Transformers-based models pre-train the model using large amounts of unlabeled text information, enabling it to learn the sequence information of the text. Subsequently, downstream tasks such as text classification and sentiment analysis can simply be fine-tuned on relevant datasets, and impressive results can be achieved using only a small amount of labeled data. Drawing on this idea, we obtain SMILES molecular sequences and use them to train a large-scale molecular structure understanding enhancement model. This model requires only a small amount of labeled data and is widely applicable to various downstream molecular property prediction tasks.

[0004] To address these issues, we will employ the Transformers model architecture to build an enhanced molecular structure understanding model. This allows us to achieve prediction accuracy for even small samples of 200 or fewer data points, matching or exceeding that of 100,000 samples. The developed natural language processing literature crawling technology and the enhanced molecular structure understanding model will be used in demonstration applications, focusing on predicting the melting point of molecules and their density at standard temperature.

[0005] Therefore, a molecular property prediction method based on attention mechanism transfer learning is proposed to solve the problem that the trained prediction model has poor prediction effect due to the lack of labeled data in the fields of chemistry and materials. The above problem is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a molecular property prediction method based on attention mechanism transfer learning, which can learn molecular structure knowledge from a large number of unlabeled molecular sequences, and downstream tasks can be transferred to the construction of regression tasks and classification task models through Fine-tune fine-tuning, thereby greatly reducing the amount of data for downstream supervised training tasks and solving the small sample problem faced by molecular property prediction.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] The molecular property prediction method based on attention mechanism transfer learning includes the following steps:

[0009] S1. Obtain unlabeled molecular structure data;

[0010] S2. Based on the Transformer model, pre-training is performed on unlabeled molecular structures to obtain a molecular structure understanding enhancement model;

[0011] S3. Using the molecular structure understanding enhancement model obtained in S2 as the large model, fine-tune the model based on limited molecular annotation data to construct a molecular property prediction model, and obtain the final molecular property prediction model;

[0012] S4. Use the labeled data to verify the accuracy of the final molecular property prediction model and obtain the data of the molecular property prediction effect.

[0013] Optionally, the basic architecture of the Transformer in S2 is: it consists of two parts: the encoder and the decoder. The encoder consists of 6 encoding blocks, and the decoder consists of 6 decoding blocks.

[0014] Optionally, the Transformer encoder is composed of multiple identical layers stacked together, each of which has two sublayers, the first sublayer is a multi-head self-attention layer, and the second sublayer is a position-based feedforward network.

[0015] Optionally, the Transformer decoder consists of multiple identical layers, each of which contains three sublayers. The first sublayer is the decoder self-attention, the second sublayer is the "encoder-decoder" attention, and the third sublayer is the position-based feedforward network.

[0016] Optionally, each sub-layer is surrounded by a residual connection and subsequent layer normalization.

[0017] Optionally, the function expression of the residual connection g(x) is:

[0018] g(x)=f(x)+x

[0019] Where x is the vector space before the main model input, and f(x) is the result after calculation by the main model.

[0020] Optionally, the Transformer workflow is as follows:

[0021] Step 1: Generate the molecular SMILES formula, convert the molecular formula into a SMILES expression in the form of a text string;

[0022] Step 2: Molecular fragment decomposition: molecules expressed in SMILES are split based on the molecular fragment decomposition algorithm to obtain molecular structure fragments that are easy to understand and can be processed by the enhanced model;

[0023] Step 3: Molecular structure fragment embedding: converting the molecular structure fragments generated after disassembly into a vector space that can be recognized by the molecular structure understanding enhancement model;

[0024] Step 4: Annotate the position information of the molecular structure fragment, marking the position information of the fragment in the SMILES format;

[0025] Step 5: Labeling of molecular reactant-product identification information. This information is also used for pre-training of the molecular structure understanding enhancement model. This information is used to distinguish reactants and products in the pre-training phase.

[0026] Step 6: Pre-training: Based on the molecular structure understanding enhancement model pre-training algorithm, perform model pre-training to obtain the molecular structure understanding enhancement model;

[0027] Step 7: Relying on the molecular structure understanding enhancement model, conduct "molecular structure-performance" training based on small sample "molecular structure-performance" data.

[0028] Optionally, the architecture of the molecular structure understanding enhancement model in S2 is: a molecular SMILES-style generator and a molecular SMILES-style word segmenter.

[0029] Optionally, the molecule SMILES generator converts the molecule formula into a SMILES representation in the form of a text string.

[0030] Optionally, the molecular SMILES tokenizer is a molecule expressed in SMILES format, which needs to be segmented by the molecular SMILES tokenizer to obtain molecular structure fragments that are easy for the model to process.

[0031] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a molecular property prediction method based on attention mechanism transfer learning, which has the beneficial effects of: the pre-trained molecular structure understanding model of the present invention can be conveniently applied to prediction tasks in various property fields. Thanks to the transferability of the model, it is only necessary to fine-tune the model under small samples in a specific field to achieve better prediction effects, which can greatly reduce the amount of data and computing power costs required for molecular property prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 A flowchart of the molecular property prediction method based on attention mechanism transfer learning provided by the present invention;

[0034] Figure 2 A diagram of the Transformer model infrastructure provided by the present invention;

[0035] Figure 3 The molecular structure understanding enhancement model architecture diagram provided by the present invention;

[0036] Figure 4 Schematic diagram of the Fine-tune regression model provided by the present invention;

[0037] Figure 5 Schematic diagram of the Fine-tune classification model provided by the present invention;

[0038] Figure 6 This is a comparison chart of the MSE values ​​provided by the present invention;

[0039] Figure 7 A comparison chart of the coefficient of determination provided by the present invention;

[0040] Figure 8 A data chart comparing the actual values ​​of water parameters provided by the present invention with the model predicted values;

[0041] Figure 9 This is a changing trend diagram of the coefficient of determination provided by the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1 As shown, the present invention discloses a molecular property prediction method based on attention mechanism transfer learning, comprising the following steps:

[0044] SS1. Obtain unlabeled molecular structure data;

[0045] S2. Based on the Transformer model, pre-training is performed on unlabeled molecular structures to obtain a molecular structure understanding enhancement model;

[0046] S3. Using the molecular structure understanding enhancement model obtained in S2 as the large model, fine-tune the model based on limited molecular annotation data to construct a molecular property prediction model, and obtain the final molecular property prediction model;

[0047] S4. Use the labeled data to verify the accuracy of the final molecular property prediction model and obtain the data of the molecular property prediction effect.

[0048] Specifically, pre-training refers to pre-training a model using a large amount of unlabeled data in the absence of real data, enabling the model to learn some common features. Pre-training can improve the model's generalization ability and reduce the training time of downstream tasks.

[0049] For further information, see Figure 2 As shown in the figure, the basic architecture of the Transformer in S2 is composed of two parts: the encoder and the decoder. The encoder consists of 6 encoding blocks, and the decoder consists of 6 decoding blocks.

[0050] Furthermore, the Transformer encoder is composed of multiple identical layers stacked together, each of which has two sub-layers. The first sub-layer is a multi-head self-attention layer, and the second sub-layer is a position-based feedforward network.

[0051] Specifically, when computing the encoder's self-attention, the query, key, and value all come from the output of the previous encoder layer. The position-based feedforward network uses the same multi-layer perceptron (MLP) to transform the representation of all positions in the sequence. This is why the feedforward network is called positionwise.

[0052] Specifically, large amounts of unlabeled data can be used for self-supervised pre-training, and then the model can be adapted to various tasks with limited training data and computing resources. Self-supervised learning refers to a machine learning method (which can be considered a special case of unsupervised learning) that directly mines self-supervised information from large amounts of unsupervised data for supervised learning and training. Self-supervised learning requires labels, but these labels do not come from manual annotations but from the data itself. Common self-supervised training methods include random masking (Mask-LanguageModel) and next-sentence prediction (Next-Sentence-Predict), which enable large-scale models to learn sequence and structure knowledge from massive amounts of unlabeled text data. The large-scale sequenced molecular dataset used for self-supervised pre-training of Transformer models contains 1 billion compounds. Before pre-training, the tokenizer strategy needs to be redesigned to adapt it to the molecular sequence. At the same time, the model structure needs to be redesigned, reducing the width of the hidden layer and increasing the depth to match the molecular sequence.

[0053] Furthermore, the Transformer decoder consists of multiple identical layers, each of which contains three sub-layers. The first sub-layer is the decoder self-attention, the second sub-layer is the "encoder-decoder" attention, and the third sub-layer is the position-based feedforward network.

[0054] Specifically, in the masked multi-head decoder self-attention layer (the first sublayer), the query, key, and value all come from the output of the previous decoder layer. Regarding the sequence-to-sequence model, during the training phase, the word units of all positions (time steps) of its output sequence are known; however, during the prediction phase, the word units of its output sequence are generated one by one. Therefore, in any decoder time step, only the generated word units can be used in the decoder's self-attention calculation. In order to retain the autoregressive property in the decoder, its masked self-attention sets the parameter dec_valid_lens so that any query will only perform attention calculations with the positions of all generated word units in the decoder (that is, up to the query position).

[0055] Specifically, the large-scale pre-trained molecular structure understanding enhancement model can be pre-trained on a large amount of molecular structure data, and then the pre-training method can be used to improve the model's performance. The large-scale pre-trained molecular structure understanding enhancement model provides a new approach to understanding molecular structure, supporting research in multiple fields. For example, in drug design, the large-scale pre-trained molecular structure understanding enhancement model can help predict the activity and properties of molecules, thereby guiding drug design.

[0056] Furthermore, each sub-layer is surrounded by residual connections and subsequent layer normalization.

[0057] Specifically, residual connections can reduce model complexity to reduce overfitting and prevent gradient disappearance, allowing the model to be trained deeper and achieve better results.

[0058] Furthermore, the function expression of the residual connection g(x) is:

[0059] g(x)=f(x)+x

[0060] Where x is the vector space before the main model input, and f(x) is the result after calculation by the main model.

[0061] Furthermore, the Transformer workflow is as follows:

[0062] Step 1: Generate the molecular SMILES formula, convert the molecular formula into a SMILES expression in the form of a text string;

[0063] Step 2: Molecular fragment decomposition: molecules expressed in SMILES are split based on the molecular fragment decomposition algorithm to obtain molecular structure fragments that are easy to understand and can be processed by the enhanced model;

[0064] Step 3: Molecular structure fragment embedding: converting the molecular structure fragments generated after disassembly into a vector space that can be recognized by the molecular structure understanding enhancement model;

[0065] Step 4: Annotate the position information of the molecular structure fragment, marking the position information of the fragment in the SMILES format;

[0066] Step 5: Labeling of molecular reactant-product identification information. This information is also used for pre-training of the molecular structure understanding enhancement model. This information is used to distinguish reactants and products in the pre-training phase.

[0067] Step 6: Pre-training: Based on the molecular structure understanding enhancement model pre-training algorithm, perform model pre-training to obtain the molecular structure understanding enhancement model;

[0068] Step 7: Relying on the molecular structure understanding enhancement model, conduct "molecular structure-performance" training based on small sample "molecular structure-performance" data.

[0069] Specifically:

[0070] (1) Molecular fragment disassembly algorithm

[0071] We will analyze a compound based on the key factors that influence its properties. In organic chemistry, the atoms or groups of atoms that determine a compound's specific properties are called functional groups. Functional groups are clusters of atoms arranged in different structures. Common functional groups include hydroxyl, carboxyl, aldehyde, ketone, ester, halogen, amino, double, and triple bonds. Furthermore, the presence of repeating units, chiral molecules, branched chains, cyclic segments, and other chain structures within the molecule also significantly influence its properties. To this end, we will perform a SMILES-style molecular decomposition targeting these influencing factors.

[0072] (2) Molecular structure understanding enhancement model pre-training algorithm

[0073] We intend to adopt the following two self-supervised pre-training methods.

[0074] (1) Mask-prediction training. After the massive SMILES chemical formula information is segmented, the molecular fragments need to be masked and predicted so that the model can learn the common SMILES serialized structural information of chemical substances. For example, after the ethyl acetate O=C(OCC)C is segmented, [O=C(O,C,C,),C] (5 fragments separated by commas) is obtained. We mask and predict some of the fragments so that the model can learn knowledge and have prediction capabilities. Similarly, the design of the algorithm will tend to mask key fragments that affect the properties of the compound, such as functional groups, so we will perform the following masking [ <mask>,C,C,),C] and predict <mask>results.

[0075] (2) Reactant-product prediction training. The pre-training idea comes from the Transformer, which predicts the next token given the previous token. The chemical reaction data comes from the chemical reaction database we established. Reaction prediction can be regarded as an NMT (Neural Machine Translation) task, where reactants are one language and products are another language. Multiple reactants or products can be segmented using special characters [SEP]. This training task is more about learning the property information of SMILES. For example, in the esterification reaction between acetic acid and ethanol CC(O)=O+CCO->O=C(OCC)C, we can design a pre-training task to predict whether ethyl acetate O=C(OCC)C is the product of the reaction between acetic acid CC(O) and ethanol CCO.

[0076] For further information, see Figure 3 As shown in Figure 2, the architecture of the molecular structure understanding enhancement model in S2 is: a molecular SMILES-style generator and a molecular SMILES-style word segmenter.

[0077] Specifically, the difficulty lies in the tokenizer's segmentation rules, as a SMILES can be segmented in multiple ways. For example, the SMILES representation of ethyl acetate, O=C(OCC)C, can be segmented into [O=C(O, C, C,), C] (one sequence, five segments, with segments separated by commas) or [O=C(O, C, C), C] (one molecular sequence, four segments). Here, the segmentation will be based on key factors that influence the properties of the compound. In organic chemistry, the atoms or atomic groups that determine the specific properties of a compound are called functional groups. Functional groups are groups of particles composed of different structural combinations of atoms. Common functional groups include hydroxyl, carboxyl, aldehyde, ketone, ester, halogen, amino, double, and triple bonds. Furthermore, the presence of repeating units, chiral molecules, branched chains, cyclic segments, and other chain segments in a molecule can significantly influence its properties. To address these factors, we will define a specialized SMILES segmentation algorithm to perform molecular SMILES-style segmentation. And a small number of characters with special functions in the training process are added to assist in expressing the information needed in the subsequent training process. <cls>The character will be added to the beginning of the molecular sequence to aggregate the molecular sequence information. Downstream prediction tasks and classification tasks will use the hidden layer parameters under the character as output, and then the predicted data results will be obtained through the reconstructed fine-tuning small model. <pad>Characters are used to fill the molecular sequence. Since the length of the molecular sequence is not uniform, the tokenizer will uniformly expand the molecular sequence to the specified length for subsequent processing. <sep>Characters are used to segment molecular sequences. For example, in the molecular reaction prediction task in the pre-training stage, there will be multiple molecular sequences as an input, and the molecular sequence splicing will be used <sep>Characters are used as separators. <unk>Characters will be used as replacements for segments that are not in the tokenizer's vocabulary during the input phase, to address the situation where the vocabulary is insufficient. <mask>The character is a masked character. In the mask prediction task in the pre-training phase, the masked token will be replaced by <mask>Characters are used for subsequent pre-training tasks. The list of special characters is shown in Table 1.

[0078] Table 1: Special characters

[0079]

[0080] Molecular structure fragment embedding layer. This layer is responsible for converting the molecular structure fragments generated after segmentation into a vector space recognizable by the algorithm model. The parameter scale is (batch_num, seq_len, hidden_state). For example, ethyl acetate is segmented into [O=C(O, C, C,), C], and the vector space is (1, 5, hidden_state).

[0081] Molecular structure fragment position information. Molecular structure fragment position information is used to distinguish the start and end positions of the fragment in the SMILES formula. For example, ethyl acetate is segmented into [O=C(O, C, C,), C], and the position information is [1, 2, 3, 4, 5].

[0082] Molecular reactant-product identification information. Molecular reactant-product identification information is used to distinguish reactant-product predictions in the pre-training phase. For example, the esterification reaction between acetic acid and ethanol CC(O)=O+CC->O=C(OCC)C is divided into [C, C(O)=O <sep>,C,C,O, <sep>, O=C(O, C, C,), C], then the identification information vector is [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]. The molecular structure fragment embedding layer, the molecular structure fragment position information, and the molecular reactant-product identification information together form the molecular structure vector space of the model training input.

[0083] The multi-head self-attention layer for molecular structure fragments is primarily used to calculate the attention parameters between different molecular fragments, specifically to determine which fragments are more critical. The multi-head structure allows the model to focus on information from different dimensions and then comprehensively assess the importance of different molecular fragments. Because molecular fragments are not the same length as traditional sentence text, directly using the basic architecture of the transformer model would result in a large amount of parameter waste due to the large length difference, leading to slower training speed and reduced model performance. To address this issue, we reduced the hidden layer width of each layer in the pre-trained model to accommodate the shorter molecular fragment lengths. At the same time, we increased the number of layers and deepened the model, improving the model's prediction performance while maintaining the same total number of parameters and computational complexity.

[0084] Normalize the molecular structure vector space. After normalization, the molecular structure vector space produces an output with a mean of 0 and a variance of 1. This is done to keep the model parameters within a reasonable range of values, ensuring that the model always converges within a faster range, accelerating model convergence and improving model training speed and effectiveness.

[0085] The molecular structure vector space exchanges structural information bit by bit. Its input is a word vector, which, after a series of linear transformations and activation functions, outputs another word vector. This primarily exchanges information between model parameters. Considering that the attention mechanism may not adequately adapt to complex processes, the model's capabilities are enhanced by adding two layers of the network.

[0086] The model can be further layered, with the output of the previous layer serving as the input to the next layer. This will result in a total number of parameters reaching tens of millions, increasing the amount of molecular structure information the model can learn. Ultimately, the output is a 3D vector space containing molecular structure information, with a parameter scale of (batch_num, seq_len, hidden_state).

[0087] Finally, we can design a fine-tuned small model based on the above pre-trained model to fit the downstream task. For example, we can use the pre-trained model + feedforward neural network to obtain the predicted value as a new model for the regression task. Figure 4 The structure shown, or the pre-trained model + feedforward neural network + softmax to obtain different types of probabilities as a new model for classification tasks, such as Figure 5 The structure shown.

[0088] The current molecular property prediction task can select a new model for the regression task, use MSELoss as the loss function, record the mean square error between the predicted value and the true value, and serve as the basis for subsequent backpropagation fine-tuning of the model.

[0089] Furthermore, the molecular SMILES generator converts the molecular formula into a SMILES expression in the form of a text string.

[0090] Furthermore, the molecule SMILES word segmenter is a molecule expressed in SMILES, which needs to be segmented by the SMILES word segmenter to obtain molecular structure fragments that are easy for the model to process.

[0091] Specifically, the model parameters are as follows: the tokenizer model has a vocabulary size of 10,000 tokens, the model's sequence width is 128, the molecular structure fragment multi-head sub-attention layer width is 256, the number of multi-heads is 6, the molecular information bit-by-bit exchange layer width is 384, the overall model has a 4-layer structure, and the number of parameters of the complete model is approximately 10 million. The tokenizer model is saved as two files, tokenizer.json and vocab.json. The pre-trained molecular structure understanding enhancement model is saved using a bin binary file, and the fine-tuned molecular property prediction model is saved using a pth file. The main difference between the two is that the bin file only saves the model's parameter information, and the model structure needs to be defined in advance and then loaded, while the pth file saves the model's structure and parameters, so the model structure does not need to be defined in advance.

[0092] When deploying, you only need to use the torch.load function of pytorch to read the model.pth model file trained by Fine-tune fine-tuning, and use the tokenizer_load function under the transformers library to read tokenizer.json and vocab.json as the word segmentation model.

[0093] In subsequent use, the imported tokenizer is used to process the SMILES molecular sequence to be input, and a numerical value that can be read by the molecular structure understanding enhancement model is obtained. After inputting the pre-trained model, the target structure is obtained through the small model of Finetune, which can be a specific numerical value or a classification probability.

[0094] Specifically, some ideas from the traditional self-supervised pre-training stage can be continued, but innovative designs are still needed based on the characteristics of chemical sequences, and the training methods need to be consistent with the chemical formula structure and reaction mode.

[0095] The first step is mask-prediction training, which masks and predicts some tokens after tokenizer processing of massive SMILES chemical formula sequence information, enabling the model to learn the common SMILES serialization structural information of chemical substances.

[0096] Next, we train reactant-product prediction. Reaction data is taken from our established chemical reaction database. Reaction prediction can be viewed as an NMT (Neural Machine Translation) task, where reactants are in one language and products are in another. Multiple reactants or products can be separated using special characters [SEP]. This training task primarily focuses on learning the properties of SMILES sequences.

[0097] The molecular property prediction model is obtained by the molecular structure understanding enhancement model Fine-tune:

[0098] Fine-tuning, a term used in the field of deep learning, involves designing a new model based on the existing large model to suit downstream tasks. This new model is then trained using a small number of samples. This process also trains the parameters of the original pre-trained large model. Because the training set for supervised training is much smaller than the pre-training set, it is also called fine-tuning.

[0099] Therefore, the key to fine-tuning is to combine the pre-trained large model to design a new model that fits the downstream tasks, and then use the new model as the cornerstone of subsequent tasks to complete related tasks such as regression and classification.

[0100] In one embodiment, a molecular property prediction model based on the molecular structure understanding enhanced model ChemBert is compared with a traditional random forest regression model RF on several molecular property prediction problems.

[0101] ChemBert uses 200 hydrophobicity parameter (XlogP) data as a training set for five epochs of supervised training and 50 data as a validation set to verify the model training results. RF uses 100,000 hydrophobicity parameter (XlogP) data for supervised training. ChemBert directly uses SMILES molecular structure sequences as input, while RF uses tools we developed to extract characteristic information from chemical molecules, such as charge, polar surface area, lipophilicity, etc., and considers using dimensionality reduction algorithms such as Principal Components Analysis (PCA) and Linear Discriminant Analysis (LDA) to process high-dimensional data to improve the fitting effect of traditional methods. PCA aims to maximize the preservation of intrinsic information in the data after dimensionality reduction and is an unsupervised dimensionality reduction technique. LDA aims to make the data easily distinguishable after dimensionality reduction and is a supervised linear dimensionality reduction method. Both methods can extract information from the most critical dimensions, thereby reducing the dimensions of the training data. This can reduce the overfitting of traditional models, increase training speed, and potentially improve prediction results. This time, we choose to retain 30 data dimensions.

[0102] After the training is completed, a test set of 200 items is used to verify the molecular property prediction ability of the model, and the mean square error (MSE) and coefficient of determination (R2) are selected as the prediction quantitative analysis algorithm.

[0103] The result is: MSE mean square error refers to the expected value of the square of the difference between the parameter estimate and the true value of the parameter, which can be understood as the gap between the predicted value and the actual value of the molecular property, so the smaller the better. Figure 6 As can be seen, ChemBert's error in supervised training with a small sample size of 200 is significantly lower than the prediction results of the traditional RF model trained on 100,000 samples. However, the prediction results of the traditional RF model after PCA and LDA dimensionality reduction were not significantly improved. This may be mainly because the training data was already large enough, and the learnable information from PCA and LDA dimensionality reduction was largely fitted by the model, resulting in results similar to those obtained from training with the original data.

[0104] The coefficient of determination, also known as the coefficient of determination, coefficient of determination, and index of determination, represents the numerical characteristics of the relationship between a random variable and multiple random variables. It is a statistical indicator used to reflect the reliability of the regression model in explaining the changes in the dependent variable. Therefore, the larger the value, the better the fit of the regression model. The value range is [0, 100%]. Figure 7 As shown in the figure, the fitting degree of the ChemBert model is significantly higher than that of the traditional RF model, which also proves the superior performance of the ChemBert model under small sample sizes. The prediction effect of the traditional RF model after dimensionality reduction is basically consistent with the above. When the training data is large enough, the impact of dimensionality reduction on the training results is small. The actual values ​​of the molecular hydrophobic parameters (XlogP) of 10 samples that are not in the training set and the validation set are compared with the predicted values ​​of the molecular structure understanding enhancement model and the traditional RF model (trained with unreduced data). Figure 8 comparative data.

[0105] The data and line chart show that ChemBert's prediction errors for molecular properties are significantly smaller than those of traditional RF models, and the predicted trends in molecular properties are also more closely aligned with real data. This leads to the conclusion that the ChemBert pre-trained model, when fine-tuned on a smaller sample, can surpass the ability of traditional RF models to predict molecular properties even with 100,000 data points.

[0106] When the molecular structure understanding enhancement model is used to train the molecular property prediction model, the effect of the training set size on the prediction performance is further explored. 50, 200, 500, 1000, and 2000 training sets are used for supervised training, and 200 data are uniformly used as the evaluation set. The trend of the coefficient of determination of each model is as follows: Figure 9 .

[0107] It can be seen that the prediction effect of the model with 50 training data is similar to that of the traditional model with 100,000 training data. The bottleneck reached by further increasing the training data is around 86%. At this time, in order to further improve the prediction effect of the model, it is necessary to improve the pre-training model's learning ability and learning effect of molecular structure in the pre-training stage. This is also one of the main directions for subsequent improvement of this project.

[0108] The present invention constructs a model for enhancing molecular structure understanding, and its methods and ideas are also applicable to the construction of models for enhancing crystal structure understanding. The construction of models for enhancing crystal structure understanding is also within the scope of protection of this patent.

[0109] The terminology of the molecular structure understanding enhancement model or the crystal structure understanding enhancement model is not used, but the transfer learning method and technical route based on the Transformer architecture proposed in this patent are used for the training of material, physical, and chemical property prediction models, which also falls within the scope of protection of this patent.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0111] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.< / sep> < / sep> < / mask> < / mask> < / unk> < / sep> < / sep> < / pad> < / cls> < / mask> < / mask>

Claims

1. A molecular property prediction method based on attention mechanism transfer learning, characterized by: The following steps are involved: S1. Obtain unlabeled molecular structure data; S2. Based on the Transformer model, pre-training is performed on unlabeled molecular structures to obtain a molecular structure understanding enhancement model; S3. Using the molecular structure understanding enhancement model obtained in S2 as the large model, fine-tune the model based on limited molecular annotation data to construct a molecular property prediction model, and obtain the final molecular property prediction model; S4. Use the annotated data to verify the accuracy of the final molecular property prediction model and obtain data on the molecular property prediction effect; The Transformer workflow is as follows: Step 1: Generate the molecular SMILES formula, convert the molecular formula into a SMILES expression in the form of a text string; Step 2: Molecular fragment decomposition: molecules expressed in SMILES are split based on the molecular fragment decomposition algorithm to obtain molecular structure fragments that are easy to understand and can be processed by the enhanced model; Step 3: Molecular structure fragment embedding: converting the molecular structure fragments generated after disassembly into a vector space that can be recognized by the molecular structure understanding enhancement model; Step 4: Annotate the position information of the molecular structure fragment, marking the position information of the fragment in the SMILES format; Step 5: Labeling of molecular reactant-product identification information. This information is also used for pre-training of the molecular structure understanding enhancement model. This information is used to distinguish reactants and products in the pre-training phase. Step 6: Pre-training: Based on the molecular structure understanding enhancement model pre-training algorithm, perform model pre-training to obtain the molecular structure understanding enhancement model; Step 7: Relying on the molecular structure understanding enhancement model, conduct "molecular structure-performance" training based on small sample "molecular structure-performance" data.

2. The molecular property prediction method based on attention mechanism transfer learning according to claim 1 is characterized in that: The basic architecture of the Transformer in S2 is composed of two parts: the encoder and the decoder. The encoder consists of 6 encoding blocks, and the decoder consists of 6 decoding blocks.

3. The molecular property prediction method based on attention mechanism transfer learning according to claim 2, characterized in that: The Transformer encoder is composed of multiple identical layers stacked together, each of which has two sub-layers. The first sub-layer is a multi-head self-attention layer, and the second sub-layer is a position-based feedforward network.

4. The molecular property prediction method based on attention mechanism transfer learning according to claim 2, characterized in that: The Transformer decoder consists of multiple identical layers, each of which contains three sub-layers. The first sub-layer is the decoder self-attention, the second sub-layer is the "encoder-decoder" attention, and the third sub-layer is the position-based feedforward network.

5. The molecular property prediction method based on attention mechanism transfer learning according to claim 3 or 4, characterized in that Each sub-layer is surrounded by residual connections and followed by layer normalization.

6. The molecular property prediction method based on attention mechanism transfer learning according to claim 5, characterized in that: The function expression of the residual connection g(x) is: g(x)=f(x)+x Where x is the vector space before the main model input, and f(x) is the result after calculation by the main model.

7. The molecular property prediction method based on attention mechanism transfer learning according to claim 1, characterized in that: The architecture of the molecular structure understanding enhancement model in S2 is: a molecular SMILES-style generator and a molecular SMILES-style word segmenter.

8. The molecular property prediction method based on attention mechanism transfer learning according to claim 7, characterized in that: The SMILES generator converts a molecular formula into a SMILES expression in the form of a text string.

9. The molecular property prediction method based on attention mechanism transfer learning according to claim 7, characterized in that: The molecular SMILES tokenizer is a molecule expressed in SMILES format. It needs to pass through the molecular SMILES tokenizer to segment the SMILES format into molecular structure fragments that are easy for the model to process.

Images