DNA fine-tuning language model primitive mining method based on long short-term memory network

Abstract

The invention relates to a DNA fine-tuning language model primitive mining method based on a long short-term memory network, which comprises the following steps of: constructing a DNA fine-tuning language model based on the long short-term memory network, pre-training the DNA fine-tuning language model by adopting human genome data, and then performing fine-tuning through a protein DNA binding data set, and finally realizing classification prediction on whether the input sequence is a potential DNA protein binding sequence or not. The DNA fine tuning language model comprises a human genome language model, a primitive fine tuning model and a primitive classification model, and the human genome language model is used for learning long-term dependency of a DNA sequence; the primitive fine tuning model is obtained by performing fine tuning on a human genome language model; and the primitive classification model performs classification learning based on the weight of the primitive fine tuning model. Compared with the prior art, the method has the advantages that the dependency relationship between the DNA sequences can be well represented, and the method also has relatively good performance under the condition of insufficient sample data volume.

Description

Technical Field

[0001] The present invention relates to the field of primitive mining, and in particular to a primitive mining method of a DNA fine-tuning language model based on a long short-term memory network. Background Art

[0002] The field of motif mining is both a data-rich and data-poor field. On the one hand, the rich and diverse biological experimental techniques have accumulated a huge amount of data; on the other hand, the existing models often require high-quality labeled samples, while labeled data is relatively scarce. Experimental methods cannot meet the needs of identifying the specific binding DNA motifs of a large number of unknown proteins one by one. The current deep learning methods are often trained from scratch, making it quite difficult to mine motifs when there are insufficient samples.

[0003] The same motif often has different functions and activities in different biological contexts, and multiple motifs that are far apart may work together, which means that there are complex semantic relationships with ambiguity and long-term dependencies in DNA sequences, such as enhancers and promoters that regulate at a distance. Current deep learning methods often use one-hot encoding as input, but any encoding vectors in one-hot encoding are mutually orthogonal, which means that the long-term dependencies of DNA sequences are cut off, which is not conducive to preserving the long-term dependencies of DNA sequences.

[0004] There are a lot of ambiguities and long-term complex semantic relationships in DNA sequences, which are also very common in human languages. Linguistic research has confirmed that DNA sequences, especially non-coding region sequences, show great similarities with human languages ​​in terms of letters, vocabulary and grammar.

[0005] In the field of natural language processing (NLP), when we talk about the context of words, we naturally think of using the ability of language modeling to build embedded representation vectors. KEGRU, ECLSTM (embedding single-layer CNN and LSTM) and ECBLSTM (embedding single-layer CNN and bidirectional LSTM) use the more commonly used word2vec word vector embedding in NLP to learn the long-term dependencies of DNA sequences. However, these models are all embedded learning for labeled downstream tasks. They fail to make reasonable use of unlabeled human genome data, which means that the learning data is limited, and they are still slightly insufficient for small sample data sets and modeling long-term dependencies of DNA sequences.

[0006] The dna2vec method developed by Patrick Ng of Cornell University is the first model to use human genome data for large-scale pre-training. Large-scale pre-training models have outstanding advantages. They can not only learn the long-term dependencies of DNA sequences from massive unlabeled genome data, but also fine-tune and learn the features of downstream tasks on a small amount of labeled data, thus better solving the problem of insufficient samples. However, dna2vec is still only a two-layer shallow neural network structure, and the relationship between words and vectors is one-to-one, so its representation ability is still relatively limited.

[0007] In summary, the existing solutions have the following problems: on the one hand, since most of the existing primitive mining deep learning algorithms are based on one-hot encoding, this encoding method is not conducive to capturing the long-term dependencies of DNA sequences; on the other hand, the currently commonly used deep learning algorithms generally perform poorly when data samples are scarce. Summary of the invention

[0008] The purpose of the present invention is to provide a DNA fine-tuning language model primitive mining method based on long short-term memory network in order to overcome the defects of the above-mentioned prior art that is not conducive to capturing the long-term dependencies of DNA sequences and the currently commonly used deep learning algorithms generally perform poorly when data samples are scarce.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A DNA fine-tuning language model primitive mining method based on long short-term memory network, comprising: constructing a DNA fine-tuning language model based on long short-term memory network, pre-training the DNA fine-tuning language model using human genome data, and then fine-tuning the DNA fine-tuning language model through a protein DNA binding data set, and finally achieving classification prediction of whether an input sequence is a potential DNA protein binding sequence;

[0011] The DNA fine-tuning language model includes a human genome language model, a primitive fine-tuning model and a primitive classification model. The human genome language model is pre-trained based on a human genome dataset and is used to learn the long-term dependencies of DNA sequences; the primitive fine-tuning model is obtained by fine-tuning the human genome language model on a protein DNA binding dataset; the primitive classification model performs classification learning based on the weights of the primitive fine-tuning model to achieve classification prediction of whether an input sequence is a potential DNA protein binding sequence.

[0012] Further, the human genome language model, primitive fine-tuning model and primitive classification model all include an embedding layer, an encoder layer and a decision layer connected in sequence;

[0013] The embedding layer is used to convert the input chromosome k-mer into a representation vector with a dimension greater than 100;

[0014] The encoder layer is used to process the input representation vector into a hidden state;

[0015] The decision layer is used to determine the classification result according to the hidden state.

[0016] Furthermore, when fine-tuning the primitive fine-tuning model, the weights of the embedding layer, encoder layer and decision layer of the human genome language model are used as the weights of the embedding layer, encoder layer and decision layer of the primitive fine-tuning model, respectively, and the first half of the encoder layer in the primitive fine-tuning model is frozen, and the second half of the encoder layer is learned.

[0017] Furthermore, the primitive classification model initializes the weights of the embedding layer and the encoder layer of the primitive classification model based on the weights of the embedding layer and the encoder layer in the primitive fine-tuning model; the weights of the decision layer of the primitive classification model are randomly initialized.

[0018] Further, the encoder layers of the human genome language model, primitive fine-tuning model, and primitive classification model are each composed of six stacked AWD-LSTM layers;

[0019] The decision layers of the human genome language model and the primitive fine-tuning model are linear layers, and the decision layer of the primitive classification model includes a first batch of normalization layers, a first linear layer, a second batch of normalization layers and a second linear layer stacked in sequence.

[0020] Furthermore, the human genome dataset is based on unlabeled human reference genome data, and autosomes are selected for chromosome k-mer segmentation processing to obtain k-mer short sequences; the goal of the human genome language model is to predict the next k-mer short sequence in the input sequence.

[0021] Furthermore, the process of chromosome k-mer segmentation processing specifically includes:

[0022] Separate the sex chromosomes from the human reference genome data S to obtain the autosomal genome S';

[0023] The obtained autosomal genome S' is divided into 22 autosomal base pairs into sequences SD of length l. j , the corresponding calculation expression is:

[0024] Chr i =SD1∪SD2∪SD3∪...SD i ...∪SD L-2 ∪SD L-1 ∪SDL

[0025] SD j =[N1,N2,N3...N m ...N l-2 ,N l-1 ,N l ]

[0026] N m =[Adenine,Thymine,Guanine,Cytosine]

[0027] In the formula, Chr i represents autosome i, L represents Chr i The total number of chromosome divisions, SD j represents the jth division of autosome i, l represents Chr i The chromosome partition length, N i represents the i-th nucleotide in the DNA sequence position, N i Any of the four nucleotides: Adenine, Thymine, Guanine or Cytosine;

[0028] The k-mer segmentation method is used to segment the sequence SD of length l j Divide into subsequences of length k, the calculation expression corresponding to the number of subsequences is:

[0029] wordsize=4 k +1

[0030] In the formula, wordsize is the number of subsequences and k represents the length of the subsequence.

[0031] Furthermore, the protein DNA binding dataset is a labeled ChIP-seq dataset, which is used for training motif fine-tuning models and motif classification models.

[0032] Furthermore, the expression of the learning rate of the encoding layer of the DNA fine-tuning language model is:

[0033] η l-1 =η l / α(2<α<3)

[0034] In the formula, η l is the learning rate of the lth layer, α is the decay rate, η l-1 is the learning rate of the l-1th layer.

[0035] Furthermore, when the DNA fine-tuning language model adopts the SGD optimizer for model training, the L2 regularization method or the weight decay method is adopted for weight decay;

[0036] When the DNA fine-tuning language model adopts the Adam optimizer for model training, the L2 regularization method is used for weight decay;

[0037] The expression of the L2 regularization method is:

[0038]

[0039] Where, L total is the learning rate of the lth layer, λ is the regularization parameter representing the weight decay rate, L is the loss function, and w is the weight;

[0040] The expression of the weight decay method is:

[0041]

[0042] In the formula, w i is the weight of the i-th layer, is the loss gradient.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] The present invention proposes a DNA fine-tuning language model primitive mining method based on long short-term memory network. First, it introduces the chromosome k-mer segmentation method to construct a pre-trained human genome language model to capture long-term dependencies from massive unlabeled human genome data; then, the trained human genome DNA language model is used to fine-tune the downstream tasks, which greatly reduces the number of samples required in training; finally, the experimental results of 69 ChIP-seq data sets show that the model has achieved significant performance advantages over DeepBind, DanQ, etc. in primitive mining tasks, and has relatively good performance even when the sample data is insufficient. In addition, through the visualization analysis of the encoding layer of one-hot encoding and DFLM, it is found that DFLM can better characterize the dependency relationship between DNA sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of a DNA fine-tuning language model based on a long short-term memory network provided in an embodiment of the present invention;

[0046] Figure 2Auc distribution diagrams of the ten models (pre3, pre4, pre5, fine3, fine4, fine5, Deep Bind, DanQ, ECLSTM, ECBLSTM) provided in the embodiments of the present invention in 69 ChIP-seq data sets;

[0047] Figure 3 The six indicators (Auc, Acc, Mcc, Recall, Precision, Specificity) of different parameter DFLM (pre3, pre4, pre5, fine3, fine4, fine5) provided in the embodiment of the present invention are shown in Figure 1. (a) is the performance of different DFLM in Auc; (b) is the performance of different DFLM in Acc; (c) is the performance of different DFLM in Mcc; (d) is the performance of different DFLM in Recall; (e) is the performance of different DFLM in Precision. (f) is the performance of DFLM in Specificity.

[0048] Figure 4 Auc distribution diagrams of four different types of complex, simple, small and large data sets for the ten models provided in the embodiments of the present invention; green represents a small data set, and yellow represents a large data set;

[0049] Figure 5 The visualization results of t-SNE using different encoding methods provided in the embodiments of the present invention; (a) is the t-SNE dimensionality reduction visualization of one-hot encoding; (b) is the t-SNE dimensionality reduction visualization of the 3-mer embedding representation pre-trained with hg38; (c) is the t-SNE dimensionality reduction visualization after fine-tuning the 3-mer embedding representation with the POLR2A dataset; (d) is the t-SNE visualization of the 3-mer embedding representation fine-tuned with the KDM5A dataset;

[0050] Figure 6 Schematic diagrams of language model learning rates and losses for different k-mers provided in an embodiment of the present invention; wherein (a) is a schematic diagram of language model learning rate and loss for 5-mer; (b) is a schematic diagram of language model learning rate and loss for 4-mer; and (c) is a schematic diagram of language model learning rate and loss for 3-mer. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment provides a DNA fine-tuning language model primitive mining method based on a long short-term memory network, including: constructing a DNA fine-tuning language model based on a long short-term memory network, pre-training the DNA fine-tuning language model using human genome data, and then fine-tuning the DNA fine-tuning language model through a protein DNA binding data set, and finally achieving classification prediction of whether an input sequence is a potential DNA protein binding sequence;

[0056] The DNA fine-tuning language model includes a human genome language model, a primitive fine-tuning model and a primitive classification model. The human genome language model is pre-trained based on the human genome dataset and is used to learn the long-term dependencies of DNA sequences. The primitive fine-tuning model is obtained by fine-tuning the human genome language model on the protein-DNA binding dataset. The primitive classification model performs classification learning based on the weights of the primitive fine-tuning model to achieve classification prediction of whether the input sequence is a potential DNA-protein binding sequence.

[0057] In other words, in order to train a model that can learn long-term dependencies of DNA sequences and adapt to small samples, this solution proposes a LSTM-based fine-tuned language model (DNA Fine-tuned Language Model, DFLM), which first uses human genome data for large-scale pre-training, and then performs primitive mining after fine-tuning. DFLM is developed based on AWD-LSTM (ASGD Weight-Dropped LSTM), which is a variant of LSTM and is particularly suitable for building language models. DFLM consists of a human genome language model, a primitive fine-tuning model, and a primitive classification model. The human genome language model is pre-trained on the human genome dataset, and can better represent the long-term dependencies of DNA sequences through AWD-LSTM. The primitive fine-tuning language model is obtained by fine-tuning the human genome language model on the primitive mining dataset, and can better learn the internal features of the primitive. The classification model completes the prediction task of the primitive by combining the features of the two language models.

[0058] The following is a detailed description of each step of this solution:

[0059] 1. Data Preparation

[0060] The human genome dataset is based on unlabeled human reference genome data. Autosomes are selected for chromosome k-mer segmentation to obtain k-mer short sequences. The goal of the human genome language model is to predict the next k-mer short sequence in the input sequence.

[0061] The protein-DNA binding dataset is an annotated ChIP-seq dataset used for training motif fine-tuning models and motif classification models.

[0062] Specifically, in this embodiment, DFLM mainly uses two types of data sets, one of which is 69 annotated ChIP-seq protein DNA binding (DBP) data sets. This part of the data set was obtained from the experiment of Alipanahi et al. The ChIP-seq data set is mainly used to train primitive fine-tuning language models and classification models. When DFLM was evaluated in 69 ChIP-seq experiments, the ChIP-seq data was divided into three parts: A, B, and C as training sets, validation sets, and test sets, where part A is the first 500 even peaks after sorting. Part B is the first 500 odd peaks after sorting, and part C is the remaining peaks. This chapter uses the same method to divide the data set. Part A and Part C are integrated into the training model, and 70% are randomly divided as training sets, 10% as validation sets, and the remaining 20% ​​as test sets. The positive sample is a 101bp region centered on each ChIP-seq peak. The negative sample is generated by randomly shuffling the positive sample.

[0063] The other is the unlabeled large-scale human reference genome data (GRCh38 / hg38). The unlabeled large-scale genome dataset hg38 is mainly used to train a general human genome language model. There are 23 pairs of chromosomes in the human genome, which contain a total of 3 billion DNA base pair sequences. Chromosomes are the carriers of genetic information and are mainly composed of DNA and proteins. These 23 pairs of chromosomes include 22 pairs of autosomes and one pair of sex chromosomes. Because the human genome data is too large and needs to be reasonably divided, and considering the particularity of sex chromosomes, this chapter chooses to use more representative autosomes for pre-training during pre-training. Therefore, when using the data, a special pre-processing method proposed in this article is introduced, which is called chromosome k-mer segmentation. The first step of chromosome k-mer segmentation is to separate the sex chromosomes from the genome S to obtain the autosomal genome S'. The formulas are shown in Equation 1 and Equation 2:

[0064] S=[Chr1,Chr2,Chr3...Chr i ...Chr 21 ,Chr 22 ,Chr 23 ] (1)

[0065] S'=S-Chr 23 =[Chr1,Chr2,Chr3,...Chr i ...Chr 21 ,Chr 22 ] (2)

[0066] Among them, Chr1 represents autosome 1, Chr2 represents autosome 2, and so on. irepresents autosome i, while Chr 23 Representative chromosomes.

[0067] After obtaining the autosomal genome S', it still needs to be segmented. The 22 autosomal base pairs are divided into sequences of length 1, which are specifically expressed as Formula 3, Formula 4 and Formula 5.

[0068] Chr i =SD1∪SD2∪SD3∪...SD i ...∪SD L-2 ∪SD L-1 ∪SD L (3)

[0069] SD j =[N1,N2,N3...N m ...N l-2 ,N l-1 ,N l ] (4)

[0070] N m =[Adenine,Thymine,Guanine,Cytosine] (5)

[0071] Among them, Chr i represents autosome i, L represents Chr i The total number of chromosome divisions, SD j represents the jth division of autosome i, l represents Chr i The length of chromosome division in this chapter is less than or equal to 2000, N i represents the i-th nucleotide in the DNA sequence position, N i It is any one of the four nucleotides: adenine (A), thymine (T), guanine (G) or cytosine (C).

[0072] After completing the chromosome division, it is also necessary to divide the sequence SD j Splitting into smaller subsequences can be compared to word segmentation in natural language processing. The most common segmentation method in genomes is k-mer, which divides longer sequences in biological sequences into subsequences of length k. If a DNA sequence is divided into a vocabulary represented by k-mer components, it will contain 4 k The specific vocabulary is shown in formula 6:

[0073]

[0074] For example, the k-mer segmentation results of the sequence ATCCGCGAT are shown in Table 1.

[0075] Table 1 k-mer with different parameters

[0076] k strides sub-sequences Wordsize 3 1 ATC TCC CCG CGC GCG CGA GAT 4^3+1＝65 3 2 ATC CCG GCG GAT 4^3+1＝65 4 1 ATCC TCCG CCGC CGCG GCGA CGAT 4^4+1＝257 5 1 ATCCG TCCGC CCGCG CGCGA GCGAT 4^5+1＝1025 6 1 ATCCGC TCCGCG CCGCGA CGCGAT 4^6+1＝4097

[0077] When the k-mer segmentation selects different k and strides, there will be different segmentation subsequences and the possible results are shown in Formula 7.

[0078] wordsize=4 k +1 (7)

[0079] Among them, k represents the length of the subsequence, strides represents the sliding step size, and wordsize is the number of all possible cases of the subsequence, which can also be understood as the number of vocabularies, plus one additional special case to represent situations other than A, T, C, and G.

[0080] DFLM Model

[0081] The human genome language model, primitive fine-tuning model, and primitive classification model all include an embedding layer, an encoder layer, and a decision layer connected in sequence;

[0082] The embedding layer is used to convert the input chromosome k-mer into a representation vector with a dimension greater than 100;

[0083] The encoder layer is used to process the input representation vector into hidden states;

[0084] The decision layer is used to determine the classification result based on the hidden state.

[0085] When fine-tuning the primitive fine-tuning model, the weights of the embedding layer, encoder layer, and decision layer of the human genome language model are used as the weights of the embedding layer, encoder layer, and decision layer of the primitive fine-tuning model, respectively, and the first half of the encoder layer in the primitive fine-tuning model is frozen, and the second half of the encoder layer is learned.

[0086] The primitive classification model initializes the weights of the embedding layer and the encoder layer of the primitive classification model based on the weights of the embedding layer and the encoder layer in the primitive fine-tuning model; the weights of the decision layer of the primitive classification model are initialized randomly.

[0087] The encoder layer of the human genome language model, primitive fine-tuning model, and primitive classification model all consists of six stacked AWD-LSTM layers;

[0088] The decision layers of the human genome language model and the primitive fine-tuning model are linear layers, and the decision layer of the primitive classification model includes the first batch of normalization layers, the first linear layer, the second batch of normalization layers, and the second linear layer stacked in sequence.

[0089] Specifically, in this embodiment, the DNA Fine-tuned Language Model (DFLM) divides the training into three steps, which have similar structures but different input data sets, such as Figure 1 shown.

[0090] (1) Human Genome Laguage Model, which is pre-trained using the human reference genome hg38.

[0091] (2) DBPs Fine-tune language model, which is fine-tuned on the DNA protein binding (DBPs) dataset for specific downstream tasks.

[0092] (3) Classification model: The protein binding site classification model is learned from the ChIP-seq dataset of DNA protein binding (DBPs) of the downstream task.

[0093] More specifically, the internal structures of these three steps of DFLM are similar, consisting of an embedding layer, an encoder layer, and a decision layer, respectively.

[0094] ① Embedding layer: It converts each input k-mer into a 200-dimensional vector representation. Embedding encoding is different from hot encoding vectors, and they are not mutually orthogonal. Using embedding layer encoding to map k-mer to a set of high-dimensional representation vectors can learn deeper relationships between k-mers, thereby obtaining meaningful contextual relationships.

[0095] ② Encoder layer: It processes the input representation vector into a hidden state. The encoder part consists of six stacked ASGD Weight-Dropped LSTM (AWD-LSTM) layers, such as Figure 1 As shown in Figure 2, AWD-LSTM is one of the best language modeling models in NLP. It can model a better language model while keeping the number of parameters small.

[0096] ③Decision layer. It will use hidden states to determine the classification results. It is particularly important to note that the structures of the language model and the classification model are different. This is because the language model predicts the next k-mer in the genome sequence, while the task of the classification model is to predict whether the input sequence is a DNA protein binding sequence, which is a binary classification problem.

[0097] (1) Human genome language model

[0098] In this step, DFLM needs to train a general genomic language model on a massive unlabeled corpus hg38. In particular, DFLM transforms and divides the long DNA sequence of hg38 into short k-mer sequences through the chromosome k-mer segmentation data set mentioned in the previous section. The language model is pre-trained through self-supervision. The goal of the model is to predict the next k-mer in the sequence, which means that DFLM can generate a large amount of pairing data between the input sequence and the next k-mer from any unlabeled genomic sequence. During the training process, DFLM masks 15% of the k-mers. The pre-training of DFLM first selects a k-mer from the adjacent k-mers for masking; then the DFLM model predicts the masked k-mer, and learns the features and embedded coding vectors in the genome in this self-supervised way.

[0099] (2)DBPs Fine-tune language model

[0100] When fine-tuning the model, DFLM does not need to randomly initialize the weights of the embedding layer, encoder layer, and decision layer, but uses the weights that have been trained in the human genome language model. However, please note that the hg38 dataset used for pre-training is different from the ChIP-seq dataset for downstream tasks. Therefore, DFLM needs to fine-tune the language model on the data of the target task. Compared with pre-training the human genome language model, the fine-tuning phase converges faster. In this chapter, the first 3 layers of the encoder are frozen when fine-tuning the language model. AWD-LSTM only learns the last 3 layers because it only needs to adapt to the characteristics of the data in a specific target domain. Freezing the first 3 layers can retain the generalization features of pre-training, and a powerful model can be trained even if the training dataset is small.

[0101] (3) Classification model

[0102] DFLM uses the weights that have been re-learned in the primitive language fine-tuning to initialize the weights of the embedding layer and encoder layer of the classification model. However, since the task of the classification model is to predict whether the input sequence is a potential DNA protein binding sequence, rather than the next k-mer, the outputs of the language model and the classification model are completely different, so the weights of the decision layer of the classification model are randomly initialized and learned again on the final task.

[0103] The classification model requires labeled data, so the training here will be done in a supervised manner. The classification task in this chapter is to identify whether a DNA sequence data binds to a protein, that is, all sequences in the classification data set must be marked as unable to bind or able to bind.

[0104] In summary, the DFLM model is roughly divided into three sub-modules: the human genome language module, the primitive fine-tuning language module, and the classification module.

[0105] The overall structure of DFLM is as described above, and the details of learning rate decay and regularization in the model will be described later. Stochastic gradient descent with a large initial learning rate is widely used to train modern neural network architectures. But a larger learning rate sometimes quickly achieves better generalization. In order to reduce repeated training for fine-tuning between models, a smaller initial learning rate initially allows faster training and better test performance, and improves the generalization and predictive ability of the model. The learning rate of the encoding layer of DFLM follows Equation 8.

[0106] η l-1 =η l / α(2<α<3) (8)

[0107] Among them, η l is the learning rate of the lth layer, and α is the decay rate.

[0108] One of the main features of the DFLM model is to effectively use regularization to improve model performance and prevent overfitting. A weight decay method is used for training in the DFLM model.

[0109] Weight decay penalizes models with large weights, which has been demonstrated by the AWD-LSTM language model. There are two ways to apply weight decay. One is the L2 regularization method, which directly adds the sum of the squares of the weights to the loss, as shown in Equation 9. Another weight decay method is to add the weight component during the gradient update process, as shown in Equation 10.

[0110]

[0111] Among them, L total is the learning rate of the lth layer, λ is the regularization parameter representing the weight decay rate, L is the loss function, w is the weight, is the loss gradient.

[0112] When using a simple optimizer like SGD, these two methods are equivalent. However, when DFLM uses the Adam optimizer to train the model, gradients can be accumulated. This means that L2 regularization will have a very different effect compared to weight decay. If DFLM uses L2 regularization, the sum of the squared weights will become part of the DFLM loss value and therefore part of the DFLM gradient. When the Adam optimizer accumulates gradients, the L2 regularization term is also accumulated at the same time. On the other hand, the weight decay method is only added in the update step, so the weight term is not accumulated during the gradient process. In practice, it is found that λ=0.1 works quite well. However, to be conservative, the weight decay in this embodiment still chooses 0.01 as the default value.

[0113] Finally, this experiment uses four different dropout probability hyperparameters.

[0114] (1) p_emb: dropout probability of the embedding layer;

[0115] (2)p_weight: dropout probability of LSTM layer weight;

[0116] (3)p_hidden: dropout probability of LSTM layer activation;

[0117] (4)p_output: dropout probability applied to the activations of the linear head.

[0118] DFLM usually uses the same ratio for all datasets and adjusts the variational dropout probability according to each dataset. One thing to note is that this paper does not conduct serious ablation experiments on the variational dropout probability configuration. The parameters used by DFLM are basically the general parameters set in the AWD-LSTM experiment, as follows:

[0119] For human genome language models and primitive fine-tuned language models.

[0120] p_emb=0.02; p_weight=0.15; p_hidden=0.1; p_output=0.25.

[0121] For classification models.

[0122] p_emb=0.1; p_weight=0.5; p_hidden=0.2; p_output=0.4.

[0123] 3. Experimental Results

[0124] This example will be divided into 4 parts for the analysis of experimental results. First, the overall performance of DFLM on 69 ChIP-seq datasets is explained. Then, this example analyzes the performance of DFLM on complex or simple datasets and large or small datasets in groups. Secondly, this example also visualizes and analyzes the encoding method to understand the DFLM model in a more intuitive way. Finally, this example also analyzes the loss of k-mer during training. The above experiments show that the DFLM model performs better than the commonly used methods, and the visualization of its encoding also shows that the DFLM model can effectively characterize DNA sequences.

[0125] 3.1. Performance of DFLM model on 69 ChIP-seq datasets

[0126] As shown in Table 2, the performance of 10 models on 69 ChIP-seq datasets was analyzed experimentally. The 10 models used in the experiment are listed in detail, which can be roughly divided into 3 groups, fine-tuning group (fine3, fine4, fine5), pre-training group (pre3, pre4, pre5) and classic method group (DeepBind, DanQ, ECLSTM, ECBLSTM). Among them, pre3, pre4 and pre5 are pre-training models with k-mer of 3, 4 and 5 on hg38, respectively, while fine3, fine4 and fine5 are fine-tuned models based on pre3, pre4 and pre5 on ChIP-seq datasets. In addition, the evaluation index Auc (Area Under Curve) in this chapter is briefly introduced. For the same False Positive Rate (FPR), when the True Positive Rate (TPR) is larger, the area is larger, that is, the Auc is larger. This means that among the samples predicted by the model as positive, the more actual positive samples, the better, and the fewer actual negative samples, the better. The meaning of Auc is the probability that the model predicts a greater value for the positive sample than for the negative sample when randomly selecting a positive sample and a negative sample and using the trained classifier for classification prediction. For other experimental evaluation indicators, see Appendix Table A1.

[0127] As shown in Table 2, it can be found that the average Auc (Average_Auc) of the classic method group (DeepBind, DanQ, ECLSTM, ECBLSTM) on 69 ChIP-seq data sets is less than 0.9, but the Average_Auc of the DFLM fine-tuning group (fine3, fine4, fine5) and pre-training group (pre3, pre4, pre5) is greater than 0.97, which is significantly higher than the previous four classic models. At the same time, the standard deviation Auc (Stdev_Auc) of the classic method group is also about four times that of DFLM. This means that the DFLM model not only performs better than the traditional model, but also performs more stably.

[0128] Table 2 Comparison results of 10 models on 69 ChIP-seq datasets

[0129]

[0130]

[0131] '+' and '-' represent operation and no operation respectively. pre3, pre4 and pre5 represent DFLM models that are pre-trained on hg38 with k-mers of 3, 4 and 5 without fine-tuning. fine3, fine4 and fine5 are DFLM models with k-mers of 3, 4 and 5 on ChIP-seq dataset.

[0132] In addition, it is not difficult to find that the performance of the DeepBind and DanQ models that do not use k-mer combined with embedded representation is relatively poor. This result also shows that the one-hot encoding of DNA sequences is too simple and often cannot fully represent the DNA sequences.

[0133] like Figure 2As shown in the figure, the experimental results show that the Auc fluctuation ranges of these 10 models vary greatly. The fine-tuning group (fine3, fine4, fine5) performs best, the pre-training group (pre3, pre4, pre5) without fine-tuning performs second, and the classic method group (DeepBind, DanQ, ECLSTM, ECBLSTM) performs worst. The Auc distribution of the best-performing fine-tuning group is concentrated around 0.98. In this group, the Auc distribution of fine3 is the most concentrated, and the overall stability is the best, but its minimum value is 0.869, slightly lower than the minimum value of fine4, 0.882. The Auc distribution of the second-best performing pre-training group is concentrated around 0.97. In this group, the Auc distribution of pre5 is the most concentrated, and the overall stability is good, but its minimum value of 0.861 is slightly lower than the minimum value of pre3, 0.869. The Auc distribution of the worst-performing classic method group is concentrated around 0.89, and the overall stability is poor. Among them, the Auc distribution of ECBLSTM using word embedding is the most concentrated.

[0134] The performance of the classic method group differs by about 35% from the best to the worst, and the generalization and robustness of the model are poor. The best performing dataset is when identifying CTCF transcription binding factors, with an Auc of 0.999, but the worst dataset is when identifying an RNA polymerase complex (POLR2A) consisting of 12 subnuclear cells, with an Auc of less than 0.65. This shows that the four models are more sensitive to the dataset and appear less stable. The difference between the best and worst Auc of the fine-tuning group and the pre-training group is about 0.15, and the Auc of the RNA polymerase complex (POLR2A) consisting of 12 subnuclear cells is 0.85. The performance of the fine-tuning group and the pre-training group is relatively stable and not much different, which also shows that the DFLM model has strong generalization ability and good robustness.

[0135] like Figure 3 As shown in Figure 2, both the fine-tuning group and the pre-training group achieved excellent results in six indicators, including Auc, Acc, Mcc, Recall, Precision, and Specificity, among which fine3 achieved the best results in all six evaluation indicators. Figure 3 (a) and Figure 3 As can be seen from (d) in Figure 2, the performance differences of these six models in terms of Auc and Recall are not obvious, with the difference being around 1%. Figure 3 (c) and Figure 3 As can be seen from (f) in the figure, the performance of the fine-tuning group is significantly better than that of the pre-training group in the performance comparison of Mcc and Specificity.

[0136] In terms of Mcc, the difference between the best performing fine3 and the worst performing pre3 is about 5.4%, while in terms of Specificity, the difference between the best performing fine3 and the worst performing pre4 is about 5.6%. Figure 3 (b) and Figure 3 As can be seen from (e), the differences in Acc and Precision among the six models are not as obvious as Mcc and Specificity, but the difference is also about 3%. Overall, the fine-tuning group outperforms the pre-training group in six indicators such as Auc, and in the fine-tuning group, the best performing model is fine3.

[0137] In summary, through experimental comparison, it was found that in the experiments on these 69 ChIP-seq datasets, the overall performance of the pre-training group and the fine-tuning group was good, and the fine3 using fine-tuning performed best overall. The overall performance of the classic method group was poor, and the ECBLSTM using word embedding performed best.

[0138] 3.2 Performance of DFLM on different grouped datasets

[0139] like Figure 4 As shown, the three models on the left are pre-trained groups, the three models in the middle are fine-tuned groups, and the four models on the right are classical method groups. The classification of the data sets is as follows: the ChIP-seq data set is divided into large and small groups based on whether the training set has more than 10,000 positive samples; the ChIP-seq data set is divided into complex and simple groups based on whether the bound protein is a component of the complex.

[0140] In order to evaluate the impact of positive sample quantity and motif type on the model, this section divides the 69 ChIP-seq datasets into four groups according to the above method. The first group is 9 complex DBP datasets with less than 10,000 positive training samples, and the second group is 11 complex DBP datasets with more than 10,000 positive training samples. The third group is 9 simple DBP datasets with less than 10,000 training samples, and the fourth group is 11 simple DBP datasets with more than 10,000 training samples.

[0141] The performance of the fine-tuning group is still the best. Both large and small training sets perform well in complex and simple groups. The Aucs of simple DBP in large datasets are mostly above 0.99, and the lowest value is also above 0.95. The Aucs of complex DBP in small datasets are mostly above 0.96, and the lowest value is also above 0.95. The Auc performance of the fine-tuning group is very good, and the performance of the three models is not much different, among which fine3 still performs best. However, in the smallest lysine demethylase 5A (KDM5A), it has only 1125 positive samples. In this dataset, fine3 performs much worse than other datasets, and its Auc value is only 0.869.

[0142] The best performing pre-trained group still performs well. In the large dataset, the Auc value of simple DBP is around 0.99, and the lowest value is close to 0.96. The Auc value of complex DBP in the small dataset is about 0.96, and the lowest value is also greater than 0.94. The performance of the three models in this group is not much different. Pre5 is the best among them, but it is far inferior to other models in the task of identifying KDM5A, just like fine3. The performance is far inferior to other datasets.

[0143] The worst performance is still in the classical method group. All four models in this group performed poorly, with ECBLSTM performing the best and DeepBind performing the worst. Identifying the binding of KDM5A protein is difficult for these four models, with an Auc value of only around 0.75. Identifying complex protein binding such as POLR2A is the most difficult for the classical method group, with an Auc of around 0.65, much worse than other datasets.

[0144] In summary, fine3 in the fine-tuning group still performs the best among all models and has made great progress, but there is still much room for improvement in cases where there are insufficient positive samples to identify, such as KDM5A.

[0145] 3.3 Visual Analysis

[0146] After analyzing the overall 69 results, we can see that the predictive ability of DFLM has been greatly improved. In order to understand DFLM more intuitively. t-Distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction algorithm, which is very suitable for reducing high-dimensional data to 2 or 3-dimensional visualization. Therefore, this section chooses t-SNE to visualize and analyze the encoded DNA sequence. In the visualization analysis, 3-mer pre-training, fine-tuning encoding layer and one-hot encoding are used for comparison, because in this case the vocabulary is the smallest in size for comparative analysis. In the visualization analysis, the coloring of the points is determined based on the first 2 bases of the 3-mer.

[0147] like Figure 5 As shown in Figure 1, the visualization results of t-SNE after one-hot encoding, pre3 encoding, and fine3 encoding on POLR2A and KDM5A. One-hot encoding is actually orthogonal. Encode the DNA sequence as a one-hot encoding of a binary vector, A = (0, 0, 0, 1), G = (0, 0, 1, 0), C = (0, 1, 0, 0) and T = (1, 0, 0, 0).

[0148] So the one-hot method ignores the relationship between the data sets themselves. Figure 5 It can be clearly seen from (a) that the distribution of one-hot encoding is chaotic, completely breaking the context of the DNA sequence. Figure 5 (b), (c), and (d) show that after representation learning, the encoding is significantly more regular. Figure 5 (b) in Figure 3 shows that when the dots have the same color, they are closer. In other words, dots consisting of the same first two bases are more similar in pre3. This can prove that pre3 is useful in learning the context dependency of DNA sequences. Figure 5 (c) and Figure 5 In (d), although similar phenomena were found, they had some special changes after fine-tuning on different datasets of POLR2A and KDM5A. But after fine-tuning the embedding vector, there are no longer points of the same color close together, which means that the first two bases can no longer determine whether they are similar. It can be seen that after the fine3 model further learns the relationship between DNA data based on pre3, fine3 re-learns specific motif relationships, not just the relationship of the DNA sequence context itself.

[0149] In summary, it can be found that one-hot encoding has no sequence dependency at all. In contrast, pre3 and fine3 learn sequence dependencies. In addition, fine3 encoding based on POLR2A and KDM5A datasets can learn different relationships between different specific motifs, which also helps the classification model achieve better performance in prediction.

[0150] 3.4 Hyperparameter Analysis

[0151] When training the language model of the human genome, the input information of the genome is divided into multiple k-mers. The training goal of the model is to input a k-mer and predict the next k-mer. DFLM also learns the structure of genome information in a completely unsupervised way based on k-mer. Then the selection of different k in k-mer has different effects on the performance of the model. This section provides an in-depth analysis and explanation of this.

[0152] As shown in Table 1, choosing different k will result in different k-mer segmentation results. When training language models to compare the performance between models, DFLM measures performance through cross entropy loss and accuracy, so the k-mer segmentation results have a great impact on the loss.

[0153] For example, when k is 5, this means there are 1025 possible k-mer segmentation results, and the cross entropy loss is. When k is 4, this means there are 257 possible k-mers, and the cross entropy loss is. When k is 3, this means there are 65 possible k-mers, and the cross entropy loss is.

[0154] The above analysis has also been verified in the practice process. Figure 6 shown. Figure 6 The 5-mer language model in (a) quickly trained to a loss of about 6.93 and then worked hard to reduce it. The model was trained to a loss of about 6.9. Figure 6 As shown in (b), the 4mer language model quickly trained to a loss of about 5.54 and then worked hard to reduce it. The model was trained to a loss of about 5.45. Figure 6 As shown in (c), the 3-mer language model quickly trains to a loss of about 4.17, and then slowly decreases. The final loss of the model is about 4.1. Therefore, when training a language model, do not be confused by the size of the loss of the language model performance, as their reference loss benchmarks are different.

[0155] IV. Conclusion

[0156] This embodiment proposes a pre-trained fine-tuning model DFLM based on AWD-LSMT. First, it can learn the long-term dependencies between DNA sequence contexts from the massive data of the unlabeled human genome; then, the trained DNA language model of the entire human genome is used to fine-tune the downstream tasks, which greatly reduces the number of samples required in training; finally, the experimental results on 69 ChIP-seq data sets show that the model has achieved significant performance advantages over DeepBind, DanQ, etc. in primitive mining tasks, and has relatively good performance even when the sample data is insufficient. In addition, through the visualization analysis of the one-hot encoding and the encoding layer of DFLM, it is found that DFLM can better characterize the dependency relationship between DNA sequences.

[0157] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A DNA fine-tuning language model primitive mining method based on long short-term memory network, characterized in that: include: Construct a DNA fine-tuning language model based on long short-term memory network, use human genome data to pre-train the DNA fine-tuning language model, and then fine-tune it with protein-DNA binding dataset to finally achieve classification prediction of whether the input sequence is a potential DNA-protein binding sequence; The DNA fine-tuning language model includes a human genome language model, a primitive fine-tuning model and a primitive classification model, wherein the human genome language model is pre-trained based on a human genome dataset and is used to learn long-term dependencies of DNA sequences; The motif fine-tuning model is obtained by fine-tuning the human genome language model on a protein DNA binding dataset; The motif classification model performs classification learning based on the weights of the motif fine-tuning model to achieve classification prediction of whether the input sequence is a potential DNA protein binding sequence.

2. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 1, characterized in that: The human genome language model, primitive fine-tuning model and primitive classification model all include an embedding layer, an encoder layer and a decision layer connected in sequence; The embedding layer is used to convert the input chromosome k-mer into a representation vector with a dimension greater than 100; The encoder layer is used to process the input representation vector into a hidden state; The decision layer is used to determine the classification result according to the hidden state.

3. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 2, characterized in that: When fine-tuning the model, the primitive fine-tuning model uses the weights of the embedding layer, encoder layer and decision layer of the human genome language model as the weights of the embedding layer, encoder layer and decision layer of the primitive fine-tuning model, respectively, freezes the first half of the encoder layer in the primitive fine-tuning model, and learns the second half of the encoder layer.

4. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 2, characterized in that: The primitive classification model initializes the weights of the embedding layer and the encoder layer of the primitive classification model based on the weights of the embedding layer and the encoder layer in the primitive fine-tuning model; the weight of the decision layer of the primitive classification model is randomly initialized.

5. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 2, characterized in that: The encoder layers of the human genome language model, primitive fine-tuning model, and primitive classification model are each composed of six stacked AWD-LSTM layers; The decision layers of the human genome language model and the primitive fine-tuning model are linear layers, and the decision layer of the primitive classification model includes a first batch of normalization layers, a first linear layer, a second batch of normalization layers and a second linear layer stacked in sequence.

6. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 1, characterized in that: The human genome dataset is based on unlabeled human reference genome data, and autosomes are selected for chromosome k-mer segmentation processing to obtain k-mer short sequences; the goal of the human genome language model is to predict the next k-mer short sequence in the input sequence.

7. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 6, characterized in that: The process of chromosome k-mer segmentation processing specifically includes: Separate the sex chromosomes from the human reference genome data S to obtain the autosomal genome S'; The obtained autosomal genome S' is divided into 22 autosomal base pairs into sequences SD of length l. j , the corresponding calculation expression is: Chr i =SD1∪SD2∪SD3∪...SD i ...∪SD L-2 ∪SD L-1 ∪SD L SD j =[N1,N2,N3...N m ...N l-2 ,N l-1 ,N l ] N m =[Adenine,Thymine,Guanine,Cytosine] In the formula, Chr i represents autosome i, L represents Chr i The total number of chromosome divisions, SD j represents the jth division of autosome i, l represents Chr i The chromosome partition length, N i represents the i-th nucleotide in the DNA sequence position, N i Any of the four nucleotides: Adenine, Thymine, Guanine or Cytosine; The k-mer segmentation method is used to segment the sequence SD of length l j Divide into subsequences of length k, the calculation expression corresponding to the number of subsequences is: wordsize=4 k +1 In the formula, wordsize is the number of subsequences and k represents the length of the subsequence.

8. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 1, characterized in that: The protein DNA binding dataset is a labeled ChIP-seq dataset, which is used for training motif fine-tuning models and motif classification models.

9. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 1, characterized in that: The expression of the learning rate of the encoding layer of the DNA fine-tuning language model is: or l-1 =the l / α(2<α<3) In the formula, η l is the learning rate of the lth layer, α is the decay rate, η l-1 is the learning rate of the l-1th layer.

10. The method for mining primitives of a DNA fine-tuned language model based on a long short-term memory network according to claim 1, characterized in that: When the DNA fine-tuning language model adopts the SGD optimizer for model training, the L2 regularization method or the weight decay method is adopted for weight decay; When the DNA fine-tuning language model adopts the Adam optimizer for model training, the L2 regularization method is used for weight decay; The expression of the L2 regularization method is: Where, L total is the learning rate of the lth layer, λ is the regularization parameter representing the weight decay rate, L is the loss function, and w is the weight; The expression of the weight decay method is: In the formula, w i is the weight of the i-th layer, is the loss gradient.

Images