Metagenome sequence traceability pre-training large model construction and fine tuning method

Abstract

The invention relates to a metagenome sequence tracing pre-training large model construction and fine tuning method, which comprises the following steps: on the basis of a pre-training model, constructing a metagenome biological classification deep learning model based on a HyenaDNA (Deoxyribose Nucleic Acid) architecture, and training by taking DNA data with different sequence lengths as a sample set to obtain a combined model for carrying out biological classification on a metagenome; and the accurate classification of the sources of the DNA sequences with different lengths in the metagenome sample is realized. According to the method, label prediction can be directly performed on the metagenome sequence without depending on a standard database, so that the biological source of the metagenome sequence can be predicted.

Description

Technical Field

[0001] The present invention relates to a technology in the field of genetic engineering, specifically a method for constructing and fine-tuning a large pre-trained model for metagenomic sequence tracing based on the HyenaDNA architecture. Background Art

[0002] Currently, taxonomic classification of metagenomic sample sources usually uses a homology-based DNA / RNA sequence comparison method to query the database. Since metagenomic samples often contain species data that are lacking in standard databases, it is difficult to classify these data using homology-based methods. Summary of the invention

[0003] In view of the defect that the existing homology-based DNA / RNA sequence comparison method is not good for the classification of metagenomic data in the absence of reference data, the present invention proposes a method for constructing and fine-tuning a large pre-training model for metagenomic sequence tracing, which can directly predict labels for metagenomic sequences without relying on standard databases, thereby predicting the biological origin of metagenomic sequences.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a method for constructing and fine-tuning a metagenomic sequence tracing pre-training large model. On the basis of the pre-training model, a metagenomic biological classification (Taxonomic Classification) deep learning model based on the HyenaDNA architecture is constructed, and DNA data of different sequence lengths are used as a sample set for training to obtain a combined model for biological classification of the metagenomics, thereby achieving accurate classification of the sources of DNA sequences of different lengths in metagenomic samples.

[0006] The pre-trained model is constructed by pre-processing DNA sequences from a standard database to form an unlabeled data set, which is used to train an initial blank model to obtain a pre-trained model that can characterize the potential genome sequence composition pattern of each species population.

[0007] The preprocessing refers to: classifying the genomes obtained from NCBI into their own data sets according to biological classification, such as archaea, eukaryotes, bacteria, and viruses, randomly shuffling the genome sequences in each category of the data set and writing them into 1000 files on average, and then dividing each category of the data set into a training set, a validation set, and a test set for deep learning model training in a ratio of 0.9:0.1:0.1.

[0008] The pre-training refers to: inputting each category of data sets into the initial blank model to perform the unlabeled task of predicting the next nucleotide token, that is, inferring the subsequent input tokens through the input tokens, and calculating the loss with the predicted results and the actual results, updating the model parameters according to the loss back propagation, and finally saving the model parameters with the minimum loss on the test set.

[0009] The initial blank model is a deep learning model based on the HyenaDNA architecture, including: a HyenaDNA encoding unit, a pooling unit and a fully connected decoding unit, wherein: the HyenaDNA encoding unit performs encoding processing according to the input DNA sequence information to obtain an embedding vector (embedding); the pooling unit performs aggregation processing according to the embedded vector information to obtain a reduced-dimensional vector; the fully connected decoding unit performs forward processing according to the reduced-dimensional vector information to obtain a prediction result.

[0010] The parameters inside each unit in the initial blank model are randomly initialized; the input length of the model is a hyperparameter, which is determined by the initialization setting and does not change with training. Other hyperparameters including the number of layers, dimensions, and hidden layers of the model are optimized with training.

[0011] The deep learning model based on the HyenaDNA architecture adopts but is not limited to Nguyen E, Poli M, Faizi M, et al. in "HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution" The technical implementation described in (Advances in neural information processing systems, 2024, 36.).

[0012] The fine-tuning mentioned above refers to: labeling the sequences of different biological groups to form a labeled data set for training the pre-trained model, and obtaining a final model that can accurately divide the metagenomic sequence directly without relying on the standard database, specifically: inputting data sets of various categories into the pre-trained model to perform multi-classification tasks. The input DNA data of each category has a label, and the model predicts the label of this sequence through the input DNA sequence, calculates the loss based on the predicted result and the actual result, and updates the model parameters based on the loss back-propagation, and finally retains the model parameters with the smallest loss on the validation set.

[0013] The different sequence lengths refer to using sequences of different lengths as input to train the model, such as 100 bp, 500 bp, 1000 bp, etc., to achieve species classification using sequences of different lengths.

[0014] The combined model refers to: combining the pre-trained models trained with sequences of different lengths as input in parallel, and inputting different models for processing according to the length interval of DNA of different lengths in the online stage, such as combining the models with 100bp, 500bp, and 1000bp inputs, and inputting the 100bp model for prediction of sequences with a sequence length less than 500bp, inputting the 500bp model for sequences greater than or equal to 500 and less than 1000bp, and inputting the 1000bp model for sequences greater than or equal to 1000bp, so as to better cope with the situation of diverse changes in DNA sequence lengths in metagenomic samples.

[0015] The present invention relates to a system for implementing the above method, comprising: a data collection unit, a data preprocessing unit, a model training unit and a model combination unit, wherein: the data collection unit obtains DNA sequence data of different biological classifications from the NCBI database according to biological classification information, and the data preprocessing unit randomly shuffles the downloaded data set under different biological classifications to divide it into a training set, a verification set and a test set. The model training unit inputs the data set obtained by the data preprocessing unit into the model and performs pretraining and fine-tuning under different input sequence lengths to obtain a classification model capable of processing different input sequence lengths. The model combination unit combines the models capable of processing different input sequence lengths obtained by model training to obtain a combination model capable of processing sequences of different lengths. Technical Effects

[0016] The present invention adopts a deep learning model based on the HyenaDNA architecture, and uses the genomes of different species on NCBI as training data to train a prediction model that can classify gene sequences of different lengths, thereby obtaining an artificial intelligence model that can trace the origin of most metagenome sequences. Compared with the prior art, the present invention can better handle sequences of variable lengths in the metagenome, make high-precision predictions of their species origins, and get rid of the dependence of gene comparison methods on standard databases. In addition, more accurate biological classification can be performed on some sequences that were not previously available in standard databases, such as some previously unknown viral sequences that can be discovered in the metagenome. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the present invention;

[0018] Figure 2 This is a schematic diagram of the model architecture;

[0019] Figure 3 Use schematics for your model;

[0020] Figure 4 A schematic diagram for evaluating the model of the embodiment;

[0021] Figure 5 It is a schematic diagram of the technical effect of the embodiment; DETAILED DESCRIPTION

[0022] This example takes the construction of biological classification models for archaea, eukaryotes, bacteria, and viruses as an example. Figure 1 The metagenome sequence traceability pre-training model is constructed and fine-tuned as shown in the figure, and the architecture is as follows Figure 2 A single prediction model is shown. Multiple models are trained at different input lengths and then combined to form Figure 3 The specific steps of the combined model include:

[0023] Step 1. Use NCBI's command-line tool to retrieve all reference genomes of archaea, eukaryotic microorganisms (including fungi, Chlorophyta and Protoctista) and bacteria from the NCBI database (as of April 12, 2024), as well as complete and chromosome genomes of viruses.

[0024] The formula is to randomly shuffle the sequences in different biological categories, divide them into 1000 files evenly according to the number of sequences, and then divide the data set of each category into training set, validation set and test set for training according to the ratio of 0.9:0.1:0.1. If the training set, validation set and test set of all categories are combined into an unlabeled data set.

[0025] Step 3: Input the unlabeled data set into the initial blank model for unlabeled task training of nextnucleotide (token) prediction to obtain a pre-trained model.

[0026] Step 4: Label the dataset according to different biological classifications (e.g., eukaryotic microorganisms are 0, bacteria are 1, archaea are 2, and viruses are 3). Input the labeled dataset into the pre-trained model and fine-tune the multi-classification task. Finally, the architecture is obtained as follows: Figure 2 Biological classification model.

[0027] Step 5: Change the input length of the model, repeat steps 3 and 4, and you can get classification models suitable for different sequence lengths (the length can be adjusted as needed, in this implementation example, 100 bp, 500 bp, 1000 bp), and combine these models to get the following Figure 3 The final combined model. When using the combined model, DNA of different lengths will be input into different models for processing according to their length intervals. For example, the models inputted with 100bp, 500bp, and 1000bp will be combined. Sequences with a length of less than 500bp will be input into the 100bp model for prediction, sequences with a length of greater than or equal to 500 and less than 1000bp will be input into the 500bp model, and sequences with a length of greater than or equal to 1000bp will be input into the 1000bp model. In the specific actual model construction, the NCBI command-line tool is used to obtain all reference genomes (reference genomes) of archaea, eukaryotic microorganisms (including fungi, Chlorophyta, and Protoctista), and bacteria from the NCBI database (as of April 12, 2024). The data sets are the genome of the virus with complete assembly level and the genome of the chromosome. The hyperparameters of the blank model are set to 4 layers, 2048 dimensions, and 256 hidden layers. The models are constructed according to the above method at three input lengths of 100bp, 500bp, and 1000bp. The performance of each model on the validation set is shown in the following figure. Figure 4 As shown, The prediction accuracy of the model with an input length of 100bp on a sequence of 100bp-500bp is 0.6030-0.7332, the prediction accuracy of the model with an input length of 500bp on a sequence of 500bp-1000bp is 0.8339-0.9008, and the prediction accuracy of the model with an input length of 1000bp on a sequence of 1000bp-10000bp is 0.9194-0.9826. By comparison, the model with an input length of 100bp in 100bp-500bp performs best, the model with an input length of 500bp on 500bp-1000bp performs best, and the model with an input length of 1000bp above 1000bp performs best, so the combined model will better handle sequences with variable lengths than using a single model. In addition, the combined model constructed by the above method was used to predict 18,960 viral sequences discovered from the intestinal metagenome (Nayfach S, Páez-Espino D, Call L, etc. Metagenomic compendium of 189,680 DNA viruses from the human gut microbiome. Nature Microbiology, 2021, 6(7):960-970.), most of which were not recorded in the standard database. The prediction results are as follows Figure 5 It can be found that 159,876 sequences were correctly traced back to viruses through the present invention, with a prediction accuracy of 83.85%, indicating that the model can also accurately trace the sequences of species data that are lacking in these standard databases.

[0028] Compared with the prior art, the present invention obtains data of different biological classifications from NCBI, constructs a pre-trained model based on the HyenaDNA architecture and fine-tunes it to obtain a combined model that can process metagenomic sequence data of different lengths, which can get rid of the comparison restrictions of the standard database and accurately trace the biological classification of metagenomic sequence data. In addition, the model can also accurately trace the biological origin of unknown sequences of species that have never been discovered in the metagenome, and can be used to mine new sequences of viruses or other biological classification levels from the metagenome.

[0029] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for constructing and fine-tuning a metagenome sequence traceability pre-training model, characterized in that: On the basis of the pre-trained model, a metagenomic biological classification (Taxonomic Classification) deep learning model based on the HyenaDNA architecture was constructed, and DNA data of different sequence lengths were used as sample sets for training to obtain a combined model for biological classification of metagenomics, thereby achieving accurate classification of the sources of DNA sequences of different lengths in metagenomic samples.

2. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1, characterized in that: The pre-trained model is constructed by pre-processing DNA sequences from a standard database to form an unlabeled data set, which is used to train an initial blank model to obtain a pre-trained model that can characterize the potential genome sequence composition pattern of each species population.

3. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1 or 2, characterized in that: The preprocessing refers to: after the genomes obtained from NCBI are classified into respective data sets according to biological classification, the genome sequences are randomly shuffled in each category of the data set, and then each category of the data set is divided into a training set (train dataset), a validation set (validate dataset) and a test set (test dataset) used for deep learning model training according to a ratio of 0.9:0.1:0.

1.

4. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1 or 2, characterized in that: The pre-training refers to: inputting each category of data sets into the initial blank model to perform the unlabeled task of predicting the next nucleotide token, that is, inferring the subsequent input tokens through the input tokens, and calculating the loss with the predicted results and the actual results, updating the model parameters according to the loss back propagation, and finally saving the model parameters with the minimum loss on the test set.

5. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1 or 2, characterized in that: The initial blank model is a deep learning model based on the HyenaDNA architecture, including: a HyenaDNA encoding unit, a pooling unit and a fully connected decoding unit, wherein: the HyenaDNA encoding unit performs encoding processing according to the input DNA sequence information to obtain an embedding vector (embedding); the pooling unit performs aggregation processing according to the embedded vector information to obtain a reduced-dimensional vector; the fully connected decoding unit performs forward processing according to the reduced-dimensional vector information to obtain a prediction result; The parameters inside each unit in the initial blank model are randomly initialized; the input length of the model is a hyperparameter, which is determined by the initialization setting and does not change with training. Other hyperparameters including the number of layers, dimensions, and hidden layers of the model are optimized with training.

6. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1, characterized in that: The fine-tuning refers to: by labeling the sequences of different biological groups to form a labeled data set, which is used to train the pre-trained model, and obtain the final model that can accurately divide the metagenomic sequence directly without relying on the standard database. Specifically, the data sets of each category are input into the pre-trained model to perform multi-classification tasks. The input DNA data of each category are all accompanied by a label. The model predicts the label of this sequence through the input DNA sequence, calculates the loss based on the predicted result and the actual result, and updates the model parameters based on the loss back-propagation, and finally retains the model parameters with the smallest loss on the validation set.

7. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 6, characterized in that: The different sequence lengths refer to using sequences of different lengths as input to train the model so as to achieve species classification using sequences of different lengths.

8. The method for constructing and fine-tuning a metagenome sequence traceability pre-training model according to claim 1, characterized in that: The combined model refers to: combining the pre-trained models trained with sequences of different lengths as input in parallel, and in the online stage, inputting different models for processing according to the length range of DNA of different lengths, so as to better cope with the situation where the length of DNA sequences in metagenomic samples varies greatly.

9. A system for constructing and fine-tuning a metagenome sequence traceability pre-training model for implementing any of the methods described in claims 1-8, characterized in that: include: A data collection unit, a data preprocessing unit, a model training unit and a model combination unit, wherein: the data collection unit obtains DNA sequence data of different biological classifications from the NCBI database according to biological classification information; the data preprocessing unit randomly shuffles the downloaded data set under different biological classifications and divides it into a training set, a validation set and a test set; the model training unit inputs the data set obtained by the data preprocessing unit into the model and performs pretraining and fine-tuning under different input sequence lengths to obtain a classification model capable of processing different input sequence lengths; the model combination unit combines the models capable of processing different input sequence lengths obtained by model training to obtain a combination model capable of processing sequences of different lengths.

Images