DNA sequence k-mer frequency statistical method based on MapReduce

Abstract

The invention discloses a DNA sequence k-mer frequency statistical method based on MapReduce. The method includes the steps that a sequence file to be processed is preprocessed in a distributed cluster environment in which a MapReduce calculation model is run, and an error sequence is removed; the processed sequence file is subjected to hash processing and then serves as the input of a Map function, and all k-mer frequencies in the k variation range are computed through the Map function by utilizing a defined algorithm and then serve as the input of a Combine function; obtained intermediate results are locally merged through the Combine function and then serve as the input of a Reduce function; key value pairs with the same key sent by all Combine nodes are merged by Reduce, and final results are input. The method has the advantages of effectively processing a large-scale sequence data set, reducing the processing device performance requirement, solving the problem of large total processing time consumption of I/O overhead in the prior art, and significantly increasing the processing speed.

Description

technical field [0001] The invention relates to the fields of bioinformatics and big data processing, in particular to a MapReduce-based DNA sequence k-mer frequency statistics method. Background technique [0002] In recent years, with the development of third-generation biological sequencing technology, the biological gene sequences of various species measured by scientific research institutions and enterprises have exploded. In the face of massive biological DNA / RNA sequence data, the rapid and effective processing and analysis of these measured data poses a severe challenge to the current computer processing capabilities. [0003] DNA / RNA sequence is the storage and control center of biological genetic information. Counting the occurrence frequency of k-long subsequence in DNA / RNA sequence is a basic and important biological problem, which is called k-mer frequency counting problem. K-mer frequency has important applications in gene sequence assembly, repeated sequence ...

AI Technical Summary

Problems solved by technology

Since the BTKC algorithm needs to count all n sequences and load the results into memory, it consumes a lot of memory

And due to the need to frequently write intermediate results to the disk, the I / O overhead of the algorithm accounts for an excessively large proportion of the total processing time

Due to the above reasons, the BTKC algorithm can only process a small amount of DNA sequence data. When processing a large amount of DNA sequence data, the performance requirements of the computer are too high, and the processing efficiency is not ideal.

Images