Method for quickly and uniformly fragmenting fastq files

Abstract

The invention provides a method for quickly and uniformly fragmenting fastq files, which comprises the following steps of: concurrently loading the fastq files through two threads, constructing read data and outputting the read data; setting a plurality of fragments, and distributing the read data to fragment caches of the corresponding fragments according to a uniform fragmentation algorithm; andwriting read data in the fragment cache into an output fastq file of the corresponding fragment through an asynchronous fragment thread in an asynchronous fragment thread pool. According to the invention, a parallel working mode is utilized, and a plurality of threads simultaneously act synergistically, so that the operation efficiency is improved; a uniform fragmentation algorithm is designed, which fragment the read data belong to is efficiently calculated in real time in the fragmentation process, the process that a fastq file needs to be completely traversed to count the total number of reads in a traditional method is avoided, and time consumed by a program is saved; by applying the fragmentation caching technology, the calling frequency of a system lock and the operation frequency of a thread safety queue and a file system data writing interface are greatly reduced, the load of an operating system is reduced, and the purpose of quickly and uniformly fragmenting fastq files is achieved.

Description

technical field [0001] Embodiments of the present invention generally relate to the field of gene sequencing, and more specifically, relate to a method for quickly and evenly segmenting fastq files. Background technique [0002] In the field of gene sequencing, the samples are sequenced by a sequencer and output files such as BCL. After further data conversion and data splitting, the fastq file will be obtained. A fast file is a text-based file used to store biological gene base sequences, corresponding quality scores, and related information. It is the most commonly used file format in the field of gene sequencing. Each fastq file is composed of read data one by one, and each read data is composed of 4 lines of data. The bioinformatics analysis of genes is actually the analysis of this fastq file, and in more detail, it is the analysis of a large number of read data contained in the fastq file. [0003] However, due to the huge amount of genetic data, the size of a fastq...

AI Technical Summary

Problems solved by technology

This processing method is not efficient. For example, because the fastq file is very large, it may contain hundreds of millions of read data. The step of counting the total number of read data alone will consume a lot of time.

Images