Methods for compressing genome sequence data

Abstract

To provide a method for compression of genome sequence data, a hardware processor, a system, and a storage device.SOLUTION: The method includes the steps of: determining whether leads of sequences of nucleotides or bases that have been aligned to a reference sequence are perfectly or imperfectly mapped with the reference sequence or whether the leads are unmapped with the reference sequence; and encoding the perfectly mapped leads according to a first encoding process and encoding the unmapped leads according to a second encoding process. In the determining step, for each imperfectly mapped lead, the number of mismatches between the lead and the reference sequence is compared with a threshold value. In the encoding step, the imperfectly mapped leads, when the number of mismatches is larger than the threshold value, are encoded according to the second encoding process, and when the number of mismatches is smaller than the threshold value, are encoded according to a third encoding process.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This subfield generally relates to a method for displaying genomic sequencing data generated by a sequencing machine, and more particularly to a computer-implemented method for compressing such genomic sequencing data. The present disclosure provides a reference-based compression method that enables fast compression and decompression while causing no loss of information and has a high compression ratio.

Background Art

[0002] Next-generation sequencing machines today generate vast amounts of sequencing data at low cost. Recent systems generate over 6 billion 150-nucleotide-long sequences sufficient to sequence 20 whole human genomes in a single 36-hour run. This opens up many new prospects for the diagnosis of genetic diseases and the development of personalized medicine aimed at adopting treatments based on people's genomic specificities.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, this also brings new problems, particularly the cost associated with storing vast amounts of data. The most commonly used file format for raw (unaligned) sequence data is the FASTQ format, which holds sequence data (a strand of A, C, T, G nucleotides, also called reads), quality values (the probability that the sequencing platform made a sequencing error for each nucleotide), and sequence names. This is usually a simple ASCII text file compressed with the general-purpose text compression scheme LZ (Lempel-Ziv scheme, implemented within the gzip software). However, the use of such compression methods has several problems, - low compression ratio due to the fact that data redundancy is not fully utilized - slow compression and decompression associated therewith.

[0004] There are also compression methods specifically designed for FASTQ encoding, which are divided using either reference-based or non-reference-based methods. However, neither perfectly satisfies the requirements, as a) reference-based methods have a good compression ratio but are slow, and b) non-reference-based methods are faster but have a lower compression ratio. An example of such a non-reference-based method is provided by the software SPRING, which is a reference-free compressor for FASTQ files (Worldwide Web address: github.com / shubhamchandak94 / SPRING). However, the compression method provided by the software SPRING has a low compression ratio.

[0005] Among reference-based compression methods, several have been proposed that use sequence alignment and aim to be faster with a good compression ratio. However, such methods suffer from several problems, the main challenge being that these problems never completely disappear. Such known reference-based compression methods are described, for example, in the International Publication of Patent Document No. 2018 / 068829(A1). In the described method, after being aligned to one or more reference sequences, the nucleotide sequences are classified by matching the degree of precision (thus creating classes of aligned reads), and then coded as syntactic elements of multiple layers using different source models and entropy coders for each layer into which the data is divided. Thus, classes of data are encoded separately and consist of syntactic elements of different layers, each layer containing a descriptor that uniquely represents the classified and aligned reads of that layer. The method is intended to obtain separate information sources with a reduction in information entropy, thereby enabling increased compression performance and selective access to specific classes of compressed data. However, this compression method reorders the reads in an order different from the order obtained at the end of the read alignment process (i.e., the reads are reordered by their respective classes). Subsequently, some information is lost during the compression process, particularly during the initial ordering. Therefore, the reproducibility of some analysis results may be affected because some downstream analysis software may depend on the order of the reads. In addition, by decompressing the data in an order different from the initial order of the reads, it becomes much more difficult to verify that the uncompressed file is identical to the original file. Furthermore, this compression method is relatively slow, especially when compared to modern, non-reference-based compression methods. [Means for solving the problem]

[0006] The following independent claims feature a method for compressing genome sequence data, thereby solving the problems of existing prior art solutions. In one embodiment, a computer implementation method for compressing genome sequence data generated by a sequencing machine, wherein the genome sequence data includes reads of nucleotide or base sequences aligned with a reference sequence, thereby creating aligned reads, and the aligned reads are stored as a list of reads in an initial file, the computer implementation method being: - For each aligned read, determine whether the read is fully or partially mapped to the reference sequence, or whether the read is not mapped to the reference sequence. -Encoding the reads by the decision, wherein reads determined to be fully mapped are encoded by a first encoding process, and reads determined to be unmapped are encoded by a second encoding process, including, - The determination process includes comparing the number of mismatches between each incompletely mapped read and the reference sequence with a threshold, - In the encoding process, reads determined to be incompletely mapped are encoded by a second or third encoding process. Incompletely mapped reads are encoded by the second encoding process when the number of mismatches is greater than a threshold, and incompletely mapped reads are encoded by the third encoding process when the number of mismatches is less than a threshold. -In the second encoding process, each nucleotide or base of the read is encoded individually. - The first encoding process and the third encoding process each include a separate set of descriptors, each set of descriptors uniquely representing a read associated with the corresponding encoding process, and each of the first encoding process and the third encoding process is an information source entropy reduction encoding process.

[0007] The present invention overcomes the shortcomings of conventional compression methods by enabling high-speed compression and decompression without causing loss of information, and by providing a high compression ratio. More specifically, the present invention focuses on encoding the most frequent occurrences in the most compact way, even if this means employing a reduced encoding mode for rare, least frequent occurrences. This leads to a significant increase in compression performance. Furthermore, the genomic information display format used in the present invention makes the compression performed by the method according to the present invention even faster. Lastly, the method according to the present invention preserves the initial order of reads and does not reorder reads by class. As a result, no information is lost during the process, which allows for easier downstream analysis and efficient consistency checks after the decompression process.

[0008] These features and advantages of the present invention, as well as other features and advantages, will become more apparent from the accompanying drawings and subsequent embodiments for carrying out the invention. In addition, although thresholds may be referred to herein as those that are exceeded or not exceeded, it will be understood that such thresholds can be conceptually employed to determine whether such thresholds are met, matched, or otherwise detected, regardless of whether the numbers or values ​​used to carry out those threshold evaluations are described using positive or negative values.

[0009] A method for compressing genome sequence data is disclosed according to one innovative aspect of the present disclosure. In one aspect, the method may include the execution of one or more operations via the execution of software instructions by one or more computers, the operations of which include: obtaining a read record by one or more computers; determining by one or more computers whether the read record corresponds to a read that is fully mapped to a reference sequence or incompletely mapped to a reference sequence, based on one or more computers determining that the read record corresponds to a read that is incompletely mapped to a reference sequence; determining by one or more computers whether the number of mismatches in the incompletely mapped read satisfies a predetermined mismatch threshold number; and encoding each mismatch of the incompletely mapped read into a record having a size of one byte, based on the determination that the number of mismatches satisfies the predetermined mismatch threshold number.

[0010] Other embodiments include corresponding systems, apparatus, and computer programs for performing actions in the manner disclosed herein, such as those defined by instructions encoded on a computer-readable storage device.

[0011] These and other versions may optionally include one or more of the following features. For example, in some implementations, determining whether the number of mismatches in incompletely mapped reads satisfies a predetermined mismatch threshold by one or more computers may include determining whether the number of mismatches in incompletely mapped reads is greater than a predetermined mismatch threshold by one or more computers.

[0012] In some implementations, each read record may include data indicating the absolute start position of the read aligned with respect to the reference sequence, data indicating the length of the read, data indicating whether the read was fully or partially mapped, data indicating the number of mismatches identified within the read, and data indicating the relative position of each of those possible mismatches within the read.

[0013] In some implementations, encoding each mismatch in an incompletely mapped read into a one-byte record involves, for each specific mismatch, one or more computers encoding the first two bits of the byte to include data indicating the alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or base in the reference sequence, and encoding the remaining six bits of the byte to include data indicating the location of the mismatch in the reference sequence, which is calculated as an offset from the preceding mismatch in the read.

[0014] In some implementations, the method may further include one or more computers determining whether the offset is greater than the maximum codeable value, and, based on the determination that the offset is greater than the maximum codeable value, one or more computers inserting at least one false mismatch between a particular mismatch and a preceding mismatch.

[0015] In some implementations, the method may further include, based on the determination that the number of mismatches does not satisfy a predetermined mismatch threshold, encoding a list of reference array positions corresponding to each mismatch position in the reference array using an information entropy reduction encoding process by one or more computers.

[0016] In some implementations, the method may further include encoding at least a portion of the read records using information entropy reduction coding by one or more computers, based on the determination that the read records correspond to reads that are fully mapped to a reference sequence.

[0017] In some implementations, one or more computers can include one or more hardware processors.

[0018] In some implementations, one or more hardware processors can include one or more field-programmable gate arrays (FPGAs).

[0019] In some implementations, the method for compressing genome sequence data can be performed by one or more hardware processors. In such implementations, the hardware processor may include hardware processing circuits configured to perform one or more operations. In one embodiment, the operations may include: acquiring a read record by the hardware processing circuit; determining by the hardware processing circuit whether the read record corresponds to a read that is fully mapped to or incompletely mapped to a reference sequence; determining by one or more computers, based on the determination by the hardware processing circuit that the read record corresponds to a read that is incompletely mapped to a reference sequence, whether the number of mismatches in the incompletely mapped reads satisfies a predetermined mismatch threshold number; and encoding each mismatch in the incompletely mapped read into a record having a size of 1 byte by the hardware processing circuit, based on the determination that the number of mismatches satisfies the predetermined mismatch threshold number.

[0020] These features and advantages of the present invention, as well as other features and advantages, will become more apparent from the accompanying drawings and the following detailed description of the invention for implementation.

[0021] In some implementations, each read record can include data indicating the absolute start position of the read aligned with respect to the reference array, data indicating the length of the read, data indicating whether the read is fully mapped or incompletely mapped, data indicating the number of mismatches identified within the read, and data indicating the relative positions of the possible mismatches within the read.

[0022] In some implementations, determining whether the number of mismatches in an incompletely mapped read satisfies a predetermined threshold number of mismatches by a hardware processing circuit can include determining whether the number of mismatches in the incompletely mapped read is greater than a predetermined threshold number of mismatches by the hardware processing circuit.

[0023] In some implementations, encoding each mismatch in an incompletely mapped read into a record having a size of 1 byte can include encoding the first 2 bits of the 1 byte to include data indicating an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or base in the reference array for each specific mismatch encoding, and encoding the remaining 6 bits of the 1 byte to include data indicating the position of the mismatch in the reference array, where the position is calculated as an offset from the previous mismatch in the read.

[0024] In some implementations, the hardware processor circuit is further configured to determine, by one or more hardware processing circuits, whether an offset is greater than the maximum encodable value, and based on determining that the offset is greater than the maximum encodable value, insert at least one false mismatch between a particular mismatch and a preceding mismatch by the hardware processing circuit.

[0025] In some implementations, the hardware processor circuit is further configured to execute, by the hardware processing circuit, an operation including encoding a list of positions of a reference array corresponding to each position of a mismatch with respect to the reference array using an entropy reduction encoding process based on determining that the number of mismatches does not meet a predetermined mismatch threshold number.

[0026] In some implementations, the hardware processor circuit is further configured to execute, by the hardware processing circuit, an operation including encoding at least a portion of a read record using entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to a reference array.

[0027] In some implementations, the hardware processing circuit includes one or more field programmable gate arrays (FPGAs).

[0028] According to another innovative aspect of the present disclosure, a computer implementation method for compressing genome sequence data generated by a sequencing machine, wherein the genome sequence data includes reads of nucleotide or base sequences aligned with a reference sequence, thereby creating aligned reads, and the aligned reads are stored as a list of reads in an initial file. In one aspect, the method may include the actions of determining, for each aligned read, whether the read is fully mapped, incompletely mapped, or not mapped to the reference sequence, and encoding the read as a result of the determination, wherein reads determined to be fully mapped are encoded by a first encoding process, and reads determined to be not mapped are encoded by a second encoding process, wherein for each incompletely mapped read, the number of mismatches between the read and the reference sequence is compared with a threshold, and in the encoding process, if the read is determined to be incompletely mapped... A defined read is encoded by a second or third encoding process, and if the number of mismatches is greater than a threshold, an incompletely mapped read is encoded by the second encoding process; if the number of mismatches is less than a threshold, an incompletely mapped read is encoded by the third encoding process, in which each nucleotide or base of the read is encoded individually; the first and third encoding processes each include a separate set of descriptors, each set of descriptors uniquely representing a read associated with the corresponding encoding process; and each of the first and third encoding processes is an encoding process for reducing source entropy.

[0029] Other embodiments include corresponding systems, apparatus, and computer programs for performing actions in the manner disclosed herein, such as those defined by instructions encoded on a computer-readable storage device.

[0030] These and other versions may optionally include one or more of the following features: For example, in some implementations, the determination step may include a further determination of whether a read is globally mapped or locally mapped in a reference sequence when it is determined that the read is incompletely mapped in the reference sequence and has a mismatch number smaller than a threshold; the third encoding process includes a first encoding subprocess and a second encoding subprocess, where reads determined to be globally mapped are encoded by the first encoding subprocess, and reads determined to be locally mapped are encoded by the second encoding subprocess, where the first and second encoding subprocesses include separate sets of descriptors, each set of descriptors uniquely representing a read associated with the corresponding encoding subprocess.

[0031] In some implementations, the descriptor of the first encoding subprocess may include the alignment start position in the reference array, the read length, and a list of mismatches due to symbol substitution, and the descriptor of the second encoding subprocess may include the local alignment start position in the reference array, the read length, a list of mismatches due to symbol substitution, and the length of the clipped portion of the read that is not part of the alignment.

[0032] In some implementations, during the encoding process, the clipped portion of the read that will be encoded by a second encoding subprocess is concatenated, and each nucleotide or base in the clipped portion is encoded individually.

[0033] In some implementations, during the encoding process, each mismatch in an incompletely mapped read is encoded into a single byte.

[0034] In some implementations, during the encoding process, each mismatch in an incompletely mapped read is encoded using the first two bits of a byte to encode an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or base in the reference sequence, and the last six bits of a byte to encode the position of the mismatch in the reference sequence, which is calculated as an offset from the preceding mismatch in the read.

[0035] In some implementations, during the encoding process, if the calculated offset between an assigned mismatch and a preceding mismatch is greater than the maximum encoding value, at least one false mismatch is inserted between the two mismatches until any offset between each of those mismatches and at least one false mismatch is less than the maximum encoding value, and a false mismatch is defined as a mismatch in which a bit of one byte is used to encode the mismatch or to encode a nucleotide or base equal to the corresponding reference nucleotide or reference base in the reference sequence.

[0036] In some implementations, the initial step of dividing the list of reads into blocks of reads involves each block beginning with a header containing the information needed to decode the block, and this compression method is performed block by block.

[0037] In some implementations, the read blocks have the same block size.

[0038] In some implementations, the final step in providing the compressed file includes a list of encoded reads, which are stored in the compressed file in the same order as the reads stored in the initial file.

[0039] In some implementations, this threshold is equal to 31.

[0040] In some implementations, the process involves determining whether each aligned read contains at least one mismatch that corresponds to a case where the sequencing machine was unable to call any base or nucleotide.

[0041] In some implementations, for each read containing at least one mismatch, which corresponds to a case where the sequencing machine was unable to call any base or nucleotide, the steps include determining the number of such mismatches and comparing that number with a reference threshold.

[0042] In some implementations, if the number of such mismatches is greater than a reference threshold during the encoding process, each nucleotide or base of the read to be encoded by the second encoding process is individually encoded into 4 bits; if the number of such mismatches is less than a reference threshold, each nucleotide or base of the read to be encoded by the second encoding process is individually encoded into 2 bits; and the encoding process further includes encoding a list of positions along a reference sequence, where the positions correspond to the positions of such mismatches in the reference sequence. [Brief explanation of the drawing]

[0043] [Figure 1] This is a flowchart illustrating the steps of the compression method according to the present invention. [Figure 2] This figure shows an apparatus for carrying out the steps of the compression method according to the present invention. [Figure 3] This figure shows the first example of a read globally mapped to a reference array. [Figure 4] This figure shows a second example of a globally mapped read in a reference array where a false mismatch must be inserted. [Modes for carrying out the invention]

[0044] The genomic sequences referred to in this invention include, for example, nucleotide sequences, deoxyribonucleic acid (DNA) sequences, ribonucleic acid (RNA) sequences, and amino acid sequences, for example, but not limited to these. While the description herein is quite detailed with respect to genomic information in the form of nucleotide sequences, it will be understood by those skilled in the art that, although there are some modifications, the compression method according to the present invention can be carried out for other genomic sequences.

[0045] Genome sequencing information is generated by sequencing machines in the form of sequences of nucleotides (or more generally, bases) represented by strings from a defined vocabulary. The smallest vocabulary is represented by five symbols (A, C, G, T, N) representing the four types of nucleotides present in DNA: adenine, cytosine, guanine, and thymine. In RNA, thymine is replaced by uracil (U). N indicates that the sequencing machine was unable to call any base, and therefore the entity at that position cannot be determined.

[0046] The nucleotide sequences generated by a sequencing machine are called “reads.” Sequence reads can be tens to thousands of nucleotides long. Some techniques generate sequence reads in pairs, where one read in a pair originates from one DNA strand and the second read from the other. Throughout this disclosure, a “reference sequence” is any sequence that aligns / maps a nucleotide or base sequence generated by a sequencing machine. An example of such a reference sequence is actually a reference genome, i.e., a sequence assembled by a scientist as a representative example of a set of gene species. However, reference sequences may also consist of synthetic sequences created simply to improve the compressibility of reads, taking into account further processing of the reads. Sequencing machines can introduce errors into sequence reads, in particular the use of incorrect symbols to represent nucleic acids or bases that actually exist in the sequenced sample (i.e., representing different nucleic acids). This is commonly called a substitution error or “mismatch.”

[0047] The present invention relates to a reference-based compression method that receives a read of a nucleotide or base sequence as input, such reads are pre-aligned to a reference sequence to create aligned reads. The aligned reads are then stored as a list of reads in an initial file. The method for aligning the reads and, once aligned, storing them in the initial file is not important to the present invention and is not the subject of this disclosure. Each read is then encoded as a list of its position on the reference sequence and the differences from the reference sequence. Each read can then be reconstructed from the alignment encoding information and the reference sequence by appropriate decompression software configured according to the present invention.

[0048] Preferably, alignment software that processes and aligns reads to a reference sequence before providing them as input to the compression software and apparatus does not consider certain types of errors introduced in the sequence reads, such as insertion or deletion errors. An insertion error is the insertion of one or more additional symbols in one sequence read that do not actually refer to any nucleic acids. A deletion error is the deletion from one or more sequence reads consisting of one or more symbols that actually represent nucleic acids present in the sequenced sample. More precisely, in the case of insertion or deletion errors in the assigned sequence reads, the alignment software will consider the resulting incorrect nucleic acid as a substitution error, also known as a “mismatch.” This preferred choice for the alignment software configuration allows for faster subsequent coding and provides a better compromise between speed and compression ratio.

[0049] The alignment software provides the compression software and device with a corresponding read record for each read. Each read record includes at least the following information: the absolute start position of the aligned read relative to the reference sequence, the length of the read, the type of read alignment, the number of mismatches identified within the read, and the relative position of any possible mismatches within the read (if necessary).

[0050] Herein, the compression method according to the present invention will be described with reference to Figure 1. The method is performed, for example, by the apparatus 20 shown in Figure 2. The apparatus comprises at least one processor 22 and one memory 24 operably connected to the processor 22 to form a computing device. The memory 24 may store computer program code or software 26 including computer executable instructions, which, when executed by the processor 22, cause the processor 22 to perform an operation including the steps of the compression method according to the present invention.

[0051] An initial file in which the aligned reads are stored as a list of reads is stored, for example, in the memory of device 20. Returning to Figure 1, the method preferably includes an initial step 2 of dividing the initial list of aligned reads into blocks of reads. Typically, the list of aligned reads is divided into blocks of 50,000 reads, and this particular value is not to be construed as limiting the scope of the invention, which may be applied in the same way as other values. Preferably, the blocks of reads have the same block size. Each block of reads begins with a header containing information necessary to decode the block, such as, for example, the size of the bytes of the block's contents, and / or an identifier for the block or its contents, and / or the number of reads contained in the block. This enables support for concatenating compressed files and streaming capabilities (each block of reads contains all the information necessary to decode the reads in the block). In addition, the compression method can then run any number of blocks, which enables multithreaded processing of the blocks of reads, thereby enabling parallelization in processing time and some resulting gains. If all reads in a given block have the same length, the read lengths are also stored in the header; otherwise, a list of each read length is explicitly stored during the compression process.

[0052] Each read record contains information about the type of read alignment. Typically, two main types of alignment can be identified: complete alignment and incomplete alignment, as well as an additional type corresponding to “unmapped” reads. “Incomplete alignment” means that at least a portion of the read matches a portion of the reference array, while the read contains at least one mismatch other than N (by this definition, an incompletely mapped read may contain one or more Ns if it also contains one or more other mismatches). In an exemplary embodiment, each read record begins with the following bit flags, each bit flag having one value between two possible values: - A first bit flag indicating forward or reverse direction for the reference array, - A second bit flag indicating whether the alignment is perfect or not. - A third bit flag indicating whether the read contains at least one N, - A fourth bit flag indicating whether the location information is encoded in 16 bits or 32 bits.

[0053] Steps 4-12 below are executed sequentially on each block of reads, and then sequentially on each read within that block.

[0054] The method includes, for each aligned read, the next step 4, determining whether the read is fully mapped, incompletely mapped, or not mapped at all in the reference sequence. For each incompletely mapped read, this determining step 4 may include comparing the number of mismatches between the read and the reference sequence to a threshold (4A). In a preferred embodiment, the threshold is equal to 31, although this should not be construed as limiting the scope of the invention. This particular value is deliberately chosen to provide the best possible compromise for storing the number of mismatches in a sufficiently compact manner, as will be better understood later with respect to step 12. In fact, statistically, in most cases, incompletely mapped reads have fewer than 31 mismatches. The principle behind this choice is to encode the most frequent cases in the most compact way, while still having a very small number of reduced cases. If a read is determined to be incompletely mapped with a number of mismatches less than the threshold, determining step 4 also includes a further determination of whether the read is globally mapped or locally mapped in the reference sequence. A "globally mapped read" is an incompletely mapped read in which the entire sequence, including the start and end of the read, is incompletely mapped to the reference sequence. A "locally mapped read" is an incompletely mapped read that contains a segment of nucleotides or bases that is incompletely mapped to the reference sequence. Therefore, that segment of nucleotides or bases corresponds to part of the first read.

[0055] Preferably, the method further includes step 6 determining, for each aligned read, whether the read contains at least one N, i.e., whether the read contains at least one mismatch corresponding to a case in which the sequencing machine could not call any base or nucleotide. The method then includes step 8 determining the number of such N mismatches for each read containing at least one N, and step 10 comparing the number of N mismatches to a reference threshold. In a preferred embodiment, the reference threshold is equal to 31, although this should not be construed as limiting the scope of the invention.

[0056] Whatever the result of step 4 to determine, the method includes at least the following step 12 of encoding the read by the determination. More precisely, reads determined to be fully mapped to a reference sequence are encoded by a first encoding process, whether they contain no N or have a number of N smaller than the reference threshold. Reads determined to be unmapped, or reads determined to be fully mapped but with a number of N greater than the reference threshold, are encoded by a second encoding process, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned or not. Reads determined to be incompletely mapped are encoded by a second or third encoding process. More precisely, reads determined to be incompletely mapped with a number of mismatches greater than the threshold are encoded by a second encoding process. If a read is determined to be incompletely mapped with a number of mismatches smaller than the threshold, and the read contains no N or has a number of N smaller than the reference threshold, the read is encoded by a third encoding process. Otherwise, i.e., if the read has a number N greater than the reference threshold, the read is encoded by a second encoding process.

[0057] Whether a given read is determined to be fully mapped, incompletely mapped, or not mapped, if the read contains at least one N but has a number of N less than a reference threshold, the encoding step 12 includes encoding a list of positions along a reference sequence, where the positions correspond to the positions of N in the reference sequence. The list of positions is then stored in the memory of a computing device, which implements a compression method. If a read contains at least one N but has a number of N less than a reference threshold and is to be encoded by a second encoding process, each nucleotide or base of the read is encoded individually with 2 bits.

[0058] If a read contains at least one N but has a number of Ns greater than the reference threshold, the read is encoded in any case by a second encoding process, where each nucleotide or base of the read is individually encoded with 4 bits. In this case, the encoding step 12 does not involve encoding and storing a list of the positions of Ns in the reference sequence. In fact, each N mismatch is encoded directly by the second encoding process in much the same way as the other nucleotides or bases of the read.

[0059] The first and third coding processes each contain a separate set of descriptors. Each set of descriptors uniquely represents a read associated with the corresponding coding process, and each of the first and third coding processes is an information entropy reduction coding process. More precisely, the third coding process includes a first coding subprocess and a second coding subprocess. Incompletely mapped reads determined to be globally mapped during step 4 are coded by the first coding subprocess. Incompletely mapped reads determined to be locally mapped during step 4 are coded by the second coding subprocess. The first and second coding subprocesses each contain a separate set of descriptors, and each set of descriptors uniquely represents a read associated with the corresponding coding subprocess.

[0060] Alignment information, which is encoded for each read and allows for the reconstruction of the entire read sequence during data decompression, then depends on the corresponding encoding process or subprocess used for that read. For example, the descriptor used for the first encoding process is: ○ The absolute start position of a read that is fully mapped to a reference array (encoded in 16 bits or 32 bits), ○ The length of the read (encoded by differential coding with respect to the length of the preceding read, having a variable-length code in the range of 2 bits to 34 bits), It is possible.

[0061] The descriptor used in the first coding subprocess is: ○ The absolute start position of a read that is incompletely mapped with respect to a reference array (encoded in 16 bits or 32 bits), ○ The length of the read (encoded by differential coding with respect to the length of the preceding read, having a variable-length code in the range of 2 bits to 34 bits), ○ A list of lead mismatches, It is possible.

[0062] The descriptor used for the second encoding subprocess is: ○ The absolute start position of the incompletely mapped portion of the read with respect to the reference sequence, also called the local alignment start position (encoded in 16 bits or 32 bits), ○ The length of the read (encoded by differential coding with respect to the length of the preceding read, having a variable-length code in the range of 2 bits to 34 bits), ○ A list of lead mismatches, ○ The length of the clipped portion of a read that is not part of the alignment (encoded in 8 bits for each clipped portion), It is possible.

[0063] Preferably, the list of mismatches encoded in the first and second subprocesses includes a header (a bit flag encoded in one byte). The first five bits of the byte are used to encode the number of mismatches in the read (in a preferred embodiment where the threshold is equal to 31, the number is in the range of [0 to 31]). Then, one bit can be used to encode whether an incompletely mapped read is globally mapped or locally mapped. Another bit can be used to encode whether a 2-bit mode is enabled for the second encoding process. The last bit can be used to encode whether a 4-bit mode is enabled for the second encoding process. Preferably, for each read encoded by the second encoding subprocess during the encoding step 12, the clipped portion of the read (i.e., the portion that is not part of the local alignment) is concatenated, and each nucleotide or base of the clipped portion is encoded individually. In a preferred implementation, each nucleotide or base of such a clipped portion of the read is encoded individually with 2 bits.

[0064] In a preferred implementation, each mismatch encoded in the list of incompletely mapped read mismatches (i.e., encoded by the first or second encoding subprocess) is encoded with one byte. More precisely, each mismatch in the incompletely mapped read that will be encoded by the first or second encoding subprocess may be encoded as follows: ○ The first two bits of a byte are used to encode alternative nucleotides or bases present in the read, instead of the corresponding reference nucleotides or bases in the reference sequence. ○ The last six bits are used to encode the position of the mismatch in the reference array, which is calculated as an offset from the preceding mismatch of the read (the relative position of the mismatch, excluding the first mismatch of the read whose absolute position is encoded). Therefore, the range of this offset encoded in 6 bits is [0 to 63].

[0065] Figure 3 provides an example of encoding mismatches in a read by the first encoding subprocess. This read is an incompletely mapped read that is globally mapped in the reference array. This read has two mismatches. ○ In the substitution of an A nucleotide in the reference sequence by a T nucleotide in the read, the first mismatch located at the 12th position in the read, ○ In the substitution of a C nucleotide in the reference sequence by a G nucleotide in the read, there is a second mismatch located at the 21st position in the read, It holds.

[0066] Next, the list of lead mismatches is: ○ <12,T>, that is, the value "12" which corresponds to the absolute position of the first mismatch in the read, and ○ <9,G>, that is, the value "9" corresponding to the relative position of the second mismatch in the read, i.e., the offset between the second mismatch and the first mismatch, It is encoded as follows.

[0067] <12,T> may be converted to the value "51" (encoded in 1 byte), for example, and <9,G> may be converted to the value "38" (encoded in 1 byte). Such 1-byte encoding is Offset position × 4 + nucleotide value (A=0, C=1, G=2, T=3) It is obtained by [method].

[0068] Preferably, for each incompletely mapped read encoded by the first or second encoding subprocess, if the calculated offset between a given mismatch and a preceding mismatch of the read is greater than the maximum encoding value, at least one "false" mismatch is inserted between the two mismatches until any offset between each of the two mismatches and at least one "false" mismatch is less than the maximum encoding value. A "false" mismatch is defined as a mismatch in which a bit of the byte used to encode the mismatch encodes a nucleotide or base equal to the corresponding reference nucleotide or base in the reference sequence. In preferred embodiments, although not to be construed as limiting the scope of the invention, the maximum encoding value is equal to 63, corresponding to the maximum value that can be encoded with 6 bits.

[0069] Figure 4 provides an example of the encoding of read mismatches by the first encoding subprocess in a case where it is necessary to insert a “false” mismatch. This read is an incompletely mapped read that is globally mapped in the reference array. This read has two mismatches, ○ In the substitution of an A nucleotide in the reference sequence by a T nucleotide in the read, the first mismatch located at the 22nd position in the read, ○ In the substitution of a C nucleotide in the reference sequence by a G nucleotide in the read, there is a second mismatch located at position 134 in the read, It holds.

[0070] The positional offset between the second mismatch and the first mismatch is 112, which is greater than the maximum codeable value of 63. Therefore, the "false" mismatch must be inserted between the two mismatches, so that any offset between each mismatch and the "false" mismatch is less than the maximum codeable value. The "false" mismatch using a T nucleotide (corresponding to the "actual" T nucleotide in the reference sequence) is inserted, for example, at position 85 in the read. The calculated positional offset between the "false" mismatch and the first mismatch is 63, which corresponds to the maximum codeable value. The calculated positional offset between the second mismatch and the "false" mismatch is 49, which is less than 63.

[0071] Next, the list of lead mismatches is: ○ <22,T>, that is, the value "22" which corresponds to the absolute position of the first mismatch in the read. ○ <63,T>, that is, the value "63" corresponding to the relative position of the "false" mismatch in the read, i.e., the offset between the "false" mismatch and the first mismatch, and ○ <49,G>, that is, the value "49" corresponding to the relative position of the second mismatch in the read, i.e., the offset between the second mismatch and the "false" mismatch, It is encoded as follows.

[0072] <22,T> may be converted to the value "91" (encoded in 1 byte), <63,T> may be converted to the value "255" (encoded in 1 byte), and <49,G> may be converted to the value "198" (encoded in 1 byte). Such 1-byte encoding is Offset position × 4 + nucleotide value (A=0, C=1, G=2, T=3) It is obtained by [method].

[0073] The method includes a final step 14 which provides a compressed file containing a list of encoded reads. The encoded reads are stored in the compressed file in the same order as the reads stored in the initial uncompressed file. Each read can then be reconstructed from alignment encoding information and a reference sequence by appropriate decompression software and / or method configured according to the present invention.

[0074] While an exemplary architecture of computing device 20 (shown in Figure 2 for illustrative purposes) has been described, the technologies of the present invention disclosed herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the computer program code may be stored on a computer medium and executed by a hardware processing unit including one or more processors, as in the case of device 20 in Figure 2. As used herein, the term “processor” should be understood to include one or more processing devices, including signal processors, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other types of processing circuits, and parts or combinations of such circuit elements. Furthermore, as used herein, the term “memory” is intended to include any combination of electronic memory associated with a processor, such as random access memory (RAM), read-only memory (ROM), or other types of memory.

[0075] Accordingly, software instructions or code for implementing the methodologies and protocols described herein may be stored in one or more associated memory devices, such as ROM, fixed memory, or removable memory, and may be loaded into RAM and executed by a processor when ready for use.

[0076] The technology described herein can be implemented in a wide range of devices or apparatus, including, for example, mobile phones, computers, servers, tablets, and similar devices.

[0077] Exemplary embodiments of the present invention have been described herein with reference to the accompanying drawings, but the present invention is not limited to the exact embodiments shown in the drawings, and various other changes and modifications can be made by those skilled in the art without departing from the scope or spirit of the invention.

[0078] Statistical and numerical examples of the compression method according to the present invention The following comparative examples were performed with respect to an uncompressed data file containing 48 million nucleotide reads or sequences. ○ Uncompressed data file size: 35,770 MB (megabytes) ○ File size compressed with gzip software: 6,649MB ○ File size compressed with non-reference-based SPRING software: 1,402MB ○ File size compressed by the reference-based compression method according to the present invention: 1,179MB ○ Compression time with non-reference-based SPRING software: 1,722 seconds ○ Compression time in the reference-based compression method according to the present invention: 181 seconds ○ Average size of uncompressed data files (ASCII encoding) per bit / nucleotide: 8 bits / nucleotide ○ Average bit / nucleotide size of compressed files using coding that conforms to the four possible characters A, T, C, G: 2 bits / nucleotide ○ Average bit / nucleotide size of files compressed by the reference-based compression method according to the present invention: 0.33 bits / nucleotide

[0079] The numerical examples shown above demonstrate that the present invention enables high-speed compression and decompression while providing a high compression ratio.

Claims

1. A method for compressing genome sequence data, wherein the method is Acquiring read records using one or more computers, The one or more computers determine whether the read record corresponds to a read that is fully mapped to a reference sequence or a read that is not fully mapped to a reference sequence. Based on the determination by one or more computers that the read record corresponds to a read that is incompletely mapped to the reference sequence, the one or more computers determine whether the number of mismatches of the incompletely mapped reads does not exceed a predetermined mismatch threshold. A method comprising, based on the determination that the number of mismatches does not exceed a predetermined mismatch threshold number, (i) obtaining an offset from a preceding mismatch smaller than the maximum encodeable offset value by one or more computers, and (ii) encoding each mismatch of the incompletely mapped read and the offset of the read from the preceding mismatch into a record having a size of one byte by one or more computers.

2. The method described above is: The one or more computers determine that the number of mismatches in the incompletely mapped reads exceeds a predetermined mismatch threshold. The method according to claim 1, further comprising encoding each base of the incompletely mapped read individually based on the determination that the number of mismatches in the incompletely mapped read exceeds a predetermined mismatch threshold number.

3. Each lead record is: Data indicating the absolute start position of a read aligned with the aforementioned reference sequence, The data indicating the length of the lead, Data indicating whether the read is fully mapped or incompletely mapped, Data showing the number of mismatches identified in the aforementioned read, The method according to claim 1, further comprising data indicating the relative positions of each of the mismatches in the lead.

4. Encoding each mismatch of the incompletely mapped read into a record having a size of one byte means that for each particular mismatch, The first two bits of the one byte are encoded by one or more computers to include data that represents an alternative nucleotide or base present in the read, instead of the corresponding reference nucleotide or reference base in the reference sequence. The method according to claim 1, comprising encoding the remaining six bits of the one byte so that it includes data for displaying the offset, using one or more computers.

5. The aforementioned method, One or more computers determine whether the offset is greater than the maximum codeable value, The method according to claim 4, further comprising: determining that the offset is greater than the maximum codeable value, one or more computers inserting at least one false mismatch between the specific mismatch and the preceding mismatch.

6. The aforementioned method, The method according to claim 1, further comprising, based on the determination that the read record corresponds to a read that is fully mapped to the reference sequence, one or more computers encoding at least a portion of the read record using information entropy reduction coding.

7. A hardware processor including hardware processing circuits configured to perform one or more operations, wherein the one or more operations are The aforementioned hardware processing circuit acquires read records, The hardware processing circuit determines whether the read record corresponds to a read that is fully mapped to the reference array or a read that is not fully mapped to the reference array. Based on the determination by the hardware processing circuit that the read record corresponds to a read that is incompletely mapped to the reference array, one or more computers determine whether the number of mismatches of the incompletely mapped reads does not exceed a predetermined mismatch threshold. A hardware processor comprising: (i) obtaining an offset from a preceding mismatch smaller than the maximum codeable offset value by the hardware processing circuit, based on the determination that the number of mismatches does not exceed a predetermined mismatch threshold; and (ii) encoding each mismatch of the incompletely mapped read and the offset of the read from the preceding mismatch into a record having a size of one byte by the hardware processing circuit.

8. Each lead record is: Data indicating the absolute start position of a read aligned with the aforementioned reference sequence, The data indicating the length of the lead, Data indicating whether the read is fully mapped or incompletely mapped, Data showing the number of mismatches identified in the aforementioned read, The hardware processor according to claim 7, comprising data indicating the relative position of the mismatch in the read.

9. Encoding each mismatch of the incompletely mapped read into a record having a size of one byte means that for each particular mismatch, The hardware processing circuit encodes the first two bits of the one byte to include data that represents an alternative nucleotide or base present in the read, instead of the corresponding reference nucleotide or reference base in the reference sequence. The hardware processor according to claim 7, comprising: encoding the remaining six bits of the one byte so as to include data for displaying the offset, using the hardware processing circuit.

10. The hardware processor circuit is The hardware processing circuit determines whether the offset is greater than the maximum codeable value, The hardware processor according to claim 9, further configured to perform an operation including, based on the determination that the offset is greater than the maximum codeable value, inserting at least one false mismatch between the specific mismatch and the preceding mismatch.

11. The hardware processor circuit is The hardware processor according to claim 7, further configured to perform an operation on which, based on determining that the read record corresponds to a read that is fully mapped to the reference array, the hardware processing circuit is configured to perform an operation that includes encoding at least a portion of the read record using information entropy reduction coding.

12. The hardware processor according to claim 7, wherein the hardware processing circuit includes one or more field-programmable gate arrays (FPGAs).

13. The one or more operations are: The hardware processing circuit determines that the number of mismatches in the incompletely mapped reads exceeds the predetermined mismatch threshold number, The hardware processor according to claim 7, further comprising encoding each base of the incompletely mapped read individually based on the determination that the number of mismatches in the incompletely mapped read exceeds a predetermined mismatch threshold number.

14. A system for compressing genome sequence data, wherein the system is The system includes one or more computers and one or more storage devices for storing instructions, wherein when an instruction is executed by one or more computers, the instructions are stored in the one or more computers. The acquisition of read records by one or more computers, The one or more computers determine whether the read record corresponds to a read that is fully mapped to a reference sequence or a read that is not fully mapped to a reference sequence. Based on the determination by one or more computers that the read record corresponds to a read that is incompletely mapped to the reference sequence, the one or more computers determine whether the number of mismatches of the incompletely mapped reads does not exceed a predetermined mismatch threshold. A system that, based on the determination that the number of mismatches does not exceed a predetermined mismatch threshold, is operable to cause one or more computers to perform an operation including: (i) obtaining an offset from a preceding mismatch smaller than the maximum codeable offset value; and (ii) encoding each mismatch of the incompletely mapped read and the offset of the read from the preceding mismatch into a record having a size of one byte.

15. Each lead record is: Data indicating the absolute start position of a read aligned with the aforementioned reference sequence, The data indicating the length of the lead, Data indicating whether the read is fully mapped or incompletely mapped, Data showing the number of mismatches identified in the aforementioned read, The system according to claim 14, further comprising data indicating the relative positions of each of the mismatches in the lead.

16. Encoding each mismatch of the incompletely mapped read into a record having a size of one byte means that for each particular mismatch, Encoding the first two bits of the one byte by one or more computers to include data that represents an alternative nucleotide or base present in the read, instead of the corresponding reference nucleotide or reference base in the reference sequence, The system according to claim 14, comprising encoding the remaining six bits of the one byte so that it includes data for displaying the offset, using one or more computers.

17. The above operation is, The one or more computers determine whether the offset is greater than the maximum codeable value, The system according to claim 16, further comprising: determining that the offset is greater than the maximum codeable value; one or more computers inserting at least one false mismatch between the specific mismatch and the preceding mismatch.

Images