Method for compression of genome sequence data

Abstract

To overcome defects of conventional compression methods by enabling high speed compression and decompression and providing a high compression ratio while causing no loss of information.SOLUTION: This invention relates to a reference-based method for the compression of genome sequence data produced by a sequence determination machine. The sequences of nucleotides or bases previously aligned to a reference sequence are determined to be perfectly mapped, imperfectly mapped or unmapped with the reference sequence, and then, coded according to the determination. The determining step comprises comparing, for each imperfectly mapped sequence, the number of mismatches between the reference sequence and a reference sequence with a reference threshold value, and encoding the imperfectly mapped sequences according to distinct encoding processes according to a result of a corresponding comparison method for the compression of genome sequence data produced by a sequence determination machine.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure provides a reference-based compression method that allows for fast compression and decompression while not causing loss of information, and has a high compression ratio. [Background technology]

[0002] Next-generation sequencing machines now cheaply generate vast amounts of sequencing data. Recent systems generate over 6 billion 150-nucleotide sequences in a single 36-hour run—enough to sequence 20 entire human genomes. This opens up many new perspectives for the diagnosis of genetic diseases and for the development of personalized medicine, aiming to adapt treatments based on people's genomic specificities.

[0003] However, this also brings new challenges, especially the cost of storing huge amounts of data. The most used file format for raw (unaligned) sequence data is the FASTQ format, which holds the sequence data (strands consisting of A, C, T, and G nucleotides, also called reads), a quality value (the probability that the sequencing platform made a sequencing error for each nucleotide), and the sequence name. This is usually a simple ASCII text file compressed with the general-purpose text compression scheme LZ (Lempel-Ziv scheme, implemented in the gzip software). However, the use of such compression methods presents several problems: -Low compression ratio due to data redundancy not being fully utilized -Slow compression and decompression accompanied by.

[0004] There are also compression methods specific to FASTQ encoding, divided into reference-based and non-reference-based methods. However, none of them are completely satisfactory, since a) reference-based methods have good compression ratios but are slow, and b) non-reference-based methods are faster but have lower compression ratios. An example of such a non-reference-based method is provided by the software SPRING, which is a reference-free compressor for FASTQ files (World Wide Web address: github.com / shubhamchandak94 / SPRING). However, the compression method provided by the software SPRING has a low compression ratio.

[0005] Among reference-based compression methods, several methods have been proposed that use sequence alignment and aim to be faster with good compression ratios. However, such methods suffer from several problems, the main one being that these problems are not completely eliminated. Such a known reference-based compression method is described, for example, in patent document WO 2018 / 068829(A1). In the described method, after being aligned to one or more reference sequences, the nucleotide sequence is classified by matching the degree of accuracy (thereby creating classes of aligned reads) and then coded as syntax elements of multiple layers using a different source model and entropy coder for each layer into which the data is divided. Thus, classes of data are coded separately and composed of syntax elements of different layers, each layer containing a descriptor that uniquely represents the classification and aligned reads of that layer. This method aims to obtain separate information sources with reduced information entropy, thereby enabling increased compression performance and selective access to specific classes of compressed data. However, such compression methods reorder the reads in an order different from the order obtained at the end of the read alignment step (i.e., the reads are reordered by their classes). Some information is then lost in the compression process, especially in the initial sequence ordering. Therefore, the reproducibility of some analysis results may be affected, since some downstream analysis software may depend on the order of the reads. In addition, decompressing data in an order different from the initial order of the reads makes it much more difficult to verify that the uncompressed file is identical to the initial file. Furthermore, such compression methods are relatively slow, especially when compared to state-of-the-art non-reference-based compression methods. Summary of the Invention [Means for solving the problem]

[0006] The present disclosure solves the problems of existing prior art solutions by providing a system, method, computer program, and hardware circuit for compression of genomic sequence data. In one aspect, a computer-implemented method for compression of genomic sequence data generated by a sequencing machine, wherein the genomic sequence data includes nucleotide or base sequence reads aligned to a reference sequence, thereby creating aligned reads, and the aligned reads are stored as a list of reads in an initial file, comprises: - for each aligned read, determining whether the read is completely or incompletely mapped to the reference sequence, or whether the read is not mapped to the reference sequence; - encoding the reads according to said determination, wherein the reads determined to be fully mapped are encoded by a first encoding process and the reads determined to be unmapped are encoded by a second encoding process; - determining comprises, for each incompletely mapped read, comparing the number of mismatches between the read and the reference sequence with a threshold; - in the encoding step, reads determined to be incompletely mapped are encoded by the second encoding process or the third encoding process, and incompletely mapped reads are encoded by the second encoding process when the number of mismatches is greater than a threshold, and incompletely mapped reads are encoded by the third encoding process when the number of mismatches is less than a threshold; in said second encoding process, each nucleotide or base of the read is encoded individually; - the first encoding process and the third encoding process include separate sets of descriptors, each set of descriptors uniquely representing a lead associated with the corresponding encoding process, and each of the first encoding process and the third encoding process is an information source entropy reduction encoding process.

[0007] The present disclosure overcomes the shortcomings of conventional compression methods by enabling fast compression and decompression and providing high compression ratios while not causing information loss. More specifically, the present disclosure focuses on encoding the most frequent cases in the most compact way, even if this means employing a degraded encoding mode for the rarest and least frequent cases. This leads to a significant increase in compression performance. Moreover, due to the genomic information representation format used by the present disclosure, compression performed by the methods described herein is faster. Lastly, the present disclosure preserves the initial order of reads and does not reorder reads by read class. As a result, no information is lost during the process, which allows for easier downstream analysis and efficient consistency checks after the decompression step.

[0008] These and other features and advantages of the present disclosure will become more apparent from the accompanying drawings and the detailed description that follows. Additionally, although thresholds may be referred to herein as being exceeded or not exceeded, it is understood that such thresholds may be employed conceptually to determine whether such thresholds are met, met, or otherwise detected, regardless of whether the numbers or values ​​used to perform those threshold evaluations are described using positive or negative values.

[0009] According to one innovative aspect of the present disclosure, a method for compressing genome sequence data is disclosed.In one aspect, this method can include one or more operations performed by one or more computers through the execution of software instructions, and these operations include: by one or more computers, obtain read records; by one or more computers, determine whether read records correspond to the reads that are completely mapped to reference sequence or incompletely mapped to reference sequence, based on determining that read records correspond to the reads that are incompletely mapped to reference sequence; by one or more computers, determine whether the number of mismatches of incompletely mapped reads meets a predetermined mismatch threshold number; and based on determining that the number of mismatches meets a predetermined mismatch threshold number, by one or more computers, encode each mismatch of incompletely mapped reads into a record with a size of 1 byte.

[0010] Other aspects include corresponding systems, apparatus, and computer programs for performing the actions of the methods as disclosed herein, as defined by instructions encoded on a computer-readable storage device.

[0011] These and other versions may optionally include one or more of the following features. For example, in some implementations, determining, by the one or more computers, whether the number of mismatches in the incompletely mapped reads satisfies a predetermined mismatch threshold number can include determining, by the one or more computers, whether the number of mismatches in the incompletely mapped reads is greater than a predetermined mismatch threshold number.

[0012] In some implementations, each read record can include data indicating the absolute starting position of the aligned read with respect to the reference sequence, data indicating the length of the read, data indicating whether the read is completely or incompletely mapped, data indicating the number of mismatches identified within the read, and data indicating the relative position of each of the possible mismatches within the read.

[0013] In some implementations, encoding each mismatch of an incompletely mapped read into a record having a size of one byte includes, for each particular mismatch, encoding, by one or more computers, the first two bits of the byte to contain data indicating an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or reference base in the reference sequence, and encoding the remaining six bits of the byte to contain data indicating the position of the mismatch in the reference sequence, the position being calculated as an offset from the previous mismatch in the read.

[0014] In some implementations, the method may further include determining, by one or more computers, whether the offset is greater than a maximum encodable value; and, based on determining that the offset is greater than the maximum encoded value, inserting, by the one or more computers, at least one false mismatch between the particular mismatch and the preceding mismatch.

[0015] In some implementations, the method may further include, based on determining that the number of mismatches does not meet a predetermined mismatch threshold number, encoding, by one or more computers, a list of positions in the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

[0016] In some implementations, the method may further include encoding, by one or more computers, at least a portion of the read record using information entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to the reference sequence.

[0017] In some implementations, one or more computers may include one or more hardware processors.

[0018] In some implementations, the one or more hardware processors may include one or more field programmable gate arrays (FPGAs).

[0019] In some implementations, the method for compressing genome sequence data can be performed by one or more hardware processors. In such implementations, the hardware processor can include a hardware processing circuit configured to perform one or more operations. In one aspect, the operation can include: acquiring a read record, by the hardware processing circuit; determining, by the hardware processing circuit, whether the read record corresponds to a read that is completely mapped to a reference sequence or that is incompletely mapped to a reference sequence; determining, by one or more computers, whether the number of mismatches in the incompletely mapped read meets a predetermined mismatch threshold number based on the hardware processing circuit determining that the number of mismatches meets the predetermined mismatch threshold number; and encoding, by the hardware processing circuit, each mismatch in the incompletely mapped read into a record having a size of 1 byte.

[0020] In some implementations, each read record can include data indicating the absolute starting position of the aligned read with respect to the reference sequence, data indicating the length of the read, data indicating whether the read is completely or incompletely mapped, data indicating the number of mismatches identified within the read, and data indicating the relative positions of those possible mismatches within the read.

[0021] In some implementations, determining by the hardware processing circuitry whether the number of mismatches of the incompletely mapped reads satisfies a predetermined mismatch threshold number can include determining by the hardware processing circuitry whether the number of mismatches of the incompletely mapped reads is greater than a predetermined mismatch threshold number.

[0022] In some implementations, encoding each mismatch of an incompletely mapped read into a record having a size of one byte can include, for each particular mismatch encoding, encoding by a hardware processing circuit the first two bits of the byte to contain data indicating an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or reference base in the reference sequence, and encoding the remaining six bits of the byte to contain data indicating the position of the mismatch in the reference sequence, the position being calculated as an offset from the previous mismatch in the read.

[0023] In some implementations, the hardware processor circuitry is further configured to perform operations including: determining, by one or more hardware processing circuits, whether the offset is greater than the maximum codeable value; and, based on determining that the offset is greater than the maximum codeable value, inserting, by the hardware processing circuitry, at least one false mismatch between the particular mismatch and the preceding mismatch.

[0024] In some implementations, the hardware processor circuitry is further configured to perform an operation, based on determining that the number of mismatches does not meet a predetermined mismatch threshold number, including encoding, by the hardware processing circuitry, a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

[0025] In some implementations, the hardware processor circuitry is further configured to perform an operation including, based on determining that the read record corresponds to a read that is fully mapped to the reference sequence, encoding at least a portion of the read record using information entropy reduction encoding by the hardware processing circuitry.

[0026] In some implementations, the hardware processing circuitry includes one or more field programmable gate arrays (FPGAs).

[0027] According to another innovative aspect of the present disclosure, a method for compressing genome sequence data is disclosed. In one aspect, the method includes the following operations: accessing, by one or more processors, a storage device that stores a plurality of read records in a manner that preserves the sequence ordering of the read records generated by the mapping and alignment module; for each specific read record among the plurality of read records, obtaining, by one or more processors, the specific read record; determining, by one or more processors, whether the specific read record corresponds to a read that is completely mapped to a reference sequence or a read that is incompletely mapped to a reference sequence; based on determining, by one or more processors, that the specific read record corresponds to a read that is incompletely mapped to the reference sequence, determining, by one or more processors, whether the incompletely mapped read meets a predetermined mismatch threshold number; based on determining that the number of mismatches meets the predetermined mismatch threshold number, encoding, by one or more processors, each mismatch of the incompletely mapped read into a compressed record having a predetermined compressed record size; and storing, by one or more processors, the compressed record in the storage device while maintaining the sequence ordering of the read records.

[0028] Other aspects include corresponding systems, apparatus, and computer programs for performing the actions of the methods as disclosed herein, as defined by instructions encoded on a computer-readable storage device.

[0029] These and other versions may optionally include one or more of the following features: For example, in some implementations, each read record of the plurality of read records may include data indicating the absolute start position of the aligned read with respect to the reference sequence, data indicating the length of the read, data indicating whether the read is fully or incompletely mapped, data indicating the number of mismatches identified in the read, data indicating whether the read includes at least one undetermined base N, data indicating the number of undetermined bases N in the read, data indicating whether the read is mapped or unmapped, data indicating the position of the read record in the array of read records output by the mapping and alignment module, and data indicating the relative position of the possible mismatch in the read.

[0030] In some implementations, the predetermined compressed record size is 1 byte.

[0031] In some implementations, n-coding each mismatch of an incompletely mapped read into a record having a size of one byte can include, for each particular mismatch encoding, encoding, by one or more computers, the first two bits of the byte to contain data indicating an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or reference base in the reference sequence, and encoding, by one or more processors, the remaining six bits of the byte to contain data indicating the position of the mismatch in the reference sequence, the position being calculated as an offset from the preceding mismatch in the read.

[0032] In some implementations, the method may further include determining, by one or more processors, whether the offset is greater than a maximum encodable value, and inserting, by the one or more processors, at least one false mismatch between the particular mismatch and the preceding mismatch based on determining that the offset is greater than the maximum encoded value.

[0033] In some implementations, the method may further include, based on determining that the number of mismatches does not meet a predetermined mismatch threshold number, encoding, by one or more processors, a list of positions in the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

[0034] In some implementations, the method may further include encoding, by one or more processors, at least a portion of the read record using information entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to the reference sequence.

[0035] In some implementations, determining, by one or more computers, whether the number of mismatches in the incompletely mapped reads meets a predetermined mismatch threshold number can include determining, by one or more processors, whether the number of mismatches in the incompletely mapped reads is greater than a reference threshold.

[0036] According to another innovative aspect of the present disclosure, a hardware processor is disclosed. In one aspect, the hardware processor can include hardware processing circuitry configured to perform one or more operations. In one aspect, the operations configured to be performed by the hardware processing circuit include: accessing, by the hardware processing circuit, a storage device that stores a plurality of read records in a manner that preserves the sequence ordering of the read records generated by the mapping and alignment module; for each particular read record among the plurality of read records, obtaining, by the hardware processing circuit, the particular read record; determining, by the hardware processing circuit, whether the particular read record corresponds to a read that is completely mapped to the reference sequence or to an incompletely mapped read to the reference sequence; based on determining, by the hardware processing circuit, that the particular read record corresponds to a read that is incompletely mapped to the reference sequence, determining, by the hardware processing circuit, whether the incompletely mapped read satisfies a predetermined mismatch threshold number; based on determining that the number of mismatches satisfies the predetermined mismatch threshold number, encoding, by the hardware processing circuit, each mismatch of the incompletely mapped read into a compressed record having a predetermined compressed record size; and storing, by the hardware processing circuit, the compressed record in the storage device while maintaining the sequence ordering of the read records.

[0037] These and other versions may optionally include one or more of the following features: For example, in some implementations, each read record among the plurality of read records accessible by the hardware processing circuitry may include data indicating the absolute start position of the aligned read with respect to the reference sequence, data indicating the length of the read, data indicating whether the read is fully or incompletely mapped, data indicating the number of mismatches identified in the read, data indicating whether the read includes at least one undetermined base N, data indicating the number of undetermined bases N in the read, data indicating whether the read is mapped or unmapped, data indicating the position of the read record in the array of read records output by the mapping and alignment module, and data indicating the relative position of the possible mismatch in the read.

[0038] In some implementations, the predetermined compressed record size generated by the hardware processing circuitry may be 1 byte.

[0039] In some implementations, encoding each mismatch of an incompletely mapped read into a compressed record having a size of one byte can include, for each particular mismatch encoding, encoding by a hardware processing circuit the first two bits of the byte to contain data indicating an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or reference base in the reference sequence, and encoding the remaining six bits of the byte to contain data indicating the position of the mismatch in the reference sequence, the position being calculated as an offset from the previous mismatch in the read.

[0040] In some implementations, the hardware processor may further be configured to include a hardware processing circuit configured to perform operations including determining whether the offset is greater than a maximum encodable value and, based on determining that the offset is greater than the maximum encoded value, inserting, by the hardware processing circuit, at least one false mismatch between the particular mismatch and the preceding mismatch.

[0041] In some implementations, the hardware processor may further be configured to include a hardware processing circuit configured to, based on determining that the number of mismatches does not meet a predetermined mismatch threshold number, perform an operation including encoding, by the hardware processing circuit, a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

[0042] In some implementations, the hardware processor may further be configured to include a hardware processing circuit configured to perform an operation including encoding, by the hardware processing circuit, at least a portion of the read record using information entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to the reference sequence.

[0043] In some implementations, determining whether the number of mismatches of the incompletely mapped reads meets a predetermined mismatch threshold number by the hardware processing circuitry comprises: Determining, by a hardware processing circuit, whether the number of mismatches in the incompletely mapped reads is greater than a predetermined threshold number of mismatches.

[0044] According to another innovative aspect of the present disclosure, there is provided a computer-implemented method for compressing genomic sequence data generated by a sequencing machine, the genomic sequence data including nucleotide or base sequence reads aligned to a reference sequence, thereby creating aligned reads, and the aligned reads being stored as a list of reads in an initial file. In one aspect, the method may include the actions of: determining, for each aligned read, whether the read is completely mapped to the reference sequence, incompletely mapped, or unmapped with the reference sequence; and encoding the read according to the determination; wherein a read determined to be completely mapped is encoded by a first encoding process, and a read determined to be unmapped is encoded by a second encoding process; the determining step includes, for each incompletely mapped read, comparing a number of mismatches between the read and the reference sequence with a threshold; and wherein the encoding step determines that the read is incompletely mapped. The incompletely mapped read is encoded by a second encoding process or a third encoding process, and if the number of mismatches is greater than a threshold, the incompletely mapped read is encoded by the second encoding process, and if the number of mismatches is less than a threshold, the incompletely mapped read is encoded by the third encoding process, in which each nucleotide or base of the read is encoded individually, the first encoding process and the third encoding process include separate sets of descriptors, each set of descriptors uniquely representing the read associated with the corresponding encoding process, and each of the first encoding process and the third encoding process is an information source entropy reduction encoding process.

[0045] Other aspects include corresponding systems, apparatus, and computer programs for performing the actions of the methods as disclosed herein, as defined by instructions encoded on a computer-readable storage device.

[0046] These and other versions may optionally include one or more of the following features: For example, in some implementations, the determining step can include a further determination as to whether a read is globally or locally mapped with a reference sequence when the read is determined to be incompletely mapped with the reference sequence and has a number of mismatches less than a threshold, wherein the third encoding process includes a first encoding sub-process and a second encoding sub-process, wherein reads determined to be globally mapped are encoded by the first encoding sub-process and reads determined to be locally mapped are encoded by the second encoding sub-process, wherein the first encoding sub-process and the second encoding sub-process include separate sets of descriptors, each set of descriptors uniquely representing a read associated with a corresponding encoding sub-process.

[0047] In some implementations, the descriptor of the first encoding sub-process may include an alignment start position in the reference sequence, a read length, and a list of mismatches due to symbol substitution, and the descriptor of the second encoding sub-process includes a local alignment start position in the reference sequence, a read length, a list of mismatches due to symbol substitution, and the length of the clipped portion of the read that is not part of the alignment.

[0048] In some implementations, in the encoding step, the clipped portions of the reads to be encoded by the second encoding sub-process are concatenated, and each nucleotide or base of the clipped portions is encoded individually.

[0049] In some implementations, in the encoding step, each mismatch in an incompletely mapped read is encoded into one byte.

[0050] In some implementations, in the encoding step, each mismatch in an incompletely mapped read is encoded using the two first bits of a byte used to encode an alternative nucleotide or base present in the read instead of the corresponding reference nucleotide or reference base in the reference sequence, and the six last bits of the byte used to encode the position of the mismatch in the reference sequence, which is calculated as an offset from the previous mismatch in the read.

[0051] In some implementations, in the encoding step, if the calculated offset between the given mismatch and the preceding mismatch is greater than the maximum encodable value, at least one false mismatch is inserted between the two mismatches until every offset between each of the mismatches and the at least one false mismatch is lower than the maximum encodable value, and a false mismatch is defined as a mismatch in which a bit of a byte is used to encode the mismatch or to encode a nucleotide or base equal to the corresponding reference nucleotide or reference base in the reference sequence.

[0052] In some implementations, the initial step is to divide the list of reads into blocks of reads, each block beginning with a header containing information needed to decode the block, and the compression method is performed block by block.

[0053] In some implementations, the blocks of reads have the same block size.

[0054] In some implementations, the final step of providing a compressed file includes a list of encoded reads, which are stored in the compressed file in the same order as the reads were stored in the initial file.

[0055] In some implementations, the threshold is equal to 31.

[0056] In some implementations, for each aligned read, determining whether the read contains at least one mismatch, corresponding to a case where the sequencing machine was unable to call any bases or nucleotides.

[0057] In some implementations, for each read containing at least one mismatch, corresponding to a case where the sequencing machine was unable to call any base or nucleotide, determining the number of such mismatches and comparing the number to a reference threshold.

[0058] In some implementations, in the encoding step, if the number of such mismatches is greater than the reference threshold, each nucleotide or base of the read to be encoded by the second encoding process is individually coded into 4 bits, and if the number of such mismatches is less than the reference threshold, each nucleotide or base of the read to be encoded by the second encoding process is individually coded into 2 bits, and the encoding step further includes coding a list of positions along the reference sequence, the positions corresponding to the positions of such mismatches in the reference sequence. [Brief explanation of the drawings]

[0059] [Figure 1] FIG. 1 is a flow diagram illustrating an example of a method described herein. [Figure 1A] FIG. 2 is a flow diagram illustrating a more detailed example of the compression method of FIG. [Figure 2] FIG. 1 illustrates an example system for implementing one or more of the compression methods described herein. [Figure 2A] FIG. 2 illustrates another example of a system for implementing the compression methods described herein. [Figure 2B] FIG. 2 illustrates another example of a system for implementing the compression methods described herein. [Figure 3] FIG. 1 is a schematic diagram showing a first example of globally mapped reads with a reference sequence. [Figure 4] FIG. 1 is a schematic diagram showing a second example of globally mapped reads with a reference sequence when a false mismatch must be inserted. [Figure 5] FIG. 1B is a diagram of an example of computing components that can be used to implement a system that performs the compression method of FIGS. 1 and 1A. [Figure 6] FIG. 1 is a diagram of several bar graphs showing experimental results of the present disclosure. [Figure 7] FIG. 10 is a bar graph showing additional experimental results of the present disclosure. [Figure 8] FIG. 10 is a bar graph showing additional experimental results of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0060] Genomic sequences referred to by the present disclosure include, for example, but not limited to, nucleotide sequences, deoxyribonucleic acid (DNA) sequences, ribonucleic acid (RNA) sequences, and amino acid sequences. Although the present disclosure is described in considerable detail herein with respect to genomic information in the form of nucleotide sequences, it will be understood that the compression methods according to the present invention can be implemented with other genomic sequences, with some variations, as will be appreciated by those skilled in the art.

[0061] Genome sequencing information is generated by a sequencing machine in the form of a sequence of nucleotides (or more generally, bases) represented by a character string from a defined vocabulary. The smallest vocabulary is represented by five symbols (A, C, G, T, N) representing the four types of nucleotides present in DNA: adenine, cytosine, guanine, and thymine. In RNA, thymine is replaced by uracil (U). An N indicates that the sequencing machine was unable to call any bases, and therefore the identity of that position is undetermined. Therefore, for purposes of this disclosure, the symbol "N" refers to an undetermined base, and the number of "Ns" in a read refers to the number of undetermined bases in the read.

[0062] Nucleotide sequences generated by a sequencing machine can be referred to as "reads." Sequence reads can be tens to thousands of nucleotides in length. Some techniques generate sequence reads in pairs, where the first read of the pair is from one DNA strand and the second read of the pair is from the other DNA strand. Throughout this disclosure, a "reference sequence" is any sequence to which a read composed of a nucleotide or base sequence generated by a sequencing machine can be aligned / mapped. An example of such a reference sequence may actually be a reference genome, i.e., a sequence assembled by scientists as a representative example of a set of genetic species. However, a reference sequence may also consist of a synthetic sequence designed simply to improve the compressibility of the reads in light of further processing of the reads.

[0063] In some cases, sequencing machines may introduce errors into sequence reads, particularly the use of incorrect symbols to represent nucleic acids or bases actually present in the sequenced sample (i.e., representing different nucleic acids). This type of substitution error may ultimately be identified as a "mismatch" by the mapping and alignment module. This is because the substitution error in the read may not match the corresponding location in the reference sequence when the read is aligned to the reference sequence. However, the meaning of "mismatch" is not limited to such a scenario. Instead, a "mismatch" may be any base or nucleotide in the read called by the sequencing device that does not match the corresponding location in the reference sequence when the read is aligned to the reference sequence with a threshold level of accuracy. Such mismatches may include candidate variants, variants, or other differences between the aligned read and the location in the reference sequence.

[0064] The present disclosure relates to reference-based compression methods that receive as input nucleotide or base sequence reads, where such reads have already been aligned to a reference sequence by a mapping and alignment module, thereby creating aligned reads. In some implementations, the already aligned reads can include reads aligned using a software mapping and alignment module that performs mapping and alignment of the received reads to a reference sequence. For example, in some implementations, a software mapper can perform hash-table-based mapping and alignment of the received reads by executing software instructions using one or more processors, such as one or more central processing units (CPUs), one or more graphical processing units (GPUs), or any combination thereof. In other implementations, the already aligned reads can include reads aligned using a hardware mapping and alignment module that performs mapping and alignment of the received reads to a reference sequence. For example, in some implementations, the hardware mapping and alignment module can perform hash table-based mapping and alignment by using one or more hardware processors, such as one or more field programmable gate arrays (FPGAs) having hard-wired digital logic circuits configured to perform hash table-based mapping and alignment of received reads.

[0065] The aligned reads are then stored as a list of reads in an initial file. The method of aligning the reads and storing the reads in the initial file once aligned is not important to the present invention and is not the purpose of this disclosure. Each read is then encoded as a list of its position on the reference sequence and its differences from the reference sequence. Each read can then be reconstructed from the alignment encoding information and the reference sequence by appropriate decompression software configured as described herein by the present disclosure.

[0066] In some implementations, the compression module of the present disclosure can be implemented to process and compress aligned reads through the execution of software instructions by one or more CPUs or GPUs, the execution of hardwired digital logic circuits in one or more hardware processors, or a combination of both. Reads can be aligned to a reference sequence before compressing the reads, without considering certain types of errors introduced into the sequence reads, such as insertion or deletion errors. An insertion error is the insertion in one sequence read of one or more additional symbols that do not refer to any actually present nucleic acid. A deletion error is the deletion from one or more sequence reads consisting of one or more symbols that represent nucleic acids actually present in the sequenced sample. More precisely, in the case of an insertion or deletion error in a given sequence read, the alignment software will consider the resulting incorrect nucleic acid as a substitution error, also called a "mismatch." This preferential selection of the alignment software configuration allows for faster subsequent encoding, providing a better compromise between speed and compression ratio.

[0067] For each aligned read, the mapping and alignment module can generate and provide a read record. In some implementations, each read record can be provided directly from the mapping and alignment module as input to the compression module. In other implementations, each read record generated by the mapping and alignment module can be output and stored in memory or other storage device. In such implementations, the compression module can later access and compress the stored read record.

[0068] Each read record generated, provided, or stored by the mapping and alignment module includes data generated by the mapping and alignment module that describes the read represented by the read record. Such a read record may include at least the following information: the absolute start position of the aligned read with respect to a reference sequence, the length of the read, the type of alignment of the read, such as whether the read is a mapped or unmapped read, the number of mismatches identified in the read, an indication of whether the read is fully mapped or incompletely mapped, the relative positions of such possible mismatches within the read, etc.

[0069] Although the examples described herein show that the data in the read records and the data contained therein are generated by the mapping and alignment module, the present disclosure is not so limited. Instead, other intermediate modules between the mapping and alignment module and the compression module can be used to generate the read records and the data contained therein.

[0070] In some implementations, the read records provided or stored by the mapping and alignment module may be provided or stored in a manner that preserves the sequential ordering of the read records generated by the mapping and alignment module. In some implementations, for example, each read record may also include data indicating the read record arrangement in the sequential ordering of the read records. Such data indicating the read record arrangement may include, for example, an array_id. In some implementations, this array_id may be, for example, a number starting with "1" for the first read record generated by the mapping and alignment module, which is then incremented for each subsequent read record generated by the mapping and alignment module. A compression module of the present disclosure can then access these read records and compress them in their current sequential order without having to reorder them into clusters of read records for compression. Compressing read records in a manner that preserves the initial ordering of the read records as generated by the mapping and alignment module provides an advantage over conventional methods by enabling lossless compression of the read records, since even the sequential ordering of the read records is preserved. Additionally, preserving the order of the lead records during compression also makes it easier to verify the compression of the lead records.

[0071] The compression method of the present disclosure will now be described with reference to Figure 1. In some implementations, for example, the method may be performed by apparatus 20 shown in Figure 2. Apparatus 20 may include at least one processor 22 and at least one memory 24 operably coupled to at least one processor 22 to form a computing device. Memory 24 may store computer program code or software 26 including computer-executable instructions that, when executed by processor 22, cause processor 22 to perform operations of a compression module, including performing stages related to one or more of the compression methods described herein. However, the present disclosure need not be limited to being implemented by apparatus 20.

[0072] For example, in some implementations, the compression methods of the present disclosure can be implemented by apparatus 20A shown in FIG. 2A . Apparatus 20A is similar to apparatus 20 in that apparatus 20A also includes processors 22 and at least one memory 24 operably coupled to the at least one processor 22 to form a computing device. The memory 24 of apparatus 20A also stores computer program code or software 26 including computer-executable instructions that, when executed by processor 22, cause processor 22 to perform operations including one or more stages of the compression methods described herein. In addition, however, apparatus 20A also includes computer program code or software 28 including computer-executable instructions that, when executed by processor 22, cause processor 22 to perform operations to implement the functionality of a mapping and alignment module. The mapping and alignment module, whose functionality is realized via the execution of computer software instructions, can generate one or more aligned reads 29 and store the aligned reads 29 in memory 24. The processor 22 can then execute the software instructions 26 of the compression module to access one or more of the aligned reads 29 and compress the one or more aligned reads 29 using one or more stages of the compression method described herein. In some implementations, the apparatus 20A can be a nucleic acid sequencing device.

[0073] As another example, in some implementations, the compression method of the present disclosure can be implemented by device 20B shown in FIG. 2B . Device 20B differs from device 20 in that device 20B includes one or more hardware processors 22B, such as one or more field programmable gate arrays (FPGAs). In this example, the one or more hardware processors can implement the functionality of one or more stages of the compression method described herein and the mapping and alignment module within the hardware circuitry of the one or more hardware processors 22B. For example, hardware processor 22B can include hard-wired digital logic circuitry 26B configured as a compression module for performing one or more stages of the compression method described herein. Similarly, hardware processor 22B can include hard-wired digital logic circuitry 28B configured to perform the operations of the mapping and alignment module, which is configured to generate aligned reads 29B and store aligned reads 29B in memory 24. A digital hardwired logic circuit 26B configured as a compression module for implementing the functionality of one or more stages of the compression methods described herein can access the aligned reads 29B from memory 24 and compress the aligned reads 29B using the compression methods described herein. In some implementations, the apparatus 20B can be a nucleic acid sequencing device. An initial file in which the aligned read records are stored as a read list is stored, for example, in the memory of the apparatus 20B. In some implementations, the read list can include multiple aligned read records stored in the memory of the apparatus in a manner that preserves the sequence ordering of the read records as generated by the mapping and alignment module. This sequence ordering of the aligned read records can be the same order as obtained at the end of the mapping and alignment stage.

[0074] In some implementations, the initial list of aligned reads can be divided into blocks of reads. For example, in some implementations, the list of aligned reads can be divided into blocks of 50,000 reads. However, this particular value of blocks of 50,000 reads should not be construed as limiting the scope of the present disclosure, as implementations of the present disclosure can be achieved in the same manner using other values.

[0075] In some implementations, blocks of reads can have the same block size. However, in other implementations, blocks of reads can have various block sizes. In any case, each block of reads can begin with a header containing information necessary to decode the block, such as the size in bytes of the block's contents, and / or an identifier for the block or its contents, and / or the number of reads contained in the block. This allows support for concatenation of compressed files and streaming capabilities (each block of reads contains all the information necessary to decode the reads of the block). Furthermore, since the compression method can then run on any number of blocks, this also allows for multi-threaded processing on blocks of reads, thereby allowing for parallelization and some resulting gains in processing time. If all reads of a given block have the same length, the read lengths are also stored in the header; otherwise, a list of each read length is explicitly stored during the compression method.

[0076] Returning to FIG. 1 , the method preferably includes an initial stage 2 in which the device retrieves aligned read records from the memory of device 20, 20A, or 20B. In some implementations, this initial stage may include the device accessing a memory or other storage device that stores multiple read records in a manner that preserves the sequence ordering of the read records generated by the mapping and alignment module. For example, the device may determine the next read record for compression based on the sequence_id of the preceding read record in memory and the sequence_id of one or more other read record stores. In some implementations, the sequence_id may be a number that increments for each subsequent read record generated by the mapping and alignment module, and the compression module may maintain a counter that increments with each iteration of the compression process of FIG. 1 to provide an indication of the next read record to be accessed in stage 2.

[0077] Each read record contains information about the type of alignment of the read. The information about the type of alignment of the read can include any information that describes the level of mapping and alignment of the read to the reference genome. In some implementations, the type of alignment can include a complete alignment, an incomplete alignment, or an "unmapped" read alignment. A "complete alignment" or a "fully mapped read" can include a read in which each nucleotide of the read maps and aligns to a portion of the reference genome. In some implementations, a "complete alignment" or a "fully mapped read" can have zero mismatches and zero undetermined bases "N". In other implementations, a "complete alignment" or a "fully mapped" read can have zero mismatches, but optionally one or more undetermined bases "N". Generally, the definition of an "incomplete alignment" or an "incompletely mapped read" depends on the meaning of a "fully mapped read" implemented in a particular implementation of the compression method described herein. For example, if an implementation is used in which a fully mapped read may contain zero mismatches and zero undetermined bases N, an "incomplete alignment" or "incompletely mapped read" means any read that matches at least a portion of a reference sequence and includes at least one mismatch or at least one N. However, if an implementation is used in which a fully mapped read may contain zero mismatches for one or more Ns, for example, an "incomplete alignment" or "incompletely mapped read" means any read in which part of the read matches part of a reference sequence but has at least one mismatch other than the undisclosed base N (according to this other definition of an incompletely mapped read, an incompletely mapped read may contain one or more Ns, provided that the read also contains one or more other mismatches).Thus, how any particular system or implementation is configured to recognize fully mapped reads will determine the meaning of incompletely mapped reads for that implementation. "Unmapped reads" can include reads that have not been mapped or aligned to a reference genome.

[0078] In some implementations, each lead record includes a plurality of bit flags that describe attributes of the lead. In some implementations, the plurality of bit flags can be stored using one or more fields at the beginning of the lead record. However, in other implementations, other fields of the lead record can be used to store the plurality of bit flags. Each bit flag of the plurality of bit flags can indicate the value of its corresponding lead attribute using one of a plurality of values. In some implementations, the following bit flags are used to indicate the value of the lead attribute for the lead record: a first bit flag indicating forward or reverse orientation relative to the reference sequence; - a second bit flag indicating perfect alignment or not, a third bit flag indicating whether the read contains at least one N; a fourth bit flag indicating whether the position information is coded in 16 bits or 32 bits; A fifth bit flag indicating whether the read is mapped or unmapped, can be used.

[0079] Stages 4-12 below are performed for each read in the plurality of reads. If the reads are grouped into blocks, stages 4-12 are performed for each read in each block of reads.

[0080] The compression method of the present disclosure may include a next stage 4 in which the apparatus 20, 20A, or 20B determines, for each aligned read, whether the read is fully mapped to the reference sequence, incompletely mapped to the reference sequence, or not mapped to the reference sequence. In some implementations, the apparatus 20, 20A, or 20B may determine whether the read is fully mapped, incompletely mapped, or not mapped based on information received from the mapping and alignment module. This information may include, for example, information such as whether the read represented by the acquired read record is mapped or not mapped, whether the read represented by the read record is fully mapped or incompletely mapped, an indication of the total number of mismatches, such as variants or sequencing errors, undetermined bases, or any combination thereof. In some implementations, this information may be included within the acquired read record itself.

[0081] In some implementations, apparatus 20, 20A, or 20B may first determine whether the aligned reads were mapped or unmapped. If apparatus 20, 20A, or 20B determines that the aligned reads were unmapped, the apparatus may continue performing the process of FIG. 1 at stage 6. Alternatively, if apparatus 20, 20A, or 20B determines that the reads were mapped, apparatus 20, 20A, or 20B may determine whether the reads were incompletely mapped or fully mapped.

[0082] In some implementations, the apparatus 20, 20A, or 20B can determine whether a read is incompletely mapped or fully mapped by evaluating the number of total mismatches in the read. In some implementations, this total mismatch number can be provided by a mapping and alignment module and obtained from the acquired read record. In such implementations, if the apparatus 20, 20A, or 20B determines that the total mismatch number is equal to 0, the apparatus 20, 20A, or 20B can determine in Stage 4 that the acquired aligned read is a fully mapped read and can continue performing the process of FIG. 1 in Stage 6. Alternatively, if the apparatus 20, 20A, or 20B determines in Stage 4 that the total mismatch number is greater than 0, the apparatus 20, 20A, or 20B can determine in Stage 4 that the read corresponding to the read record is an incompletely mapped read and can continue performing the process of FIG. 1 in Stage 6.

[0083] However, it should be noted that the preceding implementations are merely examples of how device 20, 20A, or 20B may determine whether an aligned read record is fully mapped, incompletely mapped, or not mapped. For example, in some implementations, such a determination may be made based on information contained in the acquired read record without comparing the number of mismatches to a zero threshold. By way of example, the read record may maintain a bit flag in a header or other portion of the read record that indicates whether the read is mapped or not mapped, fully mapped or incompletely mapped, etc. In such implementations, device 20, 20A, or 20B may make a determination in stage 4 regarding whether the aligned read record is mapped, not mapped, fully mapped, or not completely mapped based on the bit flag in the acquired read record without comparing the number of mismatches to a zero threshold. Other implementations are also within the scope of the present disclosure. For example, it is conceivable that an implementation could be employed that could access information stored in a data structure different from the acquired read record to read bit flags or other data and take that information into account to indicate whether a particular read record is mapped, unmapped, completely mapped, or incompletely mapped.

[0084] In some implementations, this determining step 4 can further include comparing 4a, for each incompletely mapped read, the number of mismatches between the read sequence and the reference sequence to a threshold. This can include a total mismatch count, which includes the sum of any differences between the aligned read and the reference sequence, including variants, sequencing errors, and undetermined base N. In some implementations, the mismatch count can be provided by a mapping and alignment module or obtained from the read record.

[0085] In some implementations, the threshold may be 31. This particular value may be selected to provide the best possible compromise for storing the number of mismatches in a sufficiently compact manner, as will be better understood later with respect to stage 12. Indeed, it has been statistically observed that in the majority of cases, incompletely mapped reads have fewer than 31 mismatches. The rationale behind that selection is to encode the most frequent cases in the most compact way while still having some very few reduced cases. However, while using a threshold of 31 mismatches may be advantageous in some implementations, such as implementations of short reads where the reads are approximately 150 nucleotides or bases in length, the present disclosure is not limited to only those implementations where the threshold is equal to 31. Instead, in other implementations, it may be desirable to use a threshold higher than 31. For example, while embodiments (e.g., a threshold of 31 mismatches) may be intended to be useful for compressing read records representing reads generated by short-read sequencers, it is contemplated that the genomic data compression methods of the present invention may be used in other implementations, such as compressing read records generated by long-read sequencers. Thus, in such implementations, if the reads are represented by read records significantly longer than 150 nucleotides or bases in length, the threshold can be set to a value higher than 31 to enable functionality of the compression methods of the present disclosure for long read systems.

[0086] If a read is determined to be incompletely mapped with a number of mismatches less than a threshold, decision stage 4 may also include an additional determination of whether the read is globally or locally mapped to the reference sequence. A "globally mapped read" is an incompletely mapped read in which the entire sequence, including the start and end of the read, is incompletely mapped to the reference sequence. A "locally mapped read" is an incompletely mapped read that contains a segment of nucleotides or bases that are incompletely mapped to the reference sequence. Thus, the segment of nucleotides or bases corresponds to part of the initial read.

[0087] In some implementations, the compression method can further include stage 6, determining, for each aligned read, whether the read contains at least one undetermined base "N," i.e., whether the read contains at least one mismatch corresponding to an instance where the sequencing machine was unable to call any base or nucleotide. The method then includes stage 8, for each read that contains at least one "N," determining the number of such undetermined bases "N," and stage 10, comparing the number of undetermined bases "N" to a reference threshold. In some implementations, the reference threshold can be equal to 31. However, in other implementations, other reference thresholds can be set to other values.

[0088] Whatever the outcome of the determination stage 4, the method includes a next stage 12 in which the reads are coded according to the determination. More precisely, reads determined to be completely mapped with the reference sequence are coded by a first coding process, whether they contain no undetermined bases "N" or have a number of undetermined bases "N" less than the reference threshold. Reads determined to be unmapped or completely mapped with a number of undetermined bases "N" greater than the reference threshold are coded by a second coding process in which each nucleotide or base is coded individually, regardless of whether the nucleotide or base is aligned or not. Reads determined to be incompletely mapped are coded by a second or third coding process. More precisely, reads determined to be incompletely mapped with a number of mismatches greater than the threshold are coded by the second coding process. If a read is determined to be incompletely mapped with a number of mismatches less than the threshold, the read is coded by the third coding process if it does not contain N or has a number of N less than the reference threshold, otherwise, i.e., if it has a number of N greater than the reference threshold, the read is coded by the second coding process.

[0089] Whether a given read is determined to be fully mapped, incompletely mapped, or unmapped, if the read contains at least one N but has a number of N less than the reference threshold, encoding stage 12 includes encoding a list of positions along the reference sequence, the positions corresponding to the position of N in the reference sequence. The list of positions is then stored in the memory of a computing device, which implements a compression method. If the read contains at least one N but has a number of N less than the reference threshold and is to be encoded by the second encoding process, each nucleotide or base of the read is individually coded with two bits.

[0090] If a read contains at least one N, but has a number of Ns greater than the reference threshold, the read is in any case encoded by a second encoding process, in which each nucleotide or base of the read is individually encoded with 4 bits. In this case, encoding stage 12 does not involve encoding and storing a list of the positions of Ns in the reference sequence. In fact, each N mismatch is directly encoded by the second encoding process in much the same way as any other nucleotide or base of the read.

[0091] The first and third encoding processes include separate sets of descriptors, each set uniquely representing a read associated with the corresponding encoding process, and each of the first and third encoding processes is an information entropy reduction encoding process. More precisely, the third encoding process includes a first encoding sub-process and a second encoding sub-process. Incompletely mapped reads determined to be globally mapped during stage 4 are encoded by the first encoding sub-process. Incompletely mapped reads determined to be locally mapped during stage 4 are encoded by the second encoding sub-process. The first and second encoding sub-processes include separate sets of descriptors, each set uniquely representing a read associated with the corresponding encoding sub-process.

[0092] The alignment information that is encoded for each read and that allows reconstruction of the entire read sequence during data decompression then depends on the corresponding encoding process or sub-process used for that read.

[0093] For example, in some implementations, the first set of descriptors used for the first encoding process may be: the absolute start position of the fully mapped read with respect to the reference sequence (encoded as 16 or 32 bits); the length of the read (encoded differentially relative to the length of the preceding read, with a variable length code ranging from 2 bits to 34 bits); may include:

[0094] As another example, in some implementations, the second set of descriptors used in the first encoding sub-process may be: the absolute start position of the incompletely mapped read with respect to the reference sequence (encoded as 16 or 32 bits); the length of the read (encoded differentially relative to the length of the preceding read, with a variable length code ranging from 2 bits to 34 bits); A list of lead mismatches and may include:

[0095] As another example, in some implementations, the third set of descriptors used for the second encoding sub-process may be: the absolute start position of the incompletely mapped portion of the read with respect to the reference sequence, also referred to as the local alignment start position (encoded in 16 or 32 bits); the length of the read (encoded differentially relative to the length of the preceding read, with a variable length code ranging from 2 bits to 34 bits); A list of lead mismatches and the length of the clipped portions of the reads that are not part of the alignment (encoded with 8 bits for each clipped portion); may include:

[0096] Preferably, the list of mismatches encoded in the first sub-process and the second sub-process may include a header. For example, in some implementations, the header may be encoded using bit flags and may be encoded in a single byte. In such implementations, the first five bits of the single-byte header may be used to encode the number of mismatches contained in the read. In implementations where the threshold is equal to 31, the number of mismatches may range from 0 to 31. One bit of the single-byte header may be used to encode whether the incompletely mapped read is globally mapped or locally mapped. Another bit of the single-byte header may be used to encode whether 2-bit mode is enabled for the second encoding process. The last bit of the single-byte header may be used to encode whether 4-bit mode is enabled for the second encoding process. In some implementations, for each read encoded by the second encoding sub-process during encoding stage 12, the clipped portion of the read (i.e., the portion not part of the local alignment) is concatenated, and each nucleotide or base of the clipped portion is individually encoded. In some implementations, each nucleotide or base in such a clipped portion of the read is individually coded with two bits.

[0097] In some implementations, each mismatch encoded in the list of mismatches of an incompletely mapped read (i.e., encoded by the first encoding sub-process or the second encoding sub-process) can be encoded with one byte. More precisely, each mismatch of an incompletely mapped read to be encoded by the first encoding sub-process or the second encoding sub-process may be encoded as follows: the first two bits of a byte are used to encode an alternative nucleotide or base present in the read in place of the corresponding reference nucleotide or reference base in the reference sequence; The six final bits are used to encode the position of the mismatch in the reference sequence, calculated as an offset from the preceding mismatch in the read. This calculated position may be the relative position of the mismatch, except for the first mismatch in the read, where an absolute position is encoded. Therefore, the range of this offset encoded in the six bits may be [0 to 63].

[0098] The encoded or compressed record resulting from the completion of the process of Figure 1 can be stored in the memory or other storage device of the device. In some implementations, this encoded or compressed record can be stored in the memory or other storage device of the device in a manner that preserves the ordering of the read records. This helps ensure that the compression of the aligned read records is lossless, since even the initial ordering of the aligned read records is preserved.

[0099] Obtain aligned read records at stage 102 The compression method of Figure 1 will be described in more detail with respect to the compression method 100A of Figure 1A. The execution of the compression method 100A by the device 20, 20A, or 20B can begin at an initial stage 102, which includes obtaining aligned read records (also referred to below as "obtained read records" or "unmapped reads" / "mapped reads" / "fully mapped reads" / "incompletely mapped reads" based on subsequent classification of the obtained read records during the execution of the method 100A). In some implementations, the aligned read records can be obtained from a plurality of aligned read records stored in a manner that preserves the initial order of the aligned read records provided by the sequencing device. Thus, the entire operation of the mapping and alignment module and the compression module can keep the read records in the initial order of the read records as provided by the sequencing device. In some implementations, the aligned read records can be stored in a manner that preserves the initial order of the aligned read records by using an array_id that is stored with each aligned read record and incremented with each aligned read record generated by the mapping and alignment module.

[0100] At stage 104, it is determined whether the read corresponding to the aligned read record is fully mapped, incompletely mapped, or unmapped. The compression method of the present disclosure may include a next stage 104 in which the apparatus 20, 20A, or 20B determines whether the acquired read record is fully mapped to the reference sequence, incompletely mapped to the reference sequence, or not mapped to the reference sequence. In some implementations, the apparatus 20, 20A, or 20B may determine whether the read is fully mapped, incompletely mapped, or not mapped based on information received from the mapping and alignment module. This information may include, for example, information such as whether the read represented by the acquired read record is mapped or not mapped, whether the read represented by the read record is fully mapped or incompletely mapped, an indication of the total number of mismatches, such as variants or sequencing errors, undetermined bases, or any combination thereof. In some implementations, this information may be included within the read record itself.

[0101] In some implementations, apparatus 20, 20A, or 20B may first determine whether the aligned reads were mapped or unmapped at stage 104. If apparatus 20, 20A, or 20B determines that the aligned reads were unmapped, the apparatus may continue performing process 100A of FIG. 1A at stage 120. Alternatively, if apparatus 20, 20A, or 20B determines that the reads were mapped, apparatus 20, 20A, or 20B may further determine during stage 104 whether the reads were incompletely mapped or fully mapped.

[0102] In some implementations, apparatus 20, 20A, or 20B can determine during stage 104 whether a read was incompletely mapped or completely mapped by evaluating the number of mismatches in the read. In some implementations, this mismatch number can be provided by a mapping and alignment module and obtained from the read record. The mismatch number may be counted in different ways for different implementations. In some implementations, the mismatch number at stage 104 may not include the number of undetermined bases N. In other implementations, the mismatch number determined at stage 104 may include the total number of mismatches and the number of undetermined bases N.

[0103] In the example of FIG. 1A , it is assumed that the undetermined base N is not a mismatch. As a result, a fully mapped read may contain zero mismatches and one or more undetermined bases N. Thus, in this implementation, an incompletely mapped read would need to have at least one mismatch and may or may not have any undetermined bases N. However, in other implementations, the process of FIG. 1A can be modified by assuming that the presence of N in a read may be a mismatch. In such implementations, a read may be determined to be a fully mapped read only if it is determined to have zero mismatches and zero undetermined bases N, and a read with zero mismatches and one or more undetermined bases N may be classified as an incompletely mapped read.

[0104] In a first implementation, if the apparatus 20, 20A, or 20B determines at stage 104 that the total number of mismatches is equal to 0 and the total number of undetermined bases N is 0 or greater than 1, the apparatus 20, 20A, or 20B can determine that the aligned read obtained at stage 4 is a completely mapped read, and can continue performing process 100A of FIG. 1 at stage 116. Alternatively, in the first implementation, if the apparatus 20, 20A, or 20B determines during stage 104 that the total number of mismatches is greater than 0 and the total number of undetermined bases N is 0 or greater than 1, the apparatus 20, 20A, or 20B can determine during stage 104 that the read corresponding to the obtained read record is an incompletely mapped read, and the apparatus 20, 20A, or 20B can continue performing process 100A of FIG. 1A at stage 106.

[0105] In a second and alternative implementation, if at stage 104 the total number of mismatches is equal to 0 and the total number of undetermined bases N is 0, and in such a scenario, device 20, 20A, or 20B can continue executing process 100A of FIG. 1A at stage 116, device 20, 20A, or 20B will simply determine that the read is a fully mapped read. Alternatively, in this second implementation, if device 20, 20A, or 20B determines during stage 104 that the total number of mismatches is greater than 0 or the total number of undetermined bases N is greater than 0, device 20, 20A, or 20B can determine during stage 104 that the read corresponding to the obtained read record is an incompletely mapped read, and device 20, 20A, or 20B can continue executing process 100A of FIG. 1A at stage 106.

[0106] However, it should be noted that the preceding implementations are merely examples of how device 20, 20A, or 20B may determine in stage 104 whether a read corresponding to an acquired read record is fully mapped, incompletely mapped, or unmapped. For example, in some implementations, such a determination may instead be made based on information contained in the acquired read record, without comparing the number of mismatches to a threshold, without comparing the number of undetermined bases N to a threshold, or both. As an example, the read record may maintain a bit flag in a header or other portion of the read record that indicates whether the read is mapped or unmapped, fully mapped, or incompletely mapped, etc. In such implementations, device 20, 20A, or 20B may make a determination, as in stage 4, regarding whether an aligned read record is mapped, unmapped, fully mapped, or incompletely mapped based on the bit flag in the read record, without comparing the number of mismatches or undetermined bases N to a threshold. Other implementations are also within the scope of the present disclosure. For example, it is contemplated that an implementation may be used that can access information stored in a data structure separate from the lead record and read a bit flag or other data to indicate whether the lead corresponding to a particular lead record is mapped, unmapped, completely mapped, or incompletely mapped.

[0107] Stage 104 "Incompletely Mapped Read" branch If the apparatus 20, 20A, or 20B determines in stage 104 that the read corresponding to the obtained read record is an incompletely mapped read, the apparatus 20, 20A, or 20B can determine in stage 106 whether the number of differences between the incompletely mapped read and the reference sequence exceeds a first threshold. This can include a total mismatch count, which includes the total number of mismatches including the sum of all differences between the aligned read and the reference sequence, including variants, sequencing errors, and undetermined base N. In other implementations, the number of differences in stage 106 can include only the number of mismatches, without taking into account the number of undetermined base N. In some implementations, the mismatch count can be provided by the mapping and alignment module and obtained from the read record.

[0108] In some implementations, the first threshold may be 31. This particular value may be selected to provide the best possible compromise for storing the number of mismatches in a sufficiently compact manner so that it can be better understood later for subsequent stages. Indeed, it has been statistically observed that in the majority of cases, incompletely mapped reads have fewer than 31 mismatches. The rationale behind that selection is to encode the most frequent cases in the most compact way, while leaving some very few reduced cases. However, although there are certain advantages that can be achieved using a first threshold of 31, the present disclosure is not limited to only those implementations in which the first threshold is equal to 31. Instead, in other implementations, it may be desirable to use a threshold higher than 31. For example, while embodiments (e.g., a threshold of 31 mismatches) may be intended to be useful for compressing read records representing reads generated by short-read sequencers, it is contemplated that the genomic data compression methods of the present invention may be used in other implementations, such as compressing read records generated by long-read sequencers. Thus, in such implementations, if the reads are represented by read records significantly longer than 150 nucleotides or bases in length, the threshold can be set to a value higher than 31 to enable functionality of the compression methods of the present disclosure for long read systems.

[0109] Stage 106 "Yes" branch If apparatus 20, 20A, or 20B determines in stage 106 that the number of differences between the incompletely mapped reads and the reference sequence exceeds a first threshold, the apparatus may continue executing process 100A in stage 114. In stage 114, apparatus 20, 20A, or 20B may determine whether the number of undetermined bases "N" in the incompletely mapped reads exceeds a second threshold. In some implementations, the second threshold may also be equal to 31. However, like the first threshold, the second threshold of the present disclosure is not limited to the value of 31. Instead, any number, including values ​​higher than 31, may be used for the second threshold in that implementation based on the length of the issued read. Moreover, there is no requirement that the first threshold and the second threshold use the same threshold.

[0110] Stage 114 "Yes" branch If the number of undisclosed bases "N" in an incompletely mapped read is determined by apparatus 20, 20A, or 20B to exceed a second threshold, apparatus 20, 20A, or 20B can determine that the incompletely mapped read will be encoded using a second encoding module 110 to encode the incompletely mapped read using a second encoding process. This second encoding process is the same as the second encoding process described above with respect to FIG. 1, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned. In some implementations, because apparatus 20, 20A, or 20B determined at stage 114 that the number of undisclosed bases "N" exceeded the second threshold, apparatus 20, 20A, or 20B can use the second encoding module to encode 110a the read into 4 bits using a second encoding process. Once the read has been encoded using the second encoding process 110 using the 4-bit encoding 110a, the device 20, 20A, or 20B may store the encoded read in a memory or other storage device at stage 122. The device 20, 20A, or 20B may determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. If there is another sequentially ordered, aligned read, the device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. The device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no sequentially ordered, aligned read records are identified at stage 124. Upon such a determination, the process 100A may end at stage 126.

[0111] Stage 114 "No" branch If apparatus 20, 20A, or 20B determines during stage 114 that the number of undisclosed bases "N" in the incompletely mapped read does not exceed a second threshold, apparatus 20, 20A, or 20B can determine that the incompletely mapped read will be encoded using second encoding module 110 to encode the incompletely mapped read using a second encoding process. This second encoding process is the same as the second encoding process described above with respect to FIG. 1, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned. In some implementations, because apparatus 20, 20A, or 20B determined during stage 114 that the number of undisclosed bases "N" did not exceed the second threshold, apparatus 20, 20A, or 20B can use the second encoding module to encode 110b the read into two bits using the second encoding process. Once the read has been encoded using the second encoding process 110 using the 2-bit encoding 110b, the device 20, 20A, or 20B may store the encoded read in a memory or other storage device at stage 122. The device 20, 20A, or 20B may determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. If there is another sequentially ordered, aligned read, the device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. The device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no sequentially ordered, aligned read records are identified at stage 124. Upon such a determination, the process 100A may end at stage 126.

[0112] Stage 106 "No" branch If apparatus 20, 20A, or 20B determines at stage 106 that the number of differences between the incompletely mapped read and the reference sequence does not exceed a first threshold, apparatus 20, 20A, or 20B may continue performing process 100A at stage 108. At stage 108, apparatus 20, 20A, or 20B may determine whether the incompletely mapped read contains more than a second threshold number of undetermined bases "N."

[0113] Stage 108 "Yes" branch If device 20, 20A, or 20B determines at stage 108 that the number of undisclosed bases "N" in the incompletely mapped read exceeds a second threshold, device 20, 20A, or 20B may determine at stage 108 that the incompletely mapped read will be coded using a second coding module 110 to code the incompletely mapped read using a second coding process. This second coding process is the same as the second coding process described above with respect to FIG. 1, in which each nucleotide or base is coded individually, regardless of whether the nucleotide or base is aligned. In some implementations, because device 20, 20A, or 20B determined at stage 108 that the number of undisclosed bases "N" exceeded the second threshold, device 20, 20A, or 20B may use a second coding module to code 110a the read into 4 bits using a second coding process. Once the read has been encoded using the second encoding process 110 using the 4-bit encoding 110a, the device 20, 20A, or 20B may store the encoded read in a memory or other storage device at stage 122. The device 20, 20A, or 20B may determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. If there is another sequentially ordered, aligned read, the device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. The device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no sequentially ordered, aligned read records are identified at stage 124. Upon such a determination, the process 100A may end at stage 126.

[0114] Stage 108 "No" branch If during stage 108, apparatus 20, 20A, or 20B determines that the incompletely mapped read includes a number of undetermined bases "N" that does not meet the second threshold, apparatus 20, 20A, or 20B can use third encoding module 112 to encode the incompletely mapped read using a third encoding process. The third encoding process of FIG. 1A is the same as the third encoding process described above with respect to the process of FIG. 1 and uses the same descriptors as the third encoding process described above. Once the read has been encoded using the third encoding process of third encoding module 112, apparatus 20, 20A, or 20B can store the encoded read in memory or other storage device at stage 122. Apparatus 20, 20A, or 20B can determine whether there is another sequentially ordered, aligned read to be compressed at stage 124. Alternatively, if there is another sequentially ordered, aligned read, device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. Device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no more sequentially ordered, aligned read records are identified in stage 124. Upon such a determination, process 100A may end at stage 126.

[0115] Stage 104 "fully mapped lead" branch Alternatively, if the read corresponding to the acquired read record is determined to be a fully mapped read in stage 104, device 20, 20A, or 20B may determine in stage 116 whether the fully mapped read includes a number of undetermined bases "N" that exceeds a second threshold. In some implementations, the second threshold may also be equal to 31. However, similar to the first threshold, the present disclosure is not limited to a second threshold of 31. Instead, any number, including a value higher than 31, may be used for the second threshold based on the length of the issued read in that implementation. Moreover, there is no requirement that the first threshold and the second threshold use the same threshold.

[0116] Stage 116 "No" branch If apparatus 20, 20A, or 20B determines at stage 116 that the fully mapped read does not include more than a second threshold number of undetermined bases "N," apparatus 20, 20A, or 20B may determine to encode the read using a first encoding module 122 using a first encoding process. If the fully mapped read does not include any undetermined bases "N," first encoding module 122 performs a first encoding process that is the same as, and uses the same descriptors as, the first encoding process described above with respect to FIG. 1. Alternatively, if the fully mapped read includes one or more "N," first encoding module 122 encodes the fully mapped read using the first encoding process described above with respect to FIG. 1 and using the same descriptors for the first encoding process described above. Additionally, in certain implementations where a fully mapped read contains one or more Ns (but less than a second threshold number of Ns), the first encoding module 118 may also store a list of positions on the read for undetermined bases Ns.

[0117] Once the read has been encoded using first encoding module 118, device 20, 20A, or 20B may store the encoded read in a memory or other storage device at stage 122. Device 20, 20A, or 20B may determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. If there is another sequentially ordered, aligned read, device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. Device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no sequentially ordered, aligned read records are identified at stage 124. Upon such a determination, process 100A may end at stage 126.

[0118] Stage 116 "Yes" branch However, if the device determines at stage 116 that the read contains more than a second threshold number of undetermined bases "N," the device 20, 20A, or 20B can use the second encoding module 110 to encode the read into 4 bits 110a using a second encoding process. The second encoding process of FIG. 1A is the same as the second encoding process described above with respect to FIG. 1, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned. Once the read is encoded using the second encoding process 110 using the 4-bit encoding 110a, the device 20, 20A, or 20B can store the encoded read in a memory or other storage device at stage 122. The device 20, 20A, or 20B can determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. Alternatively, if there is another sequentially ordered, aligned read, device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. Device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no more sequentially ordered, aligned read records are identified in stage 124. Upon such a determination, process 100A may end at stage 126.

[0119] Stage 104 "Unmapped Reads" branch Alternatively, if the read corresponding to the acquired read record is determined to be an unmapped read in stage 104, device 20, 20A, or 20B can determine in stage 120 whether the unmapped read includes a number of undetermined bases "N" that exceeds a second threshold. In some implementations, the second threshold may also be equal to 31. However, similar to the first threshold, the present disclosure is not limited to a second threshold of 31. Instead, any number, including a value higher than 31, can be used for the second threshold based on the length of the issued read in that implementation. Moreover, there is no requirement that the first threshold and the second threshold use the same threshold.

[0120] Stage 120 "No" branch If device 20, 20A, or 20B determines at stage 120 that the unmapped read does not contain more than a second threshold number of undetermined bases "N," device 20, 20A, or 20B can determine to encode the read using a second encoding module 110 using a second encoding process. This second encoding process is the same as the second encoding process described above with respect to FIG. 1, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned. In some implementations, because device 20, 20A, or 20B determined at stage 120 that the number of undetermined bases "N" did not exceed the second threshold, device 20, 20A, or 20B can use the second encoding module to encode 110b the read into two bits using the second encoding process. Once the read has been encoded using the second encoding process 110 using the 2-bit encoding 110b, the device 20, 20A, or 20B may store the encoded read in a memory or other storage device at stage 122. The device 20, 20A, or 20B may determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. If there is another sequentially ordered, aligned read, the device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. The device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no sequentially ordered, aligned read records are identified at stage 124. Upon such a determination, the process 100A may end at stage 126.

[0121] Stage 120 "Yes" branch However, if the apparatus determines at stage 120 that the unmapped read contains more than a second threshold number of undetermined bases "N," the apparatus 20, 20A, or 20B can use the second encoding module 110 to encode the read into 4 bits 110a using a second encoding process. The second encoding process of FIG. 1A is the same as the second encoding process described above with respect to FIG. 1, in which each nucleotide or base is encoded individually, regardless of whether the nucleotide or base is aligned. Once the read is encoded using the second encoding process 110 using the 4-bit encoding 110a, the apparatus 20, 20A, or 20B can store the encoded read in a memory or other storage device at stage 122. The apparatus 20, 20A, or 20B can determine at stage 124 whether there is another sequentially ordered, aligned read to be compressed. Alternatively, if there is another sequentially ordered, aligned read, device 20, 20A, or 20B may perform the operations of stage 102 to obtain the next sequentially ordered, aligned read record and perform process 100A again. Device 20, 20A, or 20B may instead continue to repeatedly perform process 100A until no more sequentially ordered, aligned read records are identified in stage 124. Upon such a determination, process 100A may end at stage 126.

[0122] Figure 3 provides an example of encoding mismatches in a read by the first encoding subprocess. This read is an incompletely mapped read that has been globally mapped with a reference sequence. This read contains two mismatches: a first mismatch located at the 12th position in the read, consisting of a substitution of an A nucleotide in the reference sequence with a T nucleotide in the read; a second mismatch located at the 21st position in the read, consisting of a substitution of a C nucleotide in the reference sequence with a G nucleotide in the read; It has.

[0123] The list of mismatches in the reads is then <12,T>, the value “12” corresponding to the absolute position of the first mismatch in the read, and <9,G>, i.e., the value “9” corresponding to the relative position of the second mismatch in the read, i.e., the offset between the second and first mismatch; is encoded as

[0124] <12,T> may be converted, for example, to the value "51" (encoded in one byte), and <9,G> may be converted to the value "38" (encoded in one byte). Such single-byte encodings are Offset position x 4 + nucleotide value (A=0, C=1, G=2, T=3) It is obtained by

[0125] Preferably, for each incompletely mapped read encoded by the first encoding sub-process or the second encoding sub-process, if the calculated offset between a given mismatch and the previous mismatch in the read is greater than the maximum encodable value, at least one "false" mismatch is inserted between the two mismatches until every offset between each of the two mismatches and at least one "false" mismatch is lower than the maximum encodable value. A "false" mismatch is defined as a mismatch in which the bits of the byte used to encode the mismatch encode a nucleotide or base equal to the corresponding reference nucleotide or base in the reference sequence. In some implementations, the maximum encodable value is equal to 63, which corresponds to the maximum value encodable by 6 bits. However, the present disclosure is not limited to implementations having a maximum encodable value of 63. For implementations having a maximum encodable value greater than 63, additional bits can be used to encode the value. In such implementations, this may require, for example, adjusting the length of other bits in the header for the read, increasing the header size by more than one byte, or a combination of both. Thus, while the algorithmic features of the present disclosure are flexible for particular use cases, implementations that may result from design changes may result in corresponding trade-offs in performance, which may be acceptable or even beneficial under certain circumstances of any particular implementation.

[0126] Figure 4 provides an example of mismatch encoding of a read by the first encoding sub-process in a case where a "false" mismatch needs to be inserted. This read is an incompletely mapped read that has been globally mapped with the reference sequence. This read contains two mismatches, a first mismatch located at position 22 in the read, consisting of a substitution of an A nucleotide in the reference sequence with a T nucleotide in the read; a second mismatch located at position 134 in the read, consisting of a substitution of a C nucleotide in the reference sequence with a G nucleotide in the read; It has.

[0127] The position offset between the second mismatch and the first mismatch is 112, which is greater than the maximum encodable value of 63. Therefore, a "false" mismatch needs to be inserted between the two mismatches, so that any offset between each of the mismatches and the "false" mismatch is less than the maximum encodable value. A "false" mismatch using a T nucleotide (corresponding to the "real" T nucleotide in the reference sequence) is inserted, for example, at the 85th position in the read. The calculated position offset between the "false" mismatch and the first mismatch is 63, which corresponds to the maximum encodable value. The calculated position offset between the second mismatch and the "false" mismatch is 49, which is less than 63.

[0128] The list of mismatches in the reads is then <22,T>, i.e., the value “22” corresponding to the absolute position of the first mismatch in the read; <63,T>, the value “63” corresponding to the relative position of the “false” mismatch within the read, i.e., the offset between the “false” mismatch and the first mismatch; and <49,G>, the value “49” corresponding to the relative position of the second mismatch in the read, i.e., the offset between the second mismatch and the “false” mismatch; is encoded as

[0129] For example, <22,T> may be converted to the value "91" (encoded in one byte), <63,T> may be converted to the value "255" (encoded in one byte), and <49,G> may be converted to the value "198" (encoded in one byte). Such single-byte encodings are Offset position x 4 + nucleotide value (A=0, C=1, G=2, T=3) It is obtained by

[0130] The method includes a final step 14 of providing a compressed file containing a list of encoded reads. The encoded reads are stored in the compressed file in the same order as the reads were stored in the initial uncompressed file. Each read can then be reconstructed from the alignment encoding information and the reference sequence by appropriate decompression software and / or methods configured according to the present invention.

[0131] Although described with respect to an exemplary architecture of computing device 20 (shown in FIG. 2 for illustrative purposes), the inventive techniques disclosed herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the computer program code may be stored on a computer medium and executed by a hardware processing unit, including one or more processors, as with device 20 of FIG. 2. As used herein, the term "processor" should be understood to include one or more processing devices, including a signal processor, microprocessor, microcontroller, application-specific integrated circuit (ASIC), field programmable gate array (FPGA), or other type of processing circuit, as well as portions or combinations of such circuit elements. Additionally, as used herein, the term "memory" is intended to include electronic memory associated with a processor, such as random access memory (RAM), read-only memory (ROM), or other types of memory, in any combination.

[0132] Thus, software instructions or code for carrying out the methodologies and protocols described herein may be stored in one or more associated memory devices, e.g., ROM, fixed or removable memory, and loaded into RAM and executed by the processor when ready for use.

[0133] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including, for example, mobile phones, computers, servers, tablets, and similar devices.

[0134] Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments shown in the drawings, and that various other changes and modifications can be made by those skilled in the art without departing from the scope or spirit of the invention.

[0135] FIG. 5 is a diagram of an example of computing components that can be used to implement a system that performs the compression methods of FIGS. 1 and 1A.

[0136] Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device 500 or 550 may include a Universal Serial Bus (USB) flash drive. A USB flash drive may store an operating system and other applications. A USB flash drive may include input / output components, such as a wireless transmitter or a USB connector, that can be inserted into a USB port of another computing device. The components, their connections and relationships, and their functions illustrated herein are intended to be examples only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0137] Computing device 500 includes a processor 502, memory 504, a storage device 508, a high-speed interface 508 connecting to memory 504 and a high-speed expansion port 510, and a low-speed interface 512 connecting to a low-speed bus 514 and storage device 508. Each of the components 502, 504, 508, 508, 510, and 512 are interconnected using various buses and may be mounted on a common motherboard or in any other suitable manner. Processor 502 processes instructions for execution within computing device 500, including instructions stored in memory 504 or on storage device 508, and can display graphical information for a GUI on an external input / output device, such as a display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses can be used, along with multiple memories and types of memory, as appropriate. Multiple computing devices 500 can also be connected, each providing a portion of the required computation, for example, as a server bank, a cluster of blade servers, or a multiprocessor system.

[0138] The memory 504 stores information within the computing device 500. In one implementation, the memory 504 is a volatile memory unit or units. In another implementation, the memory 504 is a non-volatile memory unit or units. The memory 504 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0139] The storage device 508 can provide mass storage for the computing device 500. In one implementation, the storage device 508 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. An information carrier is a computer-readable or machine-readable medium, such as the memory 504, the storage device 508, or memory on the processor 502.

[0140] The high-speed controller 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 512 manages low-bandwidth-intensive operations. This allocation of functionality is merely one example. In one implementation, the high-speed controller 508 is coupled to the memory 504, to the display 516, e.g., via a graphics processor or accelerator, and to a high-speed expansion port 510 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 512 is coupled to the storage device 508 and to the low-speed expansion port 514. The low-speed expansion port, which can include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet, can be coupled, e.g., via a network adapter, to one or more input / output devices, such as a keyboard, a pointing device, a microphone / speaker pair, a scanner, or a networking device, such as a switch or router. The computing device 500 can be implemented in several different forms, as shown in the figure. For example, the computing device can be implemented as a standard server 520, or multiple times in a cluster of such servers. The computing device can also be implemented as part of a rack server system 524. Additionally, the computing device can be implemented in a personal computer, such as a laptop computer 522. Alternatively, components from computing device 500 can be combined with other components in a mobile device (not shown), such as device 550. Each such device can contain one or more of computing devices 500, 550, and the entire system can be made up of multiple computing devices 500, 550 in communication with each other.

[0141] Computing device 500 can be implemented in several different forms, as shown in the figure. For example, the computing device can be implemented as a standard server 520, or multiple times in a cluster of such servers. The computing device can also be implemented as part of a rack server system 524. Additionally, a computing device can be implemented in a personal computer, such as a laptop computer 522. Alternatively, components from computing device 500 can be combined with other components in a mobile device (not shown), such as device 550. Each such device can contain one or more of computing devices 500, 550, and the entire system can be made up of multiple computing devices 500, 550 in communication with each other.

[0142] Computing device 550 includes, among other components, a processor 552, memory 564, and input / output devices such as a display 554, a communications interface 566, and a transceiver 568. Device 550 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of components 550, 552, 564, 554, 566, and 568 are interconnected using various buses, and some of the components may be implemented on a common motherboard or in other manners as appropriate.

[0143] The processor 552 can execute instructions within the computing device 550, including instructions stored in the memory 564. The processor can be implemented as a chipset of chips including separate and multiple analog and digital processors. In addition, the processor can be implemented using any of several architectures. For example, the processor 510 can be a Complex Instruction Set Computer (CISC) processor, a Reduced Instruction Set Computer (RISC) processor, or a Minimal Instruction Set Computer (MISC) processor. The processor can provide coordination of other components of the device 550, such as control of a user interface, applications run by the device 550, and wireless communication by the device 550.

[0144] The processor 552 can communicate with a user via a control interface 558 and a display interface 556 coupled to a display 554. The display 554 can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display, an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 556 can include appropriate circuitry for driving the display 554 to present graphical and other information to the user. The control interface 558 can receive commands from the user and translate the commands for transfer to the processor 552. Additionally, an external interface 562 can be provided for communication with the processor 552 to enable near-field wireless communication between the device 550 and other devices. The external interface 562 can be provided, for example, for wired communication in some implementations or for wireless communication in other implementations; multiple interfaces can also be used.

[0145] Memory 564 stores information within computing device 550. Memory 564 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Additionally, expansion memory 574 may be provided and connected to device 550 via expansion interface 572, which may include, for example, a single in-line memory module (SIMM) card interface. Such expansion memory 574 may provide additional storage space for device 550 or may store applications or other information for device 550. Specifically, expansion memory 574 may include instructions that perform or complement the processes described above and may also include secure information. Thus, for example, expansion memory 574 may be provided as a security module for device 550 and may be programmed with instructions that enable secure use of device 550. Additionally, secure applications may be provided via a SIMM card, along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0146] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is, for example, a computer-readable or machine-readable medium, such as memory 564, expansion memory 574, or memory on processor 552, which may be received via transceiver 568 or external interface 562.

[0147] Device 550 can communicate wirelessly via a communication interface 566, which can include digital signal processing circuitry if necessary. Communication interface 566 can provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, via a radio frequency transceiver 568. Additionally, short-range communication can occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a Global Positioning System (GPS) receiver module 570 can provide additional navigation-related and location-related wireless data to device 550, which can be used as appropriate by applications running on device 550.

[0148] Device 550 can also communicate audibly using audio codec 560, which can receive speech information from a user and convert this speech information into usable digital information. Audio codec 560 can also generate audible sounds for the user, such as through a speaker in the handset of device 550. Such sounds can include sounds from a voice telephone call, recorded sounds such as voice messages, music files, etc., and can also include sounds generated by applications running on device 550.

[0149] The computing device 550, as shown, can be implemented in several different forms. For example, the computing device can be implemented as a mobile phone 580. The computing device can also be implemented as part of a smartphone 582, personal digital assistant, or other similar mobile device.

[0150] Various implementations of the systems and methods described herein can be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations of such implementations. These various implementations can be special-purpose or general-purpose, and can include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0151] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device, such as a magnetic disk, optical disk, memory, programmable logic device (PLD), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0152] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input.

[0153] The systems and techniques described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or in a computing system that includes middleware components, e.g., an application server, or in a computing system that includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein, or in any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0154] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0155] Other embodiments Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the present invention. Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Additionally, other steps can be provided or eliminated from the described flows, and other components can be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0156] Experimental results Statistical and Numerical Examples of the Compression Method According to the Invention The following comparative example was performed on an uncompressed data file of size 35,770 MB containing 48 million reads or sequences of nucleotides. The results of this comparative example are also shown graphically in FIG.

[0157] The results below show the size of the compressed version of an uncompressed file of 35,770 MB in size with 48 million reads when compressed using each individual algorithm. These results are shown in Chart 610. Size of file compressed with gzip software: 6,649MB (612) Size of compressed file with non-reference-based SPRING software: 1,402MB (614) File size compressed using the reference-based compression method disclosed here: 1,179MB (6160

[0158] The results below show the amount of time it took to compare an uncompressed file of 35,770 MB in size with 48 million reads using each individual algorithm. These results are shown in Chart 620. Compression time with non-reference-based SPRING software: 1,722 seconds (622) Compression time using our reference-based compression method: 181 seconds (624)

[0159] The following results show the average size of the bits / nucleotide ratio of the compressed version of an uncompressed file of 35,770 MB in size with 48 million reads when compressed using each individual algorithm. These results are shown in Chart 630. Average size of uncompressed data files (ASCII encoded) in bits / nucleotide: 8 bits / nucleotide (630) Average size in bits / nucleotides of a file compressed with a coding adapted to the four possible characters A, T, C, G: 2 bits / nucleotide (634) Average size in bits / nucleotides of files compressed with our reference-based compression method: 0.33 bits / nucleotide (636)

[0160] Figure 7 shows additional comparative results using different compression algorithms to compress sample reads generated using WXS novaseq and the reference genome SRR8604734.

[0161] Chart 710 shows a comparison of the resulting compressed size (MB) between gzip compression of WXS novaseq reads using reference genome SRR8604734 (712), Spring compression of WXS novaseq reads using reference genome SRR8604734 (716), and the disclosed compression of WXS novaseq reads using reference genome SRR8604734 (716). The units of measurement are megabytes.

[0162] Chart 720 shows a comparison of compression speeds of compression algorithms for compressing WXS novaseq reads using reference genome SRR8604734. Spring compression speeds for compressing WXS novaseq reads using reference genome SRR8604734 are shown at (722) compared to the speed of the present disclosure for compressing WXS novaseq reads using reference genome SRR8604734 shown at 724. Measurements are in seconds.

[0163] Chart 730 shows a comparison of memory usage of compression algorithms during compression of WXS novaseq reads using reference genome SRR8604734. Spring compression utilized 13,428 MB of memory to compress WXS novaseq reads using reference genome SRR8604734 732, while the present disclosure utilized 3,604 MB of memory to compress WXS novaseq reads using reference genome SRR8604734 734. Measurements are in megabytes.

[0164] Figure 8 shows additional comparative results using different compression algorithms to compress sample reads generated using different sequencers and different reference genomes with different compression ratios.

[0165] Chart 810 shows the raw size in gigabytes (GB) of a file of reads generated using Novaseq (812). When gzip was used to compress Novaseq reads of size 100 GB, gzip compressed the raw data to 17.7 GB (814). The present disclosure compressed the same raw size 812 file of 100 GB of Novaseq-generated reads to a 3.4 GB compressed file (816). The reference genome used for compression by both gzip and the present disclosure was SRR6882909, as shown in 810, resulting in a compression ratio of 5.2x.

[0166] Chart 820 shows the raw size in gigabytes (GB) of a file of reads generated using Hiseq X Ten (822). When gzip was used to compress a 100 GB size of Hiseq X Ten reads, gzip compressed the raw data to 24.9 GB (824). The present disclosure compressed the same raw size 822 file of 100 GB of Hiseq X Ten generated reads to an 8.1 GB compressed file (826). The reference genome used for compression by both gzip and the present disclosure was SRR7725247, as shown in 820, resulting in a compression ratio of 3x.

[0167] Chart 830 shows the raw size in gigabytes (GB) of a file of reads generated using Hiseq 2000 (832). When gzip was used to compress a 100 GB size of Hiseq 2000 reads, gzip compressed the raw data to 27.6 GB (834). The present disclosure compressed the same raw size 832 file of 100 GB of Hiseq 2000 generated reads to an 11.3 GB compressed file (836). The reference genome used for compression by both gzip and the present disclosure was ERR174324, as shown in 830, resulting in a compression ratio of 2.4x.

[0168] The numerical examples presented above demonstrate that the present invention allows for fast compression and decompression while providing high compression ratios.

Claims

1. 1. A computer-implemented method for compression of genomic sequence data generated by a sequencing machine, wherein the genomic sequence data includes nucleotide or base sequence reads aligned to a reference sequence, thereby creating aligned reads, and the aligned reads are stored as a list of reads in an initial file, the method comprising: For each aligned read, determining whether the read is completely or incompletely mapped to the reference sequence, or whether the read is not mapped to the reference sequence; encoding the reads according to the determination, wherein the reads determined to be fully mapped are encoded by a first encoding process and the reads determined to be unmapped are encoded by a second encoding process; the determining step comprises, for each incompletely mapped read, comparing the number of mismatches between the read and the reference sequence to a threshold; In the encoding step, the reads determined to be incompletely mapped are encoded by the second encoding process or a third encoding process, and the incompletely mapped reads are encoded by the second encoding process when the number of mismatches is greater than the threshold, and the incompletely mapped reads are encoded by the third encoding process when the number of mismatches is less than the threshold; In the second encoding process, each nucleotide or base of the read is encoded individually; A method wherein the first encoding process and the third encoding process include separate sets of descriptors, each set of descriptors uniquely representing a lead associated with the corresponding encoding process, and each of the first encoding process and the third encoding process is an information source entropy reduction encoding process.

2. 2. The method of claim 1, wherein the determining step includes a further determination as to whether the read is globally or locally mapped with the reference sequence when the read is determined to be incompletely mapped with the reference sequence and has a number of mismatches less than the threshold; the third encoding process includes a first encoding sub-process and a second encoding sub-process, the reads determined to be globally mapped are encoded by the first encoding sub-process, and the reads determined to be locally mapped are encoded by the second encoding sub-process, and the first encoding sub-process and the second encoding sub-process include separate sets of descriptors, each set of descriptors uniquely representing the reads associated with the corresponding encoding sub-process.

3. 3. The method of claim 2, wherein the descriptors of the first encoding sub-process include an alignment start position in the reference sequence, a read length, and a list of mismatches due to symbol substitution, and the descriptors of the second encoding sub-process include a local alignment start position in the reference sequence, a read length, a list of mismatches due to symbol substitution, and the length of a clipped portion of the read that is not part of an alignment.

4. 4. The method of claim 3, wherein in the encoding step, the clipped portions of the reads to be encoded by the second encoding subprocess are concatenated and each nucleotide or base of the clipped portions is encoded individually.

5. 5. The method of claim 1, wherein in the encoding step, each mismatch of an incompletely mapped read is encoded into one byte.

6. In the encoding step, each mismatch of an incompletely mapped read is the first two bits of said byte are used to encode an alternative nucleotide or base present in said read in place of a corresponding reference nucleotide or reference base in said reference sequence; 6. The method of claim 5, wherein the six last bits of the byte are used to encode the position of the mismatch in the reference sequence, the position being calculated and encoded as an offset from the preceding mismatch of the read.

7. 7. The method of claim 6, wherein, in the encoding step, if the offset calculated between a given mismatch and the previous mismatch is greater than the maximum codable value, at least one false mismatch is inserted between the two mismatches until every offset between each of the mismatches and the at least one false mismatch is less than the maximum codable value, and a false mismatch is defined as a mismatch in which a bit of the one byte is used to code the mismatch or to code a nucleotide or base equal to the corresponding reference nucleotide or reference base in the reference sequence.

8. 8. The method of claim 1, further comprising an initial step of dividing the list of reads into blocks of reads, each block beginning with a header containing information needed to decode said block, and wherein said compression method is performed block by block.

9. The method of claim 8 , wherein the blocks of reads have the same block size.

10. 10. The method of claim 1, further comprising a final step of providing a compressed file comprising a list of encoded reads, wherein the encoded reads are stored in the compressed file in the same order as the order of the reads stored in the initial file.

11. The method according to any one of claims 1 to 10, wherein the threshold value is equal to 31.

12. 12. The method of any one of claims 1 to 11, further comprising the step of determining, for each aligned read, whether the read contains at least one mismatch corresponding to a case where the sequencing machine was unable to call any base or nucleotide.

13. 13. The method of claim 12, further comprising: for each read containing at least one mismatch corresponding to a case where the sequencing machine was unable to call any base or nucleotide, determining the number of such mismatches; and comparing said number to a reference threshold.

14. 14. The method of claim 13, wherein in the encoding step, if the number of such mismatches is greater than the reference threshold, each nucleotide or base of the read to be encoded by the second encoding process is individually coded into 4 bits, and if the number of such mismatches is less than the reference threshold, each nucleotide or base of the read to be encoded by the second encoding process is individually coded into 2 bits, and the encoding step further comprises encoding a list of positions along the reference sequence, the positions corresponding to the positions of such mismatches in the reference sequence.

15. 15. A computer program product embodied in a computer-readable storage medium, the computer program product comprising computer-executable instructions that, when executed by a processor, cause the processor to perform operations including the steps of the method of any one of claims 1 to 14.

16. A computer-readable storage medium having computer-executable instructions that, when executed by a processor, cause the processor to perform operations including the steps of the method of any one of claims 1 to 14.

17. 1. An apparatus comprising: a processor; and a memory operatively coupled to the processor to form a computing device, the memory storing processor-executable instructions that, when executed on at least the processor, cause the processor to perform operations including the steps of the method of claim 1.

18. 1. A method for compressing genomic sequence data, the method comprising: accessing, by the one or more processors, a storage device storing a plurality of read records in a manner that preserves the sequence ordering of the read records as generated by the mapping and alignment module; For each particular lead record of the plurality of lead records: obtaining, by the one or more processors, the particular lead record; determining, by the one or more processors, whether the particular read record corresponds to a read that is perfectly mapped to the reference sequence or that is incompletely mapped to the reference sequence; determining, by the one or more processors, whether the number of mismatches of the incompletely mapped reads satisfies a predetermined mismatch threshold number, based on the one or more processors determining that the particular read record corresponds to a read that is incompletely mapped to the reference sequence; encoding, by the one or more processors, each mismatch of the incompletely mapped read into a compressed read having a predetermined compressed record size based on determining that the number of mismatches meets the predetermined mismatch threshold number; storing, by the one or more processors, the compressed records in the storage device while maintaining the ordering of the read records.

19. Each of the plurality of lead records comprises: data indicating the absolute start position of the aligned reads relative to the reference sequence; Data indicating the length of the read; data indicating whether the read is completely or incompletely mapped; data indicating the number of mismatches identified in the reads; data indicating whether the read contains at least one undetermined base N; Data indicating the number of undetermined bases N in the read; data indicating whether the read is mapped or unmapped; data indicating the position of the lead record in the sequence of lead records output by the mapping and alignment module; and data indicating the relative position of the possible mismatch in the read.

20. 20. The method of claim 18, wherein the predetermined compressed recording size is 1 byte.

21. Encoding each mismatch of the incompletely mapped read into a compressed record having a size of 1 byte includes, for each particular mismatch: encoding, by one or more processors, the first two bits of the byte to contain data representing an alternative nucleotide or base present in the read in place of a corresponding reference nucleotide or base in the reference sequence; and encoding, by one or more processors, the remaining 6 bits of the byte to contain data indicating the position of a mismatch in the reference sequence, the position being calculated as an offset from a preceding mismatch in the read.

22. The method comprises: determining, by one or more processors, whether the offset is greater than a maximum codable value; 22. The method of claim 21, further comprising: inserting, by one or more processors, at least one false mismatch between the particular mismatch and the previous mismatch based on determining that the offset is greater than the maximum encoded value.

23. The method comprises: The method of claim 18, further comprising, based on determining that the number of mismatches does not meet the predetermined mismatch threshold number, encoding, by one or more processors, a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

24. The method comprises: The method of claim 18, further comprising, based on determining that the read record corresponds to a read that is fully mapped to the reference sequence, encoding at least a portion of the read record using information entropy reduction encoding by the one or more processors.

25. Determining, by the one or more computers, whether the number of mismatches in the incompletely mapped reads meets a predetermined threshold number of mismatches, 20. The method of claim 18, further comprising determining, by the one or more processors, whether the number of mismatches in the imperfectly mapped reads is greater than the reference threshold.

26. 1. A system for compressing genome sequence data, the system comprising: one or more computers; and one or more storage devices that store instructions, which when executed by the one or more computers, cause the one or more computers to: accessing, by the one or more computers, a storage device storing a plurality of read records in a manner that preserves a sequence ordering of the read records as generated by the mapping and alignment module; For each particular lead record of the plurality of lead records: obtaining, by the one or more computers, the particular lead record; determining, by the one or more computers, whether the particular read record corresponds to a read that is perfectly mapped to the reference sequence or that is incompletely mapped to the reference sequence; determining, by the one or more computers, whether the number of mismatches of the incompletely mapped reads satisfies a predetermined mismatch threshold number, based on the one or more computers determining that the particular read record corresponds to a read that is incompletely mapped to the reference sequence; encoding, by the one or more computers, each mismatch of the incompletely mapped read into a compressed read having a predetermined compressed record size based on determining that the number of mismatches meets the predetermined mismatch threshold number; and storing the compressed records in the storage device while maintaining the ordering of the read records.

27. Each of the plurality of lead records comprises: data indicating the absolute start position of the aligned reads relative to the reference sequence; Data indicating the length of the read; data indicating whether the read is completely or incompletely mapped; data indicating whether the read contains at least one undetermined base N; Data indicating the number of undetermined bases N in the read; data indicating whether the read is mapped or unmapped; data indicating the position of the lead record in the sequence of lead records output by the mapping and alignment module; and data indicating the relative position of the possible mismatch in the read.

28. 27. The system of claim 26, wherein the predetermined compressed recording size is 1 byte.

29. Encoding each mismatch of the incompletely mapped read into a compressed record having a size of 1 byte includes, for each particular mismatch: encoding, by one or more computers, the first two bits of the byte to contain data representing an alternative nucleotide or base present in the read in place of a corresponding reference nucleotide or base in the reference sequence; and encoding, by one or more computers, the remaining 6 bits of the byte to contain data indicating the position of a mismatch in the reference sequence, the position being calculated as an offset from a preceding mismatch in the read.

30. The calculation is determining, by the one or more computers, whether the offset is greater than a maximum encodable value; 30. The system of claim 29, further comprising: inserting, by the one or more computers, at least one false mismatch between the particular mismatch and the previous mismatch based on determining that the offset is greater than the maximum encoded value.

31. The calculation is The system of claim 26, further comprising, based on determining that the number of mismatches does not meet the predetermined mismatch threshold number, encoding, by one or more computers, a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

32. The calculation is The system of claim 26, further comprising encoding, by one or more computers, at least a portion of the read record using information entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to the reference sequence.

33. Determining, by the one or more computers, whether the number of mismatches in the incompletely mapped reads meets a predetermined threshold number of mismatches, 27. The system of claim 26, further comprising determining, by the one or more computers, whether the number of mismatches in the incompletely mapped reads is greater than the predetermined threshold number of mismatches.

34. 1. A computer-readable storage device having stored thereon instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform an operation for compressing genome sequence data, the operation comprising: accessing a storage device that stores a plurality of read records in a manner that preserves the sequence ordering of the read records as generated by the mapping and alignment module; For each particular lead record of the plurality of lead records: obtaining the particular lead record; determining whether the particular read record corresponds to a read that is perfectly mapped to the reference sequence or that is incompletely mapped to the reference sequence; Based on determining that the particular read record corresponds to a read that is incompletely mapped to the reference sequence, determining whether the number of mismatches of the incompletely mapped read meets a predetermined mismatch threshold number; encoding each mismatch of the incompletely mapped read into a compressed read having a predetermined compressed record size based on determining that the number of mismatches meets the predetermined mismatch threshold number; storing the compressed records within the storage device while maintaining the ordering of the read records.

35. Each of the plurality of lead records comprises: data indicating the absolute start position of the aligned reads relative to the reference sequence; Data indicating the length of the read; data indicating whether the read is completely or incompletely mapped; data indicating the number of mismatches identified in the reads; data indicating whether the read contains at least one undetermined base N; Data indicating the number of undetermined bases N in the read; data indicating whether the read is mapped or unmapped; data indicating the position of the lead record in the sequence of lead records output by the mapping and alignment module; and data indicating the relative positions of the possible mismatches in the reads.

36. 35. The computer-readable storage device of claim 34, wherein the predetermined compressed recording size is 1 byte.

37. Encoding each mismatch of the incompletely mapped read into a compressed record having a size of 1 byte includes, for each particular mismatch: encoding the first two bits of said byte to contain data representing an alternative nucleotide or base present in said read in place of a corresponding reference nucleotide or base in said reference sequence; and encoding the remaining 6 bits of said byte to contain data indicating the position of a mismatch in said reference sequence, said position being calculated as an offset from a preceding mismatch in said read.

38. The calculation is determining whether the offset is greater than a maximum codable value; 38. The computer-readable storage device of claim 37, further comprising: inserting at least one false mismatch between the particular mismatch and the previous mismatch based on determining that the offset is greater than the maximum encoded value.

39. The calculation is 38. The computer-readable storage device of claim 37, further comprising, based on determining that the number of mismatches does not meet the predetermined mismatch threshold number, encoding a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

40. The calculation is 38. The computer-readable storage device of claim 37, further comprising encoding at least a portion of the read record using information entropy reduction encoding based on determining that the read record corresponds to a read that is fully mapped to the reference sequence.

41. Determining whether the number of mismatches in the incompletely mapped reads satisfies a predetermined mismatch threshold number includes:

38. The computer-readable storage device of claim 37, further comprising determining whether the number of mismatches of the incompletely mapped reads is greater than the predetermined threshold number of mismatches.

42. 1. A hardware processor comprising hardware processing circuitry configured to perform one or more operations, the one or more operations comprising: accessing, by the hardware processing circuitry, a storage device storing a plurality of read records in a manner that preserves the sequential ordering of the read records as generated by the mapping and alignment module; For each particular lead record of the plurality of lead records: obtaining the particular read record by the hardware processing circuit; determining, by the hardware processing circuitry, whether the particular read record corresponds to a read that is perfectly mapped to the reference sequence or that is incompletely mapped to the reference sequence; determining, by the hardware processing circuit, whether the number of mismatches in the incompletely mapped reads satisfies a predetermined mismatch threshold number based on the determination by the hardware processing circuit that the incompletely mapped reads correspond to the reference sequence; encoding, by the hardware processing circuitry, each mismatch of the incompletely mapped read into a compressed read having a predetermined compressed record size based on determining that the number of mismatches meets the predetermined mismatch threshold number; storing, by the hardware processing circuitry, the compressed records in the storage device while maintaining the ordering of the read records.

43. Each of the plurality of lead records comprises: data indicating the absolute start position of the aligned reads relative to the reference sequence; Data indicating the length of the read; data indicating whether the read is completely or incompletely mapped; data indicating the number of mismatches identified in the reads; data indicating whether the read contains at least one undetermined base N; Data indicating the number of undetermined bases N in the read; data indicating whether the read is mapped or unmapped; data indicating the position of the lead record in the sequence of lead records output by the mapping and alignment module; and data indicating the relative location of the possible mismatch in the read.

44. 43. A hardware processor as in claim 42, wherein the predetermined compressed record size is 1 byte.

45. Encoding each mismatch of the incompletely mapped read into a compressed record having a size of 1 byte includes, for each particular mismatch: encoding, by the hardware processing circuitry, the first two bits of the byte to contain data representing an alternative nucleotide or base present in the read in place of a corresponding reference nucleotide or base in the reference sequence; and encoding, by the hardware processing circuitry, the remaining 6 bits of the byte to contain data indicating the position of a mismatch in the reference sequence, the position calculated as an offset from a previous mismatch in the read.

46. The hardware processor includes: determining, by the hardware processing circuitry, whether the offset is greater than a maximum codable value; 46. ​​The hardware processor of claim 45, further comprising: inserting, by the hardware processing circuitry, at least one false mismatch between the particular mismatch and the previous mismatch based on determining that the offset is greater than the maximum encoded value.

47. The hardware processor includes:

46. ​​The hardware processor of claim 45, further comprising, based on determining that the number of mismatches does not meet the predetermined mismatch threshold number, encoding, by the hardware processing circuitry, a list of positions of the reference sequence corresponding to each position of the mismatch relative to the reference sequence using an information entropy reduction encoding process.

48. The hardware processor includes:

46. ​​The hardware processor of claim 45, further comprising, based on determining that the read record corresponds to a read that is fully mapped to the reference sequence, encoding, by the hardware processing circuitry, at least a portion of the read record using information entropy reduction encoding.

49. Determining, by the hardware processing circuitry, whether the number of mismatches of the incompletely mapped reads meets a predetermined mismatch threshold number, 46. ​​The hardware processor of claim 45, further comprising determining, by the hardware processing circuitry, whether the number of mismatches of the incompletely mapped reads is greater than the predetermined threshold number of mismatches.

Images