Quality Score Compression

Abstract

A method, system, and computer program for compressing nucleic acid sequence data, the method may include: obtaining nucleic acid sequence data representing (i) a sequence read and (ii) a plurality of quality scores; determining whether the sequence read includes at least one N base; and, based on a determination that the sequence read does not include at least one N base, generating a first coded data set by encoding each of the quality scores of the sequence read using an (x minus 1)-ary number using a first coding process, where x is an integer representing the number of different quality scores used by a nucleic acid sequencing device; and encoding the first coded data set using a second coding process, thereby compressing the data to be compressed.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Patent Application No. 63 / 110,308, filed November 5, 2020, the entire contents of which are incorporated herein by reference. (Technical field) The present invention relates to quality score compression. [Background technology]

[0002] In some cases, genome sequencing describes a method for identifying nucleotides or other component parts of genome data. A computer can be used to analyze one or more sets of genome data and correlate collections of component parts, such as nucleotides, with their respective locations within a given reference genome. In this way, the computer can "map" a collection of molecular markers onto the reference genome. Summary of the Invention [Means for solving the problem]

[0003] Generally, the present disclosure relates to methods, systems, and computer programs for compressing quality scores generated by a sequencing engine based on genomic data. In one implementation, quality scores generated by a sequencing engine based on genomic data can be compressed by first grouping one or more quality scores into a single data item within an array of data items representing a set of quality scores. The array of data items can then be further compressed or encoded into a final compressed form.

[0004] According to one innovative aspect of the present disclosure, a method for compressing nucleic acid sequence data is disclosed. In one aspect, the method includes: acquiring, by one or more computers, nucleic acid sequence data representing (i) a sequence read including data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) a plurality of quality scores, each quality score of the plurality of quality scores indicating a likelihood that a particular base call of the sequence read was correctly generated by the nucleic acid sequencing device; determining, by the one or more computers, whether the sequence read includes at least one "N" base; and based on a determination that the sequence read does not include at least one "N" base, generating, by the one or more computers, a first coded data set by encoding each of the sequence read quality scores using an (x-1)-ary number using a first coding process, where x is an integer representing the number of different quality scores used by the nucleic acid sequencing device; and encoding, by the one or more computers, the first coded data set using a second coding process, thereby compressing the data to be compressed.

[0005] Other versions include corresponding systems, apparatus, and computer programs for performing the actions of the methods defined by instructions encoded on a computer-readable storage device.

[0006] These and other versions may optionally include one or more of the following features: For example, in some implementations, x is equal to 3.

[0007] In some implementations, the first encoding process can include encoding, by one or more computers, each set of five quality scores of the plurality of quality scores for the read sequence into a single byte by representing each quality score in the set of five quality scores as a base 3 number.

[0008] In some implementations, the method may further include, based on the determination that the read sequence includes at least one "N" base, generating a second coded data set by encoding, by one or more computers, each set of four quality scores of the read sequence into a single byte of memory using a third coding process, and encoding, by one or more computers, the second coded data set using a fourth coding process.

[0009] In some implementations, the second encoding process and the fourth encoding process are the same.

[0010] In some implementations, the acquired data may include a FASTQ file.

[0011] In some implementations, the first coded data set is a compressed version of the plurality of quality scores.

[0012] In some implementations, the second encoding process is a compression process.

[0013] In some implementations, the compression process includes an implementation of a partial matching prediction (PPMD) range encoder.

[0014] In some implementations, for a given value of the first coded data set, the given value is compressed according to a 4-bit context relative to the position of the given value in the first coded data set.

[0015] According to another innovative aspect of the present disclosure, another method for compressing nucleic acid sequence data is disclosed. In one aspect, the method includes: obtaining, by one or more computers, (i) a sequence read including data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) a plurality of quality scores, each quality score of the plurality of quality scores indicating a likelihood that a particular base call in the sequence read was correctly generated by the nucleic acid sequencing device; determining, by the one or more computers, a frequency of occurrence of each quality score group in the plurality of quality scores, each quality score group including a subset of quality scores of the plurality of quality scores; and, for each particular quality score in a first subset of the plurality of quality scores, performing, by the one or more computers, a quality score grouping. The method may include determining that the core is a member of a particular quality score group having an occurrence frequency that meets a predetermined threshold; generating, by one or more computers and using a predetermined group mapping, first data to be used as a single entry in a reduced array based on the determination that the quality score is a member of a particular quality score group having an occurrence frequency that meets the predetermined threshold, wherein the first data to be used as a single entry in the reduced array represents the particular quality score group; and generating, by one or more computers, a reduced array by aggregating the first data generated for each of the particular quality score groups.

[0016] Other versions include corresponding systems, apparatus, and computer programs for performing the actions of the methods defined by instructions encoded on a computer-readable storage device.

[0017] These and other versions may optionally include one or more of the following features: For example, in some implementations, the acquired data includes FASTQ files.

[0018] In some implementations, each quality score in the plurality of quality scores is data representing an ASCII value of the quality score.

[0019] In some implementations, the method further includes, for each particular quality score in the second subset of the plurality of quality scores, determining, by one or more computers, that the particular quality score in the second subset of the array of quality scores is not a member of a particular quality score group having an occurrence frequency that meets a predetermined threshold; and generating, by the one or more computers and using a predetermined single mapping, second data to be used as a single entry in the reduced array, wherein the second data to be used as the single entry in the reduced array represents a quality score that is not a member of a particular quality score group having an occurrence frequency that meets the predetermined threshold, and the predetermined single mapping defines a one-to-one mapping between each of the plurality of single quality scores and a corresponding single entry.

[0020] In some implementations, generating the reduced array by one or more computers may include aggregating, by one or more computers, first data generated for each of the particular quality score groups, and aggregating, by one or more computers, second data generated for each quality score that is not a member of the particular quality score group having an occurrence frequency that meets a predetermined threshold.

[0021] In some implementations, the method can further include identifying, by the one or more computers, a plurality of quality score groupings in the plurality of quality scores.

[0022] In some implementations, the predetermined group mapping defines a one-to-one mapping between each of a plurality of different quality score groups and a corresponding single entry.

[0023] According to another innovative aspect of the present disclosure, another method for compressing nucleic acid sequence data is disclosed. In one aspect, the method includes: acquiring, by one or more computers, nucleic acid sequence data representing (i) sequence reads including data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) a plurality of quality scores, each quality score of the plurality of quality scores indicating a likelihood that a particular base call of the sequence read was correctly identified by the nucleic acid sequencing device; determining, by the one or more computers, a frequency of occurrence of each quality score group in the plurality of quality scores, wherein each quality score group includes a subset of quality scores of the plurality of quality scores; and, for each particular quality score in a first subset of the plurality of quality scores, determining, by the one or more computers, a frequency of occurrence of each quality score group in the first subset of the plurality of quality scores. determining that a particular quality score in the table is not a member of a particular quality score group having a frequency of occurrence that meets a predetermined threshold; and generating, by one or more computers and using a predetermined single mapping, first data to be used as a single entry in a reduced array, wherein the first data to be used as the single entry in the reduced array represents a quality score that is not a member of the particular quality score group having a frequency of occurrence that meets the predetermined threshold; and generating, by the one or more computers, aggregating the generated first data for each quality score that is not a member of the particular quality score group having a frequency of occurrence that meets the predetermined threshold.

[0024] Other versions include corresponding systems, apparatus, and computer programs for performing the actions of the methods defined by instructions encoded on a computer-readable storage device.

[0025] These and other versions may optionally include one or more of the following features: For example, in some implementations, the acquired data includes FASTQ files.

[0026] In some implementations, each quality score in the plurality of quality scores is data representing an ASCII value of the quality score.

[0027] In some implementations, the method may further include: for each particular quality score in the second subset of the plurality of quality scores, determining by one or more computers that the quality score is a member of a particular quality score group having an occurrence frequency that meets a predetermined threshold; and, based on the determination that the quality score is a member of a particular quality score group having an occurrence frequency that meets the predetermined threshold, generating, by the one or more computers and using a predetermined group mapping, second data to be used as a single entry in the reduced array, wherein the second data to be used as the single entry in the reduced array represents a particular quality score group, and the predetermined group mapping defines a one-to-one mapping between each of the plurality of different quality score groups and a corresponding single entry.

[0028] In some implementations, generating the reduced array by one or more computers may include aggregating first data generated by the one or more computers for each quality score that is not a member of a particular quality score group having an occurrence frequency that meets a predetermined threshold, and aggregating second data generated by the one or more computers for each of the particular quality score group.

[0029] In some implementations, the method can further include identifying, by the one or more computers, a plurality of quality score groupings in the plurality of quality scores.

[0030] In some implementations, the predetermined single mapping defines a one-to-one mapping between each of the multiple single quality scores and a corresponding single entry.

[0031] These and other innovative aspects of the present disclosure are described herein below with reference to the detailed description, drawings, and appended claims. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 illustrates one embodiment of a system for encoding, e.g., compressing, an array of quality scores having a first data format. [Figure 2] FIG. 1 is a flow diagram illustrating an example of a process for encoding, e.g., compressing, an array of quality scores having a first data format. [Figure 3] FIG. 1 illustrates one embodiment of a system for encoding, e.g., compressing, an array of quality scores having a second data format. [Figure 4] FIG. 10 is a flow diagram illustrating one embodiment of a process for encoding, e.g., compressing, an array of quality scores having a second data format. [Figure 5] FIG. 1 is a flow diagram illustrating an embodiment of a process for recovering an array of quality scores having a first data format. [Figure 6] FIG. 10 is a flow diagram illustrating an embodiment of a process for recovering an array of quality scores having a second data format. [Figure 7] FIG. 10 is a flow diagram illustrating an embodiment of a process for determining how to compress quality scores. [Figure 8] 1 is a graphical representation of experimental results of a process for encoding an array of quality scores having a first data format. [Figure 9] 10 is a graphical representation of experimental results of a process for encoding an array of quality scores with a second data format. [Figure 10] FIG. 1 is a diagram of computer system components that may be used to implement a process for encoding an array of quality scores having a first data format. DETAILED DESCRIPTION OF THE INVENTION

[0033] Like reference numbers and designations in the various drawings indicate like elements. The present disclosure is directed to methods, systems, and computer programs for compressing data representing a sequence of quality scores for sequence reads generated by a nucleic acid sequencing device. Each quality score in the sequence of quality scores provides an indication of the likelihood that the corresponding base in the sequence read was correctly sequenced by the nucleic acid sequencing device. The methods and systems disclosed herein enable faster compression speeds and smaller compression ratios compared to conventional methods that do not utilize the technology described herein. The faster compression speeds and smaller compression ratios are achieved by performing a pre-compression encoding step to reduce the size of the input data representing the sequence of quality scores processed by the compression engine. Because the compression engine receives and processes reduced-size input representing the sequence of quality scores, the compression engine can compress the input data faster and achieve smaller compressed files compared to conventional methods. Thus, the compression methods of the present disclosure can achieve higher compression ratios than conventional systems, where the compression ratio is equal to the uncompressed file size divided by the compressed file size.

[0034] In general, the present disclosure describes systems and methods that perform an initial encoding stage on data representing a quality score array before inputting the data representing the quality score array into a compression engine. The advantages of this approach can be explained with reference to a specific example. In one or more first implementations, each quality score in the array of quality scores may be represented as an 8-bit (or one-byte) ASCII value. In such first implementations described herein, the present disclosure can perform an initial encoding for such 8-bit representations of quality scores in the array of quality scores that reduces the 8-bit representations of the quality scores to 2-bit or 1.6-bit representations of the quality scores, thus enabling four or five quality scores, respectively, to be coded into a single byte. Such an initial encoding stage can thus significantly and predictably reduce the input data size to the compression engine by one-quarter or one-fifth the size of the initial representations of the quality scores.

[0035] However, this disclosure is not limited to reducing quality scores from 8-bit representations to either 2-bit or 1.6-bit representations for input to a compression engine. Rather, similar percentage reductions can be achieved with other size representations of quality scores. These examples are provided in part to highlight the technical improvements achieved by the present disclosure.

[0036] Furthermore, other second implementation aspects of the present disclosure describe other early-stage coding engines that perform operations on data representing arrays of quality scores to generate a reduced set of arrays for input to the compression engine. Such second implementation aspects provide similar technical benefits as the above-described implementations (e.g., faster compression speeds and higher compression ratios compared to conventional methods). However, due to the variable and initial size reduction of the input data before input to the compression engine associated with the voluntary grouping or degrouping of quality scores, these second implementation aspects still result in faster compression speeds, lower compressed file sizes, and higher compression ratios than conventional methods, and may ultimately have faster speeds and compression ratios that are less predictable than the above-described first implementation aspect of the present disclosure, which may have defined data sizes at each stage.

[0037] Generally, the term "encoding," as used herein, refers to a process performed by one or more software engines, one or more hardware engines (e.g., processors), or a combination thereof, that includes receiving a first set of data and processing the first set of data to generate a second set of data that represents the first set of data in a different form. In some embodiments, the second set of data may be stored in less memory than the received first set of data. For example, one form of encoding data may include compressing the data, e.g., using a compression engine, to a size that is smaller than the size of the data before compression.

[0038] FIG. 1 illustrates an example system 100 for compressing a sequence of quality scores having a first data format. The first data format can assign “X” different quality scores to corresponding bases in a sequence read, where “X” is any positive integer less than a given threshold. The given threshold can be determined based on the number of unique quality scores generated using the system 100 shown in FIG. 1 that is more effective or practical than other systems, such as the system 300 shown in FIG. 3 . For example, the given threshold can be 8. If “X” is less than 8, the corresponding quality score can be processed by the system 100. As a specific example, in some implementations, “X” can be equal to 4, indicating that the first data format can use any one of four different quality scores to indicate the likelihood that one or more base calls in a sequence read generated by a nucleic acid sequencing device are correct. In some implementations, the likelihood can include a probability that a sequencing error occurred in one or more base calls corresponding to the quality scores. In some embodiments, the sequencing error can include a base call made by the nucleic acid sequencing device for a particular location in the sequence read that is incorrect. For example, a sequencing device may determine a base call of adenine, represented by the letter A, for a particular location in a sequence read when in fact the correct base call should be cytosine, represented by the letter C. A low quality score for a given base call may indicate that such an error is more likely, while a high quality score may indicate that such an error is less likely. A base call can include data generated by a nucleic acid sequencing device that represents a particular nucleotide in a sequence read.

[0039] The system 100 is configured to receive input data 102 from one or more data sources. In some implementations, the one or more data sources may include a nucleic acid sequencing device. The nucleic acid sequencing device may be a next-generation sequencing device such as Novaseq® 6000, Nextseq® 2000, or the like. In other implementations, the one or more data sources may include one or more processors running on a computing device such as a tablet computer, a desktop computer, one or more server computers, or a combination thereof. In some implementations, the input data 102 may be received from one or more data sources via one or more networks. The one or more networks may include a wired Ethernet network, a wired optical network, a wireless network, a LAN, a WAN, a Wi-Fi network, a cellular network, the Internet, or any combination thereof. In some implementations, the input data 102 may be received from one or more data sources via a direct connection, such as a USB cable connection, a USB-C cable connection, or the like. In still other implementations, the entire system 100 may be hosted within one or more data sources. For example, in some implementations, the entire system 100 may be hosted by a nucleic acid sequencing device.

[0040] The system 100 can include an input engine that receives input data 102. The input data 102 can include a plurality of records, each including data describing a sequence read, including data corresponding to a series of base calls and data describing a series of nucleotide or base quality scores. A base call can include data generated by a nucleic acid sequencer device that corresponds to or represents a nucleotide of a DNA fragment sequenced by the nucleic acid sequencer device. However, for purposes of this disclosure, the terms base call and nucleotide can be used interchangeably throughout to refer to data generated by a nucleic acid sequencer that corresponds to a letter, such as A, C, T, or G, in a sequence read. The meaning of such letters is explained in more detail below.

[0041] In each record, each quality score in the array of quality scores can correspond to a particular nucleotide or base in the read sequence. For example, in the example of FIG. 1 , the first quality score "F" in the quality score sequence "F#FFFF...F;FF" corresponds to the first nucleotide or base in the read sequence "CNTGTA...ATAAG." In some implementations, the input data 102 can include one or more FASTQ files, where each record of the plurality of records can include a portion of a FASTQ file, referred to herein as a FASTQ record. Each portion of a FASTQ file can include one read sequence and a corresponding array of quality scores for the read sequence.

[0042] Each read sequence can be previously generated by one or more nucleic acid sequencing devices from sequencing a biological sample by one or more nucleic acid sequencers. The biological sample can include a nucleic acid sample from any organism, such as a human, animal, or plant. Each read sequence includes a string of characters from a defined vocabulary. For example, a minimal vocabulary can be represented by a set of five symbols: {A, C, G, T, N}. The letters A, C, G, and T represent the four types of nucleotides present in deoxyribonucleic acid (DNA): adenine, cytosine, guanine, and thymine. In ribonucleic acid (RNA), thymine is replaced by uracil (U). The letter "N" can be used by the nucleic acid sequencing device to indicate that the sequencing device was unable to determine any base at the particular location in the read sequence occupied by "N," thereby leaving the actual and precise nature of the position in the read sequence undetermined. The use of the letters A, C, G, and T or U is common because these letters represent the first letter of each of the respective nucleotides. However, the present disclosure is not limited to the use of the letter "N" to represent positions within a generated read sequence that have not been determined. Instead, a nucleic acid sequencing device can use any letter or symbol to represent a location within a read sequence where the nucleic acid sequencing device cannot accurately determine the correct base. It is understood that in the implementations described herein, a letter or symbol used to represent an unknown base is equivalent to using the letter "N."

[0043] Storing quality scores for bases in sequence reads can have many useful applications. However, considering the sequenced genome of an organism such as a human, this may include over 3 billion bases and corresponding quality scores. The data corresponding to the resulting set of quality scores can be very large (e.g., gigabytes to terabytes, depending on the sequencing depth) and may require compression to efficiently store, transmit, or archive the quality score information. Aspects of the present disclosure, described with reference to FIG. 1, provide a multi-stage compression method that relies on an initial coding engine to prepare an input data set for a compression engine. By first coding the data based on the unique characteristics of the input data 102 before use in later stages of the compression engine, the system 100 can achieve high compression ratios while maintaining fast compression rates. The resulting compressed quality scores not only reduce memory footprints, which can reduce associated memory storage operational costs, but also increase processing speeds when accessing or analyzing the resulting compressed quality scores. For purposes of this disclosure, an "engine" may include one or more software modules, one or more hardware modules, or any combination thereof.

[0044] In stage A, the classifier engine 104 can obtain the input data 102 and forward targets or specific portions of the input data 102 to different initial quality score coding engines based on attributes of the specific portions of the input data 102. Within the context of this disclosure, the process of obtaining refers to the process of receiving, retrieving, or otherwise obtaining. In some implementations, the classifier engine 104 can function as a decision engine that determines whether each specific portion of the input data 102 should be provided to the initial quality score coding engine V1 106 or the initial quality score coding engine v2 118. After performing their respective initial coding operations on the portions of the input data 102 they receive, each of the initial coding engines V1, V2 can provide a respective coded output 114, 126 as an input to the compression engine 116. The compression engines can process the coded outputs 114, 126 that they receive as input and can generate a final output 128, which is a compressed version of the input data 102.

[0045] In more detail in step A of FIG. 1 , the input data 102 acquired by the classifier engine 104 can include a first record 102a, a second record 102b, and an i-th record 102c. In the example of FIG. 1 , the first record 102a, the second record 102b, and the i-th record 102c can be part of a FASTQ file, also referred to herein as a FASTQ record. The FASTQ record can include a record header (e.g., "@A0:90:H46:1...") that identifies the FASTQ file from which the FASTQ record originates and distinguishes the FASTQ record from other FASTQ records. The FASTQ record can include data representing a read sequence generated by a nucleic acid sequencer. The FASTQ record can also include a quality score sequence corresponding to the sequence of the FASTQ record. The FASTQ record can further include one or more delimiters to separate one or more data components, such as a record header, a quality score sequence, a sequence, etc.

[0046] In the example of Figure 1, the input data 102 is shown with three data records, but any number of records may be included within the input data 102. Although these three data records are shown in the subsequent process of Figure 1, any number of records may be processed in a similar manner. In stage B, the classifier engine 104 may determine a classification for each of the first record 102a, the second record 102b, and the i-th record 102c of the input data 102 based on one or more classification rules. In some implementations, the classification may be determined based on determining the sequence of the reads within each FASTQ record.

[0047] In some implementations, for example, the classifier engine 104 can classify or forward each FASTQ record to the initial quality score coding engines V1 and V2 based on whether the read sequence in the FASTQ record contains at least one "N" base. If the read sequence of the FASTQ record is determined to contain at least one "N" base, the FASTQ record can be forwarded to the initial quality score coding engine V1 106. Alternatively, if the read sequence of the FASTQ record is determined to not contain at least one "N" base, the FASTQ record can be forwarded to the initial quality score coding engine V2 118. However, this is merely one example of a classifier rule, and it is contemplated that other types of classification rules can be used to classify and forward FASTQ records between the initial quality score coding engine V1 and the initial quality score coding engine V2 in accordance with the present disclosure. In some implementations, the classifier engine 104 classifies the input data using other elements of the input data. For example, instead of classifying based on the presence or absence of the "N" base, the classifier engine 104 can classify based on a percentage or determined portion of the "N" base or other bases. Additionally, the classifier engine 104 can use other base calls or corresponding data, such as quality scores, to classify or forward one or more FAST Q records or associated data.

[0048] 1 , the classifier engine 104 can determine in step B, based on the base calls of the respective sequence reads of the first FASTQ record 102a and the i-th FASTQ record 102c, that both the first FASTQ record 102a and the i-th FASTQ record 102c include data representing a sequence read having at least one "N" base. Based on the determination that the first FASTQ record 102a and the i-th FASTQ record 102c each have at least one "N" base, the classifier engine 104 can forward the first FASTQ record 102a and the i-th FASTQ record 102c to the initial quality score encoding engine v1 106. In some implementations, the classifier engine 104 forwards data corresponding to the first FASTQ record 102a and the i-th FASTQ record 102c, for example, without transmitting one or more complete FASTQ records corresponding to the first FASTQ record 102a or the i-th FASTQ record 102c. After forwarding the data corresponding to the first FASTQ record 102a and the i-th FASTQ record 102c to the initial quality score coding engine v1 106, execution of the system 100 can continue to stage C.

[0049] In step C, the initial quality score encoding engine v1 106 can obtain quality score sequences for the first FASTQ record 102a and the i-th FASTQ record 102c, respectively, corresponding to the read sequences of the first FASTQ record 102a and the i-th FASTQ record 102c. In this example, the quality score sequence received by the initial quality score encoding engine v1 106 can include four quality scores, each represented by the 8-bit ASCII values ​​"F," ":," ",", and "#," where "#" represents a quality score for an "N" base. However, in other implementations, there can be fewer or more than four quality scores, using other characters or symbols that can be used to represent similar information.

[0050] The initial quality score encoding engine v1 106 may perform an initial encoding on the array of quality scores in the first record 102a in stage C. This initial encoding performed by the initial quality score encoding engine v1 106 may encode each quality score from the array of quality scores in the first FASTQ record 102a. This initial encoding may include encoding an 8-bit ASCII representation of each quality score "F," ":," ",", or "#" into a 2-bit representation of each respective quality score. In some implementations, encoding each 8-bit ASCII representation of each quality score in the quality score array of the first FASTQ record 102a may result in the initial quality score encoding engine 106 generating output data 114 including four quality scores per byte. This encoding ratio produced by the initial quality score encoding engine v1 106 reduces the size of the input data record to the compression engine 116 by a factor of four. In other implementations, other compression ratios may be achieved by compressing larger or smaller quality scores, or quality scores of different data sizes, into one or more bits of information.

[0051] The initial quality score encoding engine v1 106 can continue to perform the initial encoding process for each FASTQ record forwarded to the initial quality score packing engine v1 106. For example, the initial quality score encoding engine v1 106 can receive the i-th FASTQ record 102c and encode the quality score from the array of quality scores from an 8-bit ASCII value to a 2-bit representation of the quality score. This process can continue until each of the array of quality scores received by the initial quality score packing engine v1 has been processed to generate an initial encoding into a 2-bit quality score.

[0052] More specifically, the initial quality score encoding engine v1 106 can perform encoding of 8-bit quality scores into an output binary stream 114 of 2-bit quality scores by performing the processes shown in items 108, 110, and 112. The quality score encoding engine v1 106 can obtain the first four quality scores from the first record 102a. In some implementations, this can include the initial quality score encoding engine v1 106 obtaining ASCII values ​​for each of the quality scores. In other implementations, this can include obtaining other representations of the quality scores and mapping each quality score to a corresponding ASCII value as shown in 110. For example, the first four quality scores "F," "#," "F," and "F" can be mapped by the initial quality score encoding engine v1 106 to their corresponding ASCII values ​​of 70, 35, 70, and 70. The mapping of other quality scores using the first data format with "N" is shown in Table 110. The ASCII representation of the quality score can then be mapped to each quality score category, with one category corresponding to each category of quality score. In this example, the quality score categories are represented by 0, 1, 2, and 3 because there are four possible quality score ranges. However, there may be more (e.g., 0, 1, 2, 3, 4, 5, etc.) or fewer categories (e.g., 0, 1) based on the quality scoring system in use. In this example, 0 corresponds to an unidentified base and 3 corresponds to a high base quality score, although the disclosure is not limited to this implementation. Rather, for example, instead of integers, quality scores can be represented using percentages of 100%, language-based score values ​​such as low, medium, and high, or other designations known in the art.

[0053] In the example of Figure 1, the initial quality score encoding engine v1 106 may perform calculations using quality score values ​​3, 0, 3, and 3, which correspond to the four ASCII-based quality scores "F", "#", "F", and "F". The initial quality score encoding engine v1 106 may perform calculations using q1+4 * q2+42* q3+4 3* The quality coding score can be calculated using a formula such as q1, q2, q3, and q4, where q1, q2, q3, and q4 each represent a quality score value. For quality score values ​​of 3, 0, 3, and 3, the formula is 3 + 4, which equals 243 as shown in item 112. * 0+4 2* 3+4 3* 3. The initial quality score encoding engine v1 106 can generate a corresponding binary representation of these four quality scores by generating a binary representation of the number 243. This binary representation is 11110011, which can then be added to the output binary stream 114. This process can continue iteratively until each of the quality scores in the array of quality scores for the first FASTQ record 102a has been initially coded into the output binary stream 114.

[0054] After initial coding of the sequence of quality scores for the first FASTQ record 102a, the initial quality score coding engine v1 106 can continue to perform the initial coding process for each subsequent FASTQ record received. For example, the initial quality score coding engine v1 106 can continue to perform initial coding of the i-th FASTQ record 102c in the same manner as described above with reference to the first FASTQ record 102a. The initial quality score coding engine v1 106 can continue to obtain subsequent quality scores and perform initial coding in a similar manner to the four initial quality scores shown in item 108.

[0055] In stage D, the initial quality score coding engine v1 106 may send the output binary stream 114 to the compression engine 116. The compression engine 116 may then perform a subsequent compression on the output binary stream 114 corresponding to a predetermined compression method. Generally, any compression process may be used by the compression engine 116 to further compress or condense the size of the output binary stream 114. For example, in some implementations, the compression engine 116 may perform compression using level 11 of the Zstandard (ZSTD) library. However, the present disclosure is not so limited. Instead, in some implementations, other compression methods may be used, including other levels of the ZSTD library as well as other compression libraries. Generally, any compression method or combination of compression methods known in the art may be used.

[0056] In another example, the classifier engine 104 can determine that the received FASTQ record includes a read sequence that does not include at least one "N" by applying one or more classification rules. Based on a determination that the received FASTQ record does not include at least one "N," the classifier engine 104 can forward data corresponding to the second FASTQ record 102b to the initial quality score encoding engine v2 118.

[0057] Because the second FASTQ record 102b does not contain any N, the range of quality score candidates can be the number of quality score candidates minus 1. That is, if the classifier engine 104 determines that the second FASTQ record contains "X" quality score candidates, the quality score candidates that can be processed by the initial quality score encoding engine v2 118 are "X" minus 1. In this implementation, the quality score array has only three distinct quality scores. Therefore, in this example, only three distinct quality scores are compressed separately to achieve a higher compression ratio, in part.

[0058] In stage E, the initial quality score encoding engine v2 118 may obtain the quality score array of the second record 102b. Item 120 shows a subset of the quality scores of the quality score array of the second record 102b. More specifically, the initial quality score encoding engine v2 118 may perform encoding of the 8-bit quality scores into an output binary stream 126 of 1.6-bit quality scores by performing the process shown in items 120, 122, and 124. The initial quality score encoding engine v2 118 may obtain a determined number of quality scores, e.g., five quality scores, from the second FASTQ record 102b. In some implementations, this may involve the initial quality score encoding engine v2 118 simply obtaining an ASCII value for each of the quality scores. In other implementations, this may involve obtaining another representation of the quality scores and mapping each respective quality score to a corresponding ASCII value, as shown at 122. For example, the first five quality scores ":," "F," ":," ",," and "F," may be mapped by the initial quality score encoding engine v2 118 to their corresponding ASCII values ​​of 58, 70, 58, 44, and 70.

[0059] The ASCII representation of the quality score can then be mapped to each quality score category, with one category corresponding to each category of quality score. In this example, the quality score categories are represented as 0, 1, and 2 because there are a range of three possible quality scores (i.e., X possible quality scores minus 1 because there are no "N" bases in the FASTQ record transferred to the initial quality score encoding engine v2 118). In this example, the initial quality score encoding engine v2 118 can encode the 8-bit ASCII quality score into a 1.6-bit quality score by representing the 8-bit ASCII quality score as a base 3 number. Here, base 3 is used because there are three unique categories of quality score.

[0060] However, the present disclosure is not limited to the above-described examples. Alternatively, in an implementation with more quality score categories, such as a total of eight unique quality score categories, the initial quality score encoding engine v2 118 can obtain a sequence with seven unique quality score categories, where the original eight unique quality score categories include quality scores for "N" bases. In such an implementation, a base-7 number can be used to represent the initial encoding of the initial quality score encoding engine v2 118.

[0061] Similar to item 112, item 124 indicates the calculations performed by the initial coding engine 118 related to the first set of quality scores. In this case, the initial quality score packing engine v2 118 calculates q1+3 where q1, q2, q3, q4, and q5 each represent a quality score value. * q2+3 2* q3+3 3* q4+3 4* Calculate the quality packing score using a formula such as q5. For quality score values ​​of 1, 2, 1, 0, and 2, the formula is 1 + 3, which equals 178 as shown in item 124. * 2+3 2* 1+3 3* 0+3 4* 2. The binary representation of 178, for example, 10110010, can then be added to the output binary stream 126.

[0062] 1, the Initial Quality Score Packing Engine v2 118 encodes five 8-bit quality scores into a single 1-byte ASCII character, thereby compressing each quality score in the second record 102b by a factor of 5. In other implementations, other compression ratios may be achieved by compressing larger or smaller quality scores, or quality scores of different data sizes, into one or more bits of information.

[0063] Similar to the initial quality score coding engine v1 106, the initial quality score coding engine v2 118 can obtain subsequent quality scores and continue to perform coding in a manner similar to the initial five quality scores shown in item 120. In stage F, the initial quality score coding engine v2 118 can provide an output binary stream 126 as input to the compression engine 116. The compression engine 116 can then perform subsequent compression on the output binary stream 126 corresponding to a predetermined compression method. For example, in some cases, the compression engine 116 performs compression based on an implementation of a Prediction by Partial Matching of a Range Encoder (PPMD) for compressing byte strings. In some implementations, other compression methods known in the art can be used instead of or in addition to a PPMD ​​implementation. In general, the compression engine 116 can use any compression or combination of compression methods, such as any compression method known in the art.

[0064] In some implementations, each symbol in the output binary stream 126 supplied to the range encoder may be calculated according to a 4-bit context, representing the number of highest quality score values ​​in the previous score. For example, the 4-bit context may be calculated as the total number of highest quality scores in the 30 previous quality scores divided by 2. This particular 4-bit context formula can obtain values ​​ranging from 0 to 15 that fit the 4-bit context. By using a 4-bit context or other context-based coding approach, the system 100 can take into account multiple adjacent instances of a particular quality score, e.g., the highest score "F," which may be a good predictor of a future score. In some implementations, larger or smaller contexts are used depending on memory, computation, or other requirements. For example, a larger context may result in a higher compression ratio but may require more memory and computation time.

[0065] In some implementations, other coding engines can be used within the system 100. For example, in addition to the initial quality score coding engine v1 106 and the initial quality score coding engine v2 118, an implementation can include a third coding engine based on specific criteria defined by the classifier engine 104. In some implementations, more than three engines can be used. In some implementations, the classifier engine 104 can determine a third set of records in the input data 102 that uses only two distinct quality scores. In this way, the system 100 can achieve even higher compression ratios for groups that have only two distinct quality scores.

[0066] For ease of explanation, steps A through G are used. The process performed by system 100 may occur in the order shown by steps A through G, although in other implementations, the order of certain steps may differ. In some implementations, two or more steps may occur simultaneously.

[0067] 2 is a flow chart illustrating one embodiment of a process 200 for compressing quality scores based on a first input data format. Process 200 may be implemented by one or more electronic systems, such as system 100 of FIG.

[0068] Process 200 includes obtaining (202) a genetic read sequence generated by a genetic sequencing device. For example, as shown in FIG. 1, input data 102 is obtained and provided to system 100. Input data 102 can include one or more records, each record including data representing a read sequence, including data corresponding to a plurality of base calls generated by the nucleic acid sequencing device, as well as data describing quality scores for the plurality of base calls. Each quality score corresponds to a particular base call in the read sequence.

[0069] In some implementations, letters are used to represent quality scores in the input data 102. For example, the first quality score "F" in the quality score sequence "F#FFFF...F;FF" of the first FASTQ record 102a corresponds to the first nucleotide or base (hereinafter referred to as "base") in the read sequence "CNTGTA...ATAAG" of the first FASTQ record 102a. In some implementations, the input data 102 can include one or more FASTQ files, where each record of the multiple records can include a portion of the FASTQ record, referred to herein as a FASTQ record. Each portion of the FASTQ file can include one read sequence and a corresponding sequence of quality scores for the read sequence.

[0070] The process 200 can include obtaining (204) a plurality of quality scores corresponding to the genetic read sequence. For example, as shown in FIG. 1, the input data 102 includes a first record 102a, a second record 102b, and an i-th record 102c. Each of the first record 102a, the second record 102b, and the i-th record 102c includes both a genetic sequence and a quality score sequence corresponding to the genetic sequence. For example, the first record 102a includes the genetic sequence "CNTGTA...ATAAG" and a corresponding quality score sequence of "F#FFFF...F:,FF," where each value in the quality score sequence indicates the likelihood of a sequencing error at a particular location in the corresponding genetic sequence.

[0071] The process 200 includes determining (206) that the genetic read sequence includes at least one "N" base. For example, as shown in FIG. 1, the classifier engine 104 obtains input data 102 including one or more genetic read sequences. The input data 102 includes a first record 102a. The first record 102a includes a genetic sequence "CNTGTA...ATAAG." The genetic sequence "CNTGTA...ATAAG" includes the base "N." The classifier engine 104 can determine that the genetic sequence "CNTGTA...ATAAG" of the first record 102a includes the base "N," and can transfer data corresponding to the first record 102a to the initial quality score encoding engine v1 106. Similarly, the classifier engine 104 can obtain a second record 102b. The second record 102b includes a genetic sequence "GTCTAG...CACTT" that does not include the base "N." The classifier engine 104 can determine that the genetic sequence "GTCTAG...CACTT" of the second record 102b does not contain the base "N" and can forward the data corresponding to the second record 102b to the initial quality score encoding engine v2 118.

[0072] The process 200 includes generating a first coded data set by coding each of the quality scores using an x-ary number, where x is an integer number representing the number of different quality scores used by the genetic sequencing device (208). For example, the initial quality score coding engine v1 106 obtains the quality score array of the first record 102a. The quality score array of the first record 102a includes four unique quality scores: "F," ":," ",", and "#." Other suitable symbols or values ​​can be used in other implementations. The initial quality score coding engine v1 106 can then calculate an integer based on the quaternary number. For example, as shown in item 108 of FIG. 1 , the initial quality score coding engine v1 106 generates a value 3033 corresponding to the genetic quality score "F#FF" based on the mapping shown in item 110. The initial quality score coding engine v1 106 then generates an integer based on the value 3033 as if the value 3033 were written in base-4 notation. As shown in item 112, the resulting integer is equal to 243, which can be written as "11110011" in 8 binary bits. The formula used to generate the binary form of a group of quality scores, such as the quality score "F#FF," can be constructed so that the integer value can be represented using 8 or fewer bits. For example, the formula used by the initial quality score encoding engine v1 106 can be constructed so that the integer value is less than 255.

[0073] In another example, the initial quality score coding engine v2 118 obtains the quality score array of the second record 102b. In contrast to the quality score array of the first record 102a, the quality score array of the second record 102b includes three unique quality scores: “F”, “:”, and “,”. Other suitable symbols or values ​​can be used in other implementations. The initial quality score coding engine v2 118 can then calculate an integer based on the ternary number. Because there are fewer unique quality scores, the initial quality score coding engine v2 118 can code an additional quality score (e.g., 5 instead of 4) into 8-bit binary. For example, as shown in item 124 of FIG. 1 , the initial quality score coding engine v2 118 generates a value 12102 corresponding to the genetic quality score “:F:,F” based on the mapping shown in item 122. The initial quality score coding engine v2 118 then generates an integer based on the value 12102, as if the value 12102 were written in ternary notation. As shown in item 124, the resulting integer is equal to 178, which can be written as "10110010" with 8 binary bits. The formula used to generate the binary form of a group of quality scores, such as the quality score ":F:,F", can be constructed so that the integer value can be represented using 8 or fewer bits. For example, the formula used by the initial quality score coding engine v2 118 can be constructed so that the integer value is less than 255.

[0074] The process 200 includes generating 210 a second coded data set by coding the first coded data set using a second coding algorithm. For example, as shown in FIG. 1 , the compression engine 116 generates the output 128 based on inputs provided by the initial quality score coding engine v1 106 or the initial quality score coding engine v2 118. In some implementations, the compression engine 116 combines multiple outputs from different compression processes to generate the output 128. For example, both the initial quality score coding engine v1 106 and the initial quality score coding engine v2 118 can generate data for the compression engine 116, as shown in coded outputs 114 and 126.

[0075] In some implementations, the compression engine 116 performs one or more types of compression based on the acquired data. For example, the compression process used to compress the output of the initial quality score coding engine v1 106 may differ from the compression process used to compress the output of the initial quality score coding engine v2 118. As discussed herein, the compression of the coded output 114 of the initial quality score coding engine v1 106 may include compression using Level 11 of the Z Standard (ZSTD) library, or other forms of compression. The compression of the coded output 126 of the initial quality score coding engine v2 118 may include compression using an implementation of a Prediction by Partial Matching (PPMD) range encoder, or other forms of compression. In some implementations, the context around a given value of the array may be used to compress a given array. For example, a 4-bit context representing the number of the highest quality score value in the previous score may be used by the compression engine 116. In some cases, multiple compression processes may be combined to generate the compressed output.

[0076] 3 illustrates one embodiment of a system for compressing an array of quality scores having a second data format. System 300 includes a quality score array 302 formatted according to the Q40 data format. Quality score array 302 is processed by a group identification engine 304, a frequency counter engine 310, a reduced array generation engine 316, a single mapping engine 320, a group mapping engine 326, and a compression engine 334 to generate a reduced array 332 and subsequent output 336 that represents a compressed version of quality score array 302.

[0077] In step A of Figure 3, a quality score array 302 is generated and sent to a group identification engine 304. In the example of Figure 3, the quality score array 302 is generated by a sequencer that uses the Q40 data format to encode the quality scores associated with bases, as described above. Generally, some sequencers that use more than a threshold number of unique quality scores can compress the resulting output data using the process performed by system 300 shown in Figure 3 or a similar process. For example, if a sequencer uses more than eight unique quality scores to encode the quality scores corresponding to a genetic sequence, the sequencer can use the encoding and compression process discussed with reference to Figure 3.

[0078] In stage B, group identification engine 304 retrieves quality score array 302 and generates one or more groups based on quality score array 302, as shown in item 306. In this implementation, group identification engine 304 groups adjacent quality scores in groups of three. In other implementations, other numbers of quality scores may be included in one or more groups. As shown in item 316, the first group of three quality scores are the characters "@," "C," and "@," corresponding to characters in quality score array 302. In the example of FIG. 3, each character in quality score array 302 represents a quality score that indicates the likelihood of a sequencing error.

[0079] The group identification engine 304 generates quality score groups 308 and sends the quality score groups 308 to the frequency counter engine 310. In stage C, the frequency counter engine 310 obtains the quality score groups 308 and determines the number of occurrences for each of the groups in the quality score groups 308, as shown in item 312.

[0080] In some implementations, the group identification engine 304 and the frequency counter engine 310 operate at least partially in parallel. For example, the group identification engine 304 can identify a single group based on the quality score array 302. The group identification engine 304 can then send the identified single group to the frequency counter engine 310. The frequency counter engine 310 can then determine the number of occurrences of the identified single group in quality scores. In some cases, generating the groups from the quality score array 302 can include identifying one or more quality scores in the quality score array 302. In general, any process described herein can be threaded or executed concurrently with another process, and two or more processes can be executed on one or more devices or instances of software.

[0081] The frequency counter engine 310 generates quality score group frequencies 314 and sends the quality score group frequencies 314 to the reduced array generation engine 316. The quality score group frequencies 314 may include the number of occurrences of one or more quality scores in the quality score array 302. In stage D, the reduced array generation engine 316 obtains the quality score group frequencies 314 and can communicate with both the single mapping engine 320 and the group mapping engine 326. That is, the reduced array generation engine 316 can use the single mapping engine 320 to generate entries in the reduced array 332 in stage E if a quality score in the quality score array 320 is not part of a group of quality scores that exceeds the threshold number of quality scores. Alternatively, the reduced array generation engine 316 can use the group mapping engine 326 in stage F if a quality score in the quality score array 320 is part of a group of quality scores that exceeds the threshold number of quality scores. For purposes of this disclosure, an "entry" or "single entry" of reduced array 332 may include a single value, such as "72," in reduced array 332 used to replace a single quality score or group of quality scores.

[0082] For purposes of this specification, whether a group of quality scores exceeds a threshold number of quality scores can be determined by using a threshold number of quality scores and a positive or negative representation of the number of quality scores in the group. Thus, it is consistent with this specification to determine whether the number of quality scores in a group of quality scores "meets" a threshold, rather than simply whether the number of quality scores is greater than a threshold. This is because such a relationship may be described as a group of quality scores having four quality scores above a threshold of three quality scores, or a group of quality scores having negative four quality scores below a negative threshold of three quality scores. In either case, the group of quality scores has more than three quality scores, regardless of how the threshold is implemented.

[0083] For ease of explanation, this specification describes the process of a single mapping engine 320 before describing the process of the group mapping engine 326. However, parallel processing and other similar methods may be used so that the single mapping engine 320 does not require a complete single mapping process before the group mapping engine 326 completes the group mapping process. Instead, whether the reduced sequence generation engine 316 invokes the single mapping engine 320 or the group mapping engine 326 is determined based on the particular quality scores of the sequence of quality scores 302 processed by the reduced sequence generation engine 326.

[0084] In step E, the single mapping engine 320 obtains the quality score array 302. The single mapping engine 320 uses the quality score array 302 and the single mapping character list 322 to generate a predetermined single mapping 324, as shown in item 321. In the example of FIG. 3, the single mapping character list 322 includes integer values ​​0 through 63. The ASCII values ​​of the quality score array 302 include values ​​33 through 96. In this manner, each value in the quality score array 302 may be mapped to a specific value between 0 and 63. For example, the "@" character in the quality score array 302 corresponding to the ASCII value 64 may be mapped to the integer value 31. Similarly, the "A" character in the given quality score array corresponding to the ASCII value 65 may be mapped to the integer value 33, and so on.

[0085] As shown in the predetermined single mapping 324, the ASCII character "!", which corresponds to a value of 33, is mapped to a value of 0 within the range of 0 to 63. The ASCII character """, which corresponds to a value of 34, is mapped to a value of 1. Similarly, the ASCII character "_", which corresponds to a value of 95, is mapped to a value of 62, and the ASCII character "`", which corresponds to a value of 96, is mapped to a value of 63. Other mappings not shown in the predetermined single mapping 324 may also be generated by the single mapping engine 320.

[0086] In other implementations, other mappings can be used. For example, instead of 0 to 63, a smaller or larger range can be used in which values ​​in the quality score array 302 are mapped to values ​​between 33 and 96, such as ASCII values ​​between 33 and 96. Other mappings, such as those generated by the group mapping engine 326, can account for value ranges other than 33 to 96. In some implementations, the number of unique quality scores in the quality score array 302 is used to determine the range to which the quality scores in the quality score array 302 are mapped. For example, if the quality score array 302 includes 63 unique quality scores, the range to which the quality scores in the quality score array 302 are mapped can include 63 values. In some implementations, other ranges are used. For example, if the quality score array 302 includes a first number of unique quality scores, a range can be calculated, for example, by a single mapping engine 320 to include the first number of unique quality scores divided by 2, or other calculation based on the first number of unique quality scores. In some implementations, subsequent operations are applied to the processing results used to determine the mapping range. For example, the first number of intrinsic quality scores may be odd, the first operation applied to the first number of intrinsic quality scores may be dividing by the integer 2, and the second operation may include rounding up or down the corresponding processing result, depending on the implementation.

[0087] In step F, the group mapping engine 326 obtains the quality score group frequencies 314. The group mapping engine 326 uses the quality score group frequencies 314 or other data associated with the quality score array 302, along with a group mapping character list 328, to generate a predetermined group mapping 330, as shown in item 327. In the example of FIG. 3, the group mapping character list 328 includes integer values ​​64 through 245, which correspond to the 190 most frequently occurring groups. In some implementations, more or fewer groups may be mapped by the group mapping engine 326. For example, instead of generating a predetermined group mapping 330 including the 190 most frequently occurring groups, the group mapping engine 326 could generate a mapping including 200, 230, 185, or any other number of most frequently occurring groups. ASCII values ​​corresponding to the quality score group frequencies 314 include values ​​33 through 96. The group mapping engine 326 can determine a portion of the groups for mapping based on the quality score group frequencies 314. For example, the group mapping engine 326 may determine a certain number of most frequently occurring groups (e.g., 190 frequently occurring groups) and assign a value (e.g., an integer value between 64 and 254) to each group in the most frequently occurring groups.

[0088] As shown in predetermined group mappings 330, in the example of FIG. 3, the quality score group represented by the letters "ACD" is mapped to a value of 64. The quality score group represented by the letters "FFF" is mapped to a value of 72. The quality score group represented by the letters "HIJ" is mapped to a value of 73. Other mappings not shown in predetermined group mappings 330 may also be generated by group mapping engine 326.

[0089] In other implementations, other mappings can be used. For example, instead of 64 to 254, a smaller or larger range can be used to which the groups in quality score group frequencies 314 are mapped. For example, the groups in quality score group frequencies 314 can be mapped to values ​​from 0 to 255. Other mappings, such as mappings generated by a single mapping engine 320, can occupy values ​​other than those used for the 0 to 255 group mapping.

[0090] The reduced array generation engine 316 processes the quality score array 302 using the group mapping engine 326 and the single mapping engine 320 to generate a reduced array 332. The data derived from the quality score array 302 may include the quality score array 332 itself. Alternatively, the data derived from the quality score array 302 may include the data output by the frequency counter engine 310. The data output by the frequency counter engine 310 may include the quality score group frequencies 314. The reduced array 332 is a combination of values ​​from the predetermined single mapping 324 and the predetermined group mapping 330.

[0091] The reduced array generation engine 316 processes the quality score array 302, the quality score group frequencies, or both, and can determine whether to use a single mapping engine 320 or a group mapping engine 326 to generate entries in the reduced array 332. Occurrences of groups in the quality score array 302 that belong to a group in a given group mapping 330 are replaced with a value from the given group mapping 330. For example, an occurrence of an "A" quality score followed by a "C" quality score followed by a "D" quality score is replaced by the value 64 in the reduced array 332.

[0092] For a quality score that is not a member of one of the groups in the predetermined group mappings 330, the predetermined single mapping 324 is used. For example, the first value in the quality score array 302 is the character "@", which corresponds to the ASCII value 64. When the character "@" appears in the quality score array 302, it is not a member of a group in the predetermined group mappings 330. The character "@" is then mapped based on the predetermined single mapping 324. The mapping of "@" based on the predetermined single mapping 324 is the value 31. The reduced array generation engine 316 adds the value 31, which corresponds to the position of "@" in the reduced array 332.

[0093] In stage G, reduced sequence generation engine 316 generates reduced sequence 332 using data derived from predetermined group mapping 330, predetermined single mapping 324, and quality score sequence 302, as described above and shown in item 331. Reduced sequence generation engine 316 sends reduced sequence 332 to compression engine 334.

[0094] In stage H, the compression engine 334 performs a subsequent compression on the reduced array 332 corresponding to a predetermined compression method. For example, in some cases, the compression engine 334 performs the compression based on an implementation of Prediction by Partial Matching of a Range Encoder (PPMD) to compress the reduced array 332. In some implementations, other compression methods known in the art can be used instead of or in addition to a PPMD ​​implementation. In general, the compression engine 334 can use any compression or combination of compression methods, such as any compression method known in the art. The compression engine 334 generates an output 336 based on the reduced array 332 as input to the compression method used by the compression engine 334. The output 336 represents a compressed version of the quality score array 302.

[0095] In some implementations, each value of the reduced array 332 provided to the compression engine 334 is compressed according to a one-byte context. For example, for a given value of the reduced array 332, the previous value of the reduced array 332 may be used as the context to compress the given value of the reduced array 332. The one-byte context may be used to compress the reduced array 332 according to a PPMD ​​implementation of a range encoder or another compression or encoding method. A larger or smaller context may be used to balance speed, resulting compressed output, or other parameters. In some cases, the resulting compression ratio of the output 336 may be two or more bits per quality score. For example, four or more quality scores may be compressed into one byte or eight bits of memory space. In general, the output 336, similar to the output 128 of FIG. 1, may be used in any further processes or may be stored, depending on the implementation.

[0096] For ease of explanation, steps A through H are used. The process performed by system 300 may occur in the order shown by steps A through H, although in other implementations, the order of certain steps may differ. In some implementations, two or more steps may occur simultaneously.

[0097] 4 is a flow diagram illustrating an example of a process for compressing an array of quality scores having a second data format. Process 400 can be performed by one or more electronic systems, such as system 300 of FIG.

[0098] Process 400 includes obtaining a genetic read sequence generated by a genetic sequencing device (402) and obtaining a plurality of quality scores corresponding to the genetic read sequence (404). For example, as shown in FIG. 3, a quality score sequence 302 is generated by a sequencer based on a genetic data sequence and transmitted to a group identification engine 304. In the example of FIG. 3, the quality score sequence 302 is generated by a sequencer that uses the Q40 data format to encode base-related quality scores as described above. In general, any sequencer that uses more than a threshold number of unique quality scores can compress the resulting output data using the process performed by system 300 shown in FIG. 3 or a similar process. For example, if a sequencer uses more than eight unique quality scores to encode quality scores corresponding to a genetic sequence, the sequencer can use the encoding and compression process discussed with reference to FIG. 3.

[0099] 3, the group identification engine 304 retrieves the quality score array 302 and generates several quality score groups based on the quality scores in the quality score array 302. In some implementations, the groups of quality scores include quality scores that are adjacent to each other in the quality score array 302. For example, if the quality score array 302 includes quality scores "@C@DFFFF...", the groups determined by the group identification engine 304 may include the quality score "@C@" in an implementation in which a quality score of 3 is used to generate the groups.

[0100] In some implementations, the group identification engine 304 sends one or more groups to the frequency counter engine 310, which determines the frequency of occurrence of each quality score group. For example, the frequency counter engine 310 can determine, for each group determined by the group identification engine 304, how many occurrences of the group are present in the quality score array 302. In the example of FIG. 3, the quality scores in the quality score array 302 are grouped into groups of three. However, in other implementations, other numbers of quality scores can be used. For example, the group identification engine 304 can determine groups of four quality scores, two quality scores, or any other number of quality scores to generate the quality score groups.

[0101] The process 400 includes, for each quality score of the plurality of quality scores, determining 408 whether the quality score is a member of a particular quality score group having an occurrence frequency that meets a predetermined threshold. For example, as shown in FIG. 3 , the reduced sequence generation engine 316, the single mapping engine 320, and the group mapping engine 326 generate a reduced sequence 332 based on a predetermined single mapping 324 and a predetermined group mapping 330. For each quality score in the reduced sequence 332, the reduced sequence generation engine 316 determines whether the quality score is a member of a group included in the predetermined group mapping 330. If the quality score is a member of a group included in the predetermined group mapping 330, the quality score is coded as a single entry in the reduced sequence 332 along with other quality scores in the group.

[0102] In some implementations, each group in a predetermined group mapping 330 satisfies a predetermined threshold. For example, each group in a predetermined group mapping 330 may occur within the quality score array 302 a predetermined number of times included in the group mapping. Each group in a predetermined group mapping 330 may belong to a predetermined number of groups that occur more frequently than other groups in the quality score array 302. In the example of FIG. 3 , the predetermined group mapping 330 includes the 190 most frequently occurring quality score groups. A threshold may then be defined as the number of occurrences corresponding to the 191st most frequently occurring quality score group, the 190th most frequently occurring quality score group, etc., depending on the implementation, among other things. All 190 most frequently occurring quality score groups in a predetermined group mapping 330 then satisfy this threshold.

[0103] In some implementations, a predetermined group mapping 330 can be generated using different amounts of quality score groups. For example, the number of groups in the predetermined group mapping 330 can be determined based on the number of quality scores used by the sequencer. The mapping can then be generated to satisfy an optimization process. For example, the optimization process can include maximizing the number of groups coded as a single value and minimizing the number of quality scores that do not belong to a quality score group in the predetermined group mapping 330 and are coded as a single value. Parameters to optimize can include the number of groups included in a group mapping, such as the predetermined group mapping 330 of FIG. 3, as well as the number of quality scores used to generate one or more groups within the predetermined group mapping 330. The number of quality scores in a single mapping, such as the predetermined single mapping 324 of FIG. 3, can correspond to the number of unique quality scores used by the sequencer to represent quality scores corresponding to genetic sequence reads.

[0104] In some implementations, the quality score groupings are based on adjacent quality scores that appear together in the quality score array. For example, a quality score array 302 represented as "@C@DFFFF..." includes the group of adjacent quality scores "FFF." The group of adjacent quality scores "FFF" is included in a predetermined group mapping 330. According to the predetermined group mapping 330, the group of adjacent quality scores "FFF" should be coded by the reduced array generation engine 316 as a single value 72. Of course, other implementations can use other values ​​for mapping or coding purposes.

[0105] In some implementations, scores that are not members of a quality score group are mapped as a single entry in the reduced array. For example, in an implementation that uses three quality scores to generate groups of quality scores, the first value "@" in quality score array 302, represented as "@C@DFFFF...", is not a member of a particular quality score group. Therefore, the first value "@" is coded as a single value 31 in reduced array 332 according to predetermined single mapping 324. In the example of FIG. 3, predetermined single mapping 324 is a one-to-one match of the quality score array 302 and the values ​​used to represent a new range of quality scores of the same length, such that any quality score that is not part of a group in predetermined group mapping 330 is coded in reduced array 332 as the original value plus or minus a shifted value used to generate predetermined single mapping 324. For example, in FIG. 3, the single value corresponding to a quality score that is not a member of a quality score group is shifted by a value of 33. Shifting can be used to create dedicated continuous ranges of values ​​for either group mappings or single mappings.

[0106] The process 400 includes generating (410) a single entry for inclusion in a reduced array as a representation of the particular quality score group based on a determination that the quality score is a member of a particular quality score group. For example, the reduced array generation engine 316 may determine that the quality score array 302, represented as "@C@DFFFF...," includes the quality score group "FFF." The quality score group "FFF" is included in a predetermined group mapping 330. According to the predetermined group mapping 330, the adjacent quality score group "FFF" should be coded by the reduced array generation engine 316 as a single value 72. However, the present disclosure is not limited to replacing a group of quality scores with a single entry or value. In other implementations, a single quality score can be replaced with a single entry or value using a predetermined single mapping. Such a single quality score replacement may be beneficial because the single entry or value may be selected to represent a single quality score having a smaller bit size than the single quality score.

[0107] The process 400 includes generating 412 a reduced array by aggregating each of the generated entries. For example, the reduced array generation engine 316 may aggregate a first coded value of 31 corresponding to the quality score "@," a second coded value of 34 corresponding to the quality score "C," a third coded value of 31 corresponding to the quality score "@," a further coded value of 35 corresponding to the quality score "D," and a fifth coded value of 72 corresponding to the quality score group "FFF" in the quality score array 302, represented as "@C@FFFF...." The reduced array generation engine 316 may generate subsequent coded values ​​based on the subsequent values ​​of the quality score array 302. The reduced array generation engine 316 may continue until all values ​​of the quality score array 302 are represented in the reduced array 332.

[0108] In some implementations, further compression steps are performed based on the reduced array 332. For example, as shown in FIG. 3, the reduced array 332 may be sent to a compression engine 334. The compression engine 334 may then perform one or more compression processes on the reduced array 332 to generate an output 336. Similar to the process shown in FIG. 1, a first encoding represented by the reduced array 332 generated before a subsequent compression step performed by the compression engine 334 may be advantageous to improve the resulting compression of the output 336. The form of the reduced array 332 may be such that the compression engine 334 can compress data more quickly or effectively than the original quality score array 302. For example, the reduced array 332 may be a compressed version of the quality score array 302, and the duration or quality of the compression performed by the compression engine 334 may depend on the size of the input. In this manner, the initial encoding step to generate the reduced array 332 may reduce the time it takes to compress and increase the quality of compression achieved by the compression engine 334.

[0109] 5 is a flow diagram illustrating one embodiment of a process 500 for recovering an array of quality scores having a first data format. Process 500 may be performed by one or more electronic systems, such as system 100 of FIG. 1.

[0110] The process 500 includes obtaining a first coded data set generated by encoding each of a plurality of quality scores using an x-ary number, where x is an integer number representing the number of different quality scores used by the genetic sequencing device (502). For example, the decoding engine can obtain the binary output 114 or the binary output 126.

[0111] Process 500 includes generating a first decoded data set (504) using the x-ary number. For example, similar to the processes shown in items 108 and 120, the first coded data set may be decoded based on the x-ary number used for coding, where x is an integer corresponding to the number of unique quality scores present in the plurality of quality scores. In some implementations, the integer value of the binary representation of the first decoded data may be repeatedly divided by the x-ary number to generate the x-ary number.

[0112] The process 500 includes ordering (506) the first decoded data set within one or more other decoded data sets. In some implementations, an ordering engine may obtain the decoded data set and one or more other decoded data sets and order the first decoded data set based on the first coded data set. For example, the ordering engine may determine a portion of the first coded data set, which is then decoded to generate the first decoded data set. The ordering engine may also determine a portion of the first coded data set that is decoded to generate one or more other decoded data sets. Based on the original location of the first decoded data set and the one or more other decoded data sets within the first coded data set, the ordering engine may order the first decoded data set within the one or more other decoded data sets.

[0113] The process 500 includes generating (508) an aggregate decoded data set based on the first decoded data set and one or more other decoded data sets. For example, based on an ordering of the first decoded data set within the one or more other decoded data sets, the aggregate decoded data set engine can generate an aggregate decoded data set that includes the first decoded data set and one or more other decoded data sets. The aggregate decoded data set can include data similar to the data used to generate the first coded data set. The aggregate decoded data set can be used in other processes or operations depending on the implementation.

[0114] 6 is a flow diagram illustrating an embodiment of a process 600 for recovering an array of quality scores having a second data format. Process 600 may be performed by one or more electronic systems, such as system 300 of FIG.

[0115] Process 600 includes obtaining 602 a reduced sequence generated based on the single mapping database and the group mapping database. For example, the decoding engine may obtain reduced sequence 332 or other related data from system 300 or another system. The process shown corresponding to reduced sequence generation engine 316 may then be performed in reverse to generate quality score sequence 302.

[0116] The process 600 includes generating 604 a first decoded data set based on the reduced sequence and the single mapping database and the group mapping database. As described above, the decoding engine may be similar to the reduced sequence generation engine 316 but may perform operations in reverse order to generate the quality score sequence 302. For example, the decoding engine may obtain the reduced sequence 332 and decode the reduced sequence 332 using the predetermined group mapping 330 and the predetermined single mapping 324 to generate a first decoded data set corresponding to at least a portion of the quality score sequence 302.

[0117] The process 600 includes ordering (606) the first decoded data set within one or more other decoded data sets. For example, the decoding engine may decode a first portion of the reduced array 332 and determine a first order corresponding to the first portion of the reduced array 332. In some implementations, the order of the reduced array 332 may be used to determine the order of the first decoded data set and one or more other decoded data sets. For example, the first decoded data set may correspond to the first portion of the reduced array 332. As a result, the first decoded data set may be ordered at the start of the final aggregated decoded data set. Then, one or more subsequent other decoded data sets may be ordered based on the order of the data corresponding to the reduced array 332 used to generate the one or more other decoded data sets.

[0118] Process 600 includes generating 608 an aggregate decoded data set based on the first decoded data set and one or more other decoded data sets. For example, an aggregate decoded data set engine can be used to generate the aggregate decoded data set based on ordering the first decoded data set within one or more other decoded data sets as described above. The aggregate decoded data set can include data similar to the data used to generate a reduced array, such as reduced array 332. The aggregate decoded data set can be used in other processes or operations depending on the implementation.

[0119] 7 is a flow diagram illustrating an embodiment of a process 700 for determining how to compress quality scores. Process 700 may be performed by one or more electronic systems, such as system 300 of FIG. 3 or system 100 of FIG. 1.

[0120] Process 700 includes obtaining genetic data from a genetic sequencer (702). Various forms of genetic sequencers are known to those skilled in the art. For example, a decision engine can obtain genetic data corresponding to one or more quality scores generated by a given genetic sequencer.

[0121] Process 700 includes determining (704) the number of unique quality scores in the genetic data. For example, depending on the type, model, or particular software of a given sequencer, the quality scores of a base call may be represented by one or more symbols or values. The number of unique symbols or values ​​used to represent the one or more quality scores can then be used to determine which compression method to use for the quality scores of the genetic data.

[0122] Process 700 includes a first decision path corresponding to determining that there are unique quality scores of 8 or less in the genetic data obtained from the genetic sequencer, and a second decision path corresponding to determining that there are unique quality scores of more than 8 in the genetic data obtained from the genetic sequencer. If there are unique quality scores of 8 or less in the genetic data, a process corresponding to process 200 described in Figure 2 can be performed to compress the quality scores of the genetic data. If there are unique quality scores of more than 8 in the genetic data, a process corresponding to process 400 described in Figure 4 can be performed to compress the quality scores of the genetic data.

[0123] In some implementations, other thresholds are used to determine which compression method to use. For example, instead of a unique quality score of 8, the system may determine that a unique quality score of 7, 9, or 10 is required to perform a process corresponding to process 400 described in Figure 4. In general, the system can use any suitable threshold or decision modeling to determine which of multiple different compression methods to use to compress a given set of data, such as a given set of quality scores.

[0124] Genomic data referred to by this disclosure (e.g., input data 102) can include, for example, but not limited to, nucleotide sequences, deoxyribonucleic acid (DNA) sequences, ribonucleic acid (RNA) sequences, and amino acid sequences. While the description herein is fairly specific regarding genomic information in the form of nucleotide sequences, it will be understood that the ordered data arrangements of this specification can be implemented with other genomic data, with some variations, as will be appreciated by those skilled in the art.

[0125] FIG. 8 is a graphical representation of experimental results of a process for encoding (e.g., compressing) an array of quality scores having a first data format. FIG. 8 shows results from compressing dataset “SRR6882909_1.fastq.” Dataset “SRR6882909_1.fastq” is formatted using the Q4 format described above. The quality scores for dataset “SRR6882909_1.fastq” include four unique quality scores. Chart 802 shows the relative size of the raw data corresponding to dataset “SRR6882909_1.fastq,” as well as compressed versions of dataset “SRR6882909_1.fastq” after compression using various techniques, including gzip level 9 compression, zstd level 11 compression, and Lena Q4 compression. Lena Q4 compression corresponds to process 100 shown in FIG. 1 and methods described herein.

[0126] Chart 802 shows that the raw data for dataset "SRR6882909_1.fastq" is 7402335856 bytes, the compressed version of dataset "SRR6882909_1.fastq" using gzip level 9 compression is 417512395 bytes, the compressed version of dataset "SRR6882909_1.fastq" using zstd level 11 compression is 452733689 bytes, and the compressed version of dataset "SRR6882909_1.fastq" using Lena Q4 compression is 259865991 bytes. The compression achieved by Lena Q4 compression of dataset "SRR6882909_1.fastq" is greater than the compression achieved by other alternative compression methods. Legend 806 indicates which bars correspond to which compression method and which bars correspond to the raw data of the dataset "SRR6882909_1.fastq."

[0127] Chart 804 shows the compression time for each of the compression methods used on the dataset "SRR6882909_1.fastq." Chart 804 shows that it took 1980 seconds to generate a compressed version of the dataset "SRR6882909_1.fastq" using gzip level 9 compression, 108 seconds to generate a compressed version of the dataset "SRR6882909_1.fastq" using zstd level 11 compression, and 36 seconds to generate a compressed version of the dataset "SRR6882909_1.fastq" using Lena Q4 compression. The time to compress the dataset "SRR6882909_1.fastq" using Lena Q4 compression is less than the compression time achieved by other alternative compression methods.

[0128] FIG. 9 is a graphical representation of experimental results of a process for encoding (e.g., compressing) an array of quality scores having a second data format. FIG. 9 shows results from compression of dataset "ERR1744700_1.fastq." Dataset "ERR1744700_1.fastq" is formatted using the Q40 format described above. The quality scores for dataset "ERR1744700_1.fastq" include unique quality scores greater than 4. Chart 902 shows the relative size of the raw data corresponding to dataset "ERR1744700_1.fastq," as well as compressed versions of dataset "ERR1744700_1.fastq" after compression using various techniques, including gzip level 9 compression, zstd level 11 compression, and Lena Q40 compression. Lena Q40 compression corresponds to process 300 shown in FIG. 3.

[0129] Chart 902 shows that the raw data for dataset "ERR1744700_1.fastq" is 5033592178 bytes, the compressed version of dataset "ERR1744700_1.fastq" using gzip level 9 compression is 1289564690 bytes, the compressed version of dataset "ERR1744700_1.fastq" using zstd level 11 compression is 1290828665 bytes, and the compressed version of dataset "ERR1744700_1.fastq" using Lena Q40 compression is 1228518456 bytes. The compression achieved by Lena Q40 compression of dataset "ERR1744700_1.fastq" is greater than the compression achieved by other alternative compression methods. Legend 906 indicates which bar corresponds to which compression method and which bar corresponds to the raw data of the dataset "ERR1744700_1.fastq".

[0130] Chart 904 shows the compression time for each of the compression methods used on the dataset "ERR1744700_1.fastq." Chart 904 shows that it took 3762 seconds to generate a compressed version of the dataset "ERR1744700_1.fastq" using gzip level 9 compression, 306 seconds to generate a compressed version of the dataset "ERR1744700_1.fastq" using zstd level 11 compression, and 101 seconds to generate a compressed version of the dataset "ERR1744700_1.fastq" using Lena Q40 compression. The time to compress the dataset "ERR1744700_1.fastq" using Lena Q40 compression is less than the compression time achieved by other alternative compression methods.

[0131] FIG. 10 is a diagram of components of a computer system 1000 that can be used to implement a system for generating medical analyses using joint models based on multivariate ordinal data.

[0132] Computing device 1000 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device 1000 or 1050 may include a Universal Serial Bus (USB) flash drive. A USB flash drive may store an operating system and other applications. A USB flash drive may include input / output components, such as a wireless transmitter or a USB connector, that can be inserted into a USB port of another computing device. The components, their connections and relationships, and their functions illustrated herein are intended to be examples only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0133] Computing device 1000 includes a processor 1002, memory 1004, a storage device 1008, a high-speed interface 1008 connecting to memory 1004 and a high-speed expansion port 1010, and a low-speed interface 1012 connecting to a low-speed bus 1014 and storage device 1008. Each of components 1002, 1004, 1008, 1008, 1010, and 1012 are interconnected using various buses and may be mounted on a common motherboard or in any other suitable manner. Processor 1002 processes instructions for execution within computing device 1000, including instructions stored in memory 1004 or on storage device 1008, and can display graphical information for a GUI on an external input / output device, such as a display 1016 coupled to high-speed interface 1008. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Additionally, multiple computing devices 1000 can be connected together, each providing a portion of the required computation, for example as a server bank, a cluster of blade servers, or a multi-processor system.

[0134] The memory 1004 stores information within the computing device 1000. In one implementation, the memory 1004 is a volatile memory unit or units. In another implementation, the memory 1004 is a non-volatile memory unit or units. The memory 1004 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0135] The storage device 1008 can provide mass storage for the computing device 1000. In one implementation, the storage device 1008 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. An information carrier is a computer-readable or machine-readable medium, such as the memory 1004, the storage device 1008, or memory on the processor 1002.

[0136] The high-speed controller 1008 manages bandwidth-intensive operations for the computing device 1000, while the low-speed controller 1012 manages low-bandwidth-intensive operations. This allocation of functionality is merely one example. In one implementation, the high-speed controller 1008 is coupled to the memory 1004, to the display 1016, for example, via a graphics processor or accelerator, and to a high-speed expansion port 1010 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 1012 is coupled to the storage device 1008 and the low-speed expansion port 1014. The low-speed expansion port, which can include various communication ports, for example, USB, Bluetooth, Ethernet, wireless Ethernet, can be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, a pointing device, a microphone / speaker pair, a scanner, or a networking device such as a switch or router. The computing device 1000, as shown in the figure, can be implemented in several different forms. For example, the computing device can be implemented as a standard server 1020, or multiple times in a cluster of such servers. The computing device can also be implemented as part of a rack server system 1024. Additionally, the computing device can be implemented in a personal computer, such as a laptop computer 1022. Alternatively, components from the computing device 1000 can be combined with other components in a mobile device (not shown), such as device 1050. Each such device can contain one or more of the computing devices 1000, 1050, and the entire system can be made up of multiple computing devices 1000, 1050 in communication with each other.

[0137] Computing device 1000, as shown in the figure, can be implemented in several different forms. For example, the computing device can be implemented as a standard server 1020, or multiple times in a cluster of such servers. The computing device can also be implemented as part of a rack server system 1024. Additionally, a computing device can be implemented in a personal computer, such as a laptop computer 1022. Alternatively, components from computing device 1000 can be combined with other components in a mobile device (not shown), such as device 1050. Each such device can contain one or more of computing devices 1000, 1050, and the entire system can be made up of multiple computing devices 1000, 1050 in communication with each other.

[0138] Computing device 1050 includes, among other components, a processor 1052, memory 1064, and input / output devices such as a display 1054, a communications interface 1066, and a transceiver 1068. Device 1050 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068 are interconnected using various buses, and some of the components may be implemented on a common motherboard or in other manners as appropriate.

[0139] The processor 1052 can execute instructions within the computing device 1050, including instructions stored in the memory 1064. The processor can be implemented as a chipset of chips including separate and multiple analog and digital processors. In addition, the processor can be implemented using any of several architectures. For example, the processor 1010 can be a Complex Instruction Set Computer (CISC) processor, a Reduced Instruction Set Computer (RISC) processor, or a Minimal Instruction Set Computer (MISC) processor. The processor can provide coordination of other components of the device 1050, such as control of a user interface, applications run by the device 1050, and wireless communication by the device 1050.

[0140] The processor 1052 can communicate with a user via a control interface 1058 and a display interface 1056 coupled to a display 1054. The display 1054 can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display, an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 1056 can include appropriate circuitry for driving the display 1054 to present graphical and other information to the user. The control interface 1058 can receive commands from the user and translate the commands for input to the processor 1052. Additionally, an external interface 1062 can be provided for communication with the processor 1052 to enable near-field wireless communication between the device 1050 and other devices. The external interface 1062 can be provided, for example, for wired communication in some implementations or for wireless communication in other implementations; multiple interfaces can also be used.

[0141] Memory 1064 stores information within computing device 1050. Memory 1064 may be implemented as one or more of a computer-readable medium(s), a volatile memory unit(s), or a non-volatile memory unit(s). Expansion memory 1074 may also be provided and connected to device 1050 via expansion interface 1072, which may include, for example, a Single In-Line Memory Module (SIMM) card interface. Such expansion memory 1074 may provide additional storage space for device 1050 or may store applications or other information for device 1050. In particular, expansion memory 1074 may include instructions that perform or complement the processes described above and may also include secure information. Thus, for example, expansion memory 1074 may be provided as a security module for device 1050 and may be programmed with instructions that enable secure use of device 1050. Additionally, secure applications can be provided via the SIMM card with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0142] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is, for example, a computer-readable or machine-readable medium, such as memory 1064, expansion memory 1074, or memory on processor 1052, which may be received via transceiver 1068 or external interface 1062.

[0143] Device 1050 can communicate wirelessly via communication interface 1066, which can include digital signal processing circuitry if necessary. Communication interface 1066 can provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, via radio frequency transceiver 1068. Additionally, short-range communication can occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a Global Positioning System (GPS) receiver module 1070 can provide additional navigation-related and location-related wireless data to device 1050, which can be used as appropriate by applications running on device 1050.

[0144] Device 1050 can also communicate audibly using audio codec 1060, which can receive speech information from a user and convert this speech information into usable digital information. Audio codec 1060 can also generate audible sounds for the user, such as through a speaker in the handset of device 1050. Such sounds can include sounds from a voice telephone call, recorded sounds such as voice messages, music files, etc., and can also include sounds generated by applications running on device 1050.

[0145] The computing device 1050, as shown, can be implemented in several different forms. For example, the computing device can be implemented as a mobile phone 1080. The computing device can also be implemented as part of a smartphone 1082, personal digital assistant, or other similar mobile device.

[0146] Various implementations of the systems and methods described herein can be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations of such implementations. These various implementations can be special-purpose or general-purpose, and can include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0147] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device, such as a magnetic disk, optical disk, memory, programmable logic device (PLD), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0148] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input.

[0149] The systems and techniques described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or in a computing system that includes middleware components, e.g., an application server, or in a computing system that includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein, or in any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0150] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0151] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the present invention. Additionally, the logic flow depicted in the figures does not require the particular order shown, or sequential order, to achieve desired results. Additionally, other steps can be provided or steps can be eliminated from the described flow, and other components can be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0152] All of the embodiments and functional operations of the invention described herein can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" encompasses all apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus can include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiving apparatus.

[0153] A computer program (also known as a program, software, software application, script, or code) can be written in any type of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple associated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to run on one computer or on multiple computers, located at a single location or distributed across multiple locations and interconnected by a communications network.

[0154] The processes, operations, and / or logic flows described herein may be implemented by one or more central processing units (CPUs) or graphics processing units (GPUs) executing one or more computer software instructions to implement the functions of the processes, operations, and / or logic flows described herein. The processes, operations, and / or logic flows may also be implemented in hardware circuitry. For example, in some implementations, the operations of the present disclosure may be performed by a processing engine implemented using logic gates of a field programmable gate array (FPGA) programmably configured to implement the functions of the processes, operations, and / or logic flows described herein. As another example, the operations of the present disclosure may be performed by a processing engine implemented using logic gates of an application-specific integrated circuit (ASIC) configured to implement the functions of the processes, operations, and / or logic flows described herein. In still other implementations, some of the processes, operations, and / or portions of the logic flows may be implemented by one or more CPUs or one or more GPUs, and some of the processes, operations, and / or portions of the logic flows may be implemented by hardware circuitry and in any order.

[0155] Processors suitable for executing a computer program include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes or is operatively coupled to one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, for receiving data from, transmitting data to, or both. However, a computer need not include such devices. Furthermore, a computer can be incorporated into another device, e.g., a tablet computer, a mobile phone, a personal digital assistant (PDA), a portable audio player, or a global positioning system (GPS) receiver, to name a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0156] To provide for interaction with a user, embodiments of the present invention can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input.

[0157] Embodiments of the present invention can be implemented in a computing system including back-end components, e.g., as a data server, or middleware components, e.g., an application server, or front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the present invention, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0158] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0159] While this specification contains many details, these should not be construed as limiting the scope of the invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may relate to a subcombination or a variation of a subcombination.

[0160] Similarly, while operations are depicted in the figures in a particular order, it should not be understood that such operations need to be performed in the particular order shown, or sequentially, or that all of the illustrated operations need to be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may typically be integrated together in a single software product or packaged within multiple software products.

[0161] In each instance where a particular file format is mentioned, other file types or formats may be substituted. For example, an HTML file may be replaced by an XML, JSON, plain text, or other type of file. Additionally, where a table or hash table is mentioned, other data structures (such as a spreadsheet, relational database, or structured file) may be used.

[0162] While specific embodiments of the present invention have been described, other embodiments are within the scope of the following claims. For example, the steps recited in the claims can be performed in a different order and still achieve desirable results.

[0163] Other embodiments While specific embodiments of the present invention have been described, other embodiments are within the scope of the following claims. For example, the steps recited in the claims can be performed in a different order and still achieve desirable results. [Explanation of symbols]

[0164] 100, 300 systems 1000 computing devices

Claims

1. 1. A system for compressing nucleic acid sequence data, the system comprising: one or more computers; and one or more storage devices having stored thereon instructions operable, when executed by the one or more computers, to cause the one or more computers to perform operations, the operations comprising: (i) a sequence read comprising data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) acquiring nucleic acid sequence data representing a plurality of quality scores, each quality score of the plurality of quality scores indicating the likelihood that a particular base call in the sequence read was correctly generated by a nucleic acid sequencing device; determining whether the read sequence includes at least one base at the location of the read sequence that was not determined by the nucleic acid sequencing device; selecting an encoding process based on whether the read sequence includes at least one base that was not determined by the nucleic acid sequencing device at the location of the read sequence; generating a first encoded data set by encoding the sequence reads using the selected encoding process; encoding the first encoded data set using a second encoding process.

2. The operation generating a third coded data set by encoding the sequence reads using an (x-1)-ary number using a third coding process based on a determination that the sequence reads do not include at least one base not determined by the nucleic acid sequencing device at the location of the sequence reads, where x is an integer representing the number of different quality scores used by the nucleic acid sequencing device; and encoding the third coded data set using a fourth coding process.

3. The system of claim 1 , wherein x is equal to 3.

4. 4. The system of claim 3, wherein the selected encoding process includes encoding each set of five quality scores of the plurality of quality scores for the read sequence into a single byte by representing each quality score in the set of five quality scores as a base 3 number.

5. The operation generating a second encoded data set by encoding the sequence read into a single byte of memory using a third encoding process based on a determination that the sequence read includes at least one base not determined by the nucleic acid sequencing device at the location of the sequence read; and encoding the second coded data using a fourth coding process.

6. The system of claim 5 , wherein the second encoding process and the fourth encoding process are the same.

7. The system of claim 1 , wherein the acquired data comprises a FASTQ file.

8. The system of claim 1 , wherein the first coded data set is a condensed version of the plurality of quality scores.

9. The system of claim 1 , wherein the second encoding process is a compression process.

10. The system of claim 9 , wherein the compression process includes an implementation of a partial matching prediction by range encoder (PPMD).

11. 11. The system of claim 10, wherein for a given value of the first coded data set, the given value is compressed according to a 4-bit context relative to the position of the given value within the first coded data set.

12. The system of claim 1, wherein selecting the encoding process includes selecting one of two different encoding processes.

13. The system described in claim 1, wherein at least one base not determined by the nucleic acid sequencing device at the location of the read sequence is represented by an "N" base within the read sequence.

14. A method for compressing nucleic acid sequence data, comprising: (i) a sequence read comprising data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) acquiring nucleic acid sequence data representing a plurality of quality scores, each quality score of the plurality of quality scores indicating the likelihood that a particular base call in the sequence read was correctly generated by a nucleic acid sequencing device; determining whether the read sequence includes at least one base at the location of the read sequence that was not determined by the nucleic acid sequencing device; selecting an encoding process based on whether the read sequence includes at least one base that was not determined by the nucleic acid sequencing device at the location of the read sequence; generating a first encoded data set by encoding the sequence reads using the selected encoding process; encoding the first encoded data set using a second encoding process.

15. Based on a determination that the read sequence does not contain at least one base not determined by the nucleic acid sequencing device at the location of the read sequence, generating a third coded data set by coding the read sequence using an (x-1)-ary number using a third coding process, where x is an integer representing the number of different quality scores used by the nucleic acid sequencing device; and encoding the third coded data set using a fourth coding process.

16. The method of claim 14, wherein x is equal to 3.

17. The method of claim 16, wherein the selected encoding process includes encoding each set of five quality scores of the plurality of quality scores for the read sequence into a single byte by representing each quality score in the set of five quality scores as a ternary number.

18. Based on a determination that the read sequence includes at least one base not determined by the nucleic acid sequencing device at the location of the read sequence, generating a second encoded data set by encoding the read sequence into a single byte of memory using a third encoding process; and encoding the second coded data using a fourth coding process.

19. The method described in claim 14, wherein at least one base not determined by the nucleic acid sequencing device at the location of the read sequence is represented by an "N" base within the read sequence.

20. One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including: (i) a sequence read comprising data corresponding to a plurality of base calls generated by a nucleic acid sequencing device; and (ii) acquiring nucleic acid sequence data representing a plurality of quality scores, each quality score of the plurality of quality scores indicating the likelihood that a particular base call in the sequence read was correctly generated by a nucleic acid sequencing device; determining whether the read sequence includes at least one base at the location of the read sequence that was not determined by the nucleic acid sequencing device; selecting an encoding process based on whether the read sequence includes at least one base that was not determined by the nucleic acid sequencing device at the location of the read sequence; generating a first encoded data set by encoding the sequence reads using the selected encoding process; and encoding the first coded data set using a second coding process.

21. The operation generating a third coded data set by encoding the sequence reads using an (x-1)-ary number using a third coding process based on a determination that the sequence reads do not include at least one base not determined by the nucleic acid sequencing device at the location of the sequence reads, where x is an integer representing the number of different quality scores used by the nucleic acid sequencing device; and encoding the third coded data set using a fourth coding process.

22. 21. The computer storage medium of claim 20, wherein x is equal to 3.

23. 23. The computer storage medium of claim 22, wherein the selected encoding process comprises encoding each set of five quality scores of the plurality of quality scores for the read sequence into a single byte by representing each quality score in the set of five quality scores as a base 3 number.

24. The operation is generating a second encoded data set by encoding the sequence read into a single byte of memory using a third encoding process based on a determination that the sequence read includes at least one base not determined by the nucleic acid sequencing device at the location of the sequence read; and encoding the second coded data using a fourth coding process.

25. 25. The computer storage medium of claim 24, wherein the second encoding process and the fourth encoding process are the same.

26. The computer storage medium of claim 20, wherein at least one base not determined by the nucleic acid sequencing device at the location of the read sequence is represented by an "N" base within the read sequence.

Images