FastQ / FastA compression system and method

Abstract

The present invention discloses a system and method for analyzing and massively compressing FastQ and / or FastA datasets. The methodology includes algorithms for compressing sequences, quality scores and identifiers in read files by reducing the dimensionality and redundancy of genomic data in a unique and optimal manner and in a binary format. The methodology also includes a decoding protocol for loss-free decompression of the compressed data.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 280,721, filed November 18, 2021, and U.S. Provisional Application No. 63 / 409,993, filed September 26, 2022, the entire contents of both of which are incorporated by reference into this application. [Background technology]

[0002] With recent advances in next-generation sequencing (NGS) technologies, large amounts of genomic data are being generated at high speed and low cost. It is estimated that 2-40 exabytes of data will be generated within the next decade. However, storage technologies are advancing at a much slower rate, posing technical and economic challenges to NGS data storage. Additionally, the transmission of these massive datasets is also very costly and time-consuming, delaying and limiting the use of these datasets.

[0003] General-purpose compression methods (e.g., Gzip, Bzip2) have a poor track record in compressing genomic data, while existing domain-specific compression methods (e.g., SPRING, LFastqC, DSRC2, LFQC, SeqSqueeze1, Quip, FQZComp) have inherent drawbacks, such as low compression ratios, low computational efficiency, lossy compression, dependency on quality score-based sequence encoding, ignoring irregular non-N bases, and stringent protocols.

[0004] Therefore, the demand for storing and transmitting vast amounts of genomic data and the inadequacies of existing compression methods call for efficient, domain-specific methods for NGS data compression. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made to solve the problems in the conventional techniques described above. [Means for solving the problem]

[0006] Disclosed embodiments provide a system for compressing genomic data in a more efficient and lossless manner. In certain embodiments, the system receives a data file comprised of sequence bases, quality scores, and identifiers, and applies an optimization algorithm to the quality scores, sequence k-mers from the sequence bases, and the identifiers. The system performs dimensionality reduction on the sequence bases, maps and ranks the quality scores, and stores templates of the identifiers that are consistent across the data files. The system then binary encodes and compresses the optimized data in the file in a lossless format.

[0007] In certain embodiments, the identifier is composed of sequencing run data and cluster data.

[0008] In another embodiment, the quality score comprises a sequence of quality values ​​for the base sequence.

[0009] In yet other embodiments, the sequence bases are made up of regular and irregular bases.

[0010] In another embodiment, the data file is a FastQ file or a FastA file.

[0011] In certain other embodiments, the optimization algorithm determines optimal hyperparameter values ​​for each of the quality scores, sequence k-mers from the bases, and the identifiers. [Brief description of the drawings]

[0012] The present invention and many of the attendant advantages will become better understood and more completely appreciated by reference to the following detailed description, when taken in conjunction with the accompanying drawings, the description of which is set forth below. [Figure 1] FIG. 2 is a diagram of an exemplary embodiment of the hardware of the system of the present invention. [Diagram 2]1 is a flowchart of a process for compressing genomic data according to an exemplary embodiment of the present invention. [Diagram 3] 4 is a series of tables illustrating an example of duplicate removal performed by the system of the present invention. [Figure 4] 1 is a series of tables showing examples of k-mer removal performed by the system of the present invention. [Diagram 5] 1 is a table showing an example of companion selection performed as part of the k-mer companion method. [Figure 6] 13 is a table showing an example of a final k-mer dictionary. [Figure 7] 1 is a series of tables showing examples of how semi-duplicate elimination is performed on similar data reads by the system of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In describing the preferred embodiments of the invention shown in the drawings, specific terminology is used for the sake of clarity. However, it is not intended that the invention be limited to the specific terminology so selected, and it is understood that each specific term includes all technical equivalents that operate in a similar manner to accomplish a similar purpose. Some preferred embodiments of the invention have been described for purposes of illustration, and it is understood that the invention may be embodied in other forms not specifically shown in the drawings.

[0014] Next generation sequencing (NGS) analysis utilizes bioinformatics, including signal-to-data translation, annotation or cataloging information, and actionable knowledge, to convert machine signals into meaningful information. The basic next generation sequencing process involves fragmenting DNA / RNA into multiple fragments, adding adapters, sequencing the library, and reassembling the library to form the genome sequence. Millions or billions of DNA strands can be sequenced in parallel, greatly increasing throughput and minimizing the need for fragment cloning methods. NGS can be used to rapidly sequence the entire human genome in a short period of time. See Behjati S, Tarpey PS. What is next generation sequencing?. Arch Dis Child Educ Pract Ed. 2013;98(6):236-238. doi:10.1136 / archdischild-2013-304340.

[0015] NGS generates large amounts of genomic data, so current technology is designed to compress the sequences, quality scores, and identifiers of read files in novel ways that allow for more efficient storage of that data, while also providing decoding protocols to decompress the compressed data with virtually no loss.

[0016] FIG. 1 is an exemplary embodiment of the system of the present invention. In the exemplary system 100, one or more peripherals 110 are connected to one or more computers 120 via a network 130. The peripherals / locations 110 include smartphones, tablets, wearable devices, and any other electronic devices known in the art that collect and transmit data over a network. The network 130 can be a wide area network such as the Internet, or a local area network such as an intranet. Due to the network 130, the physical locations of the peripherals 110 and the computers 120 do not affect the functionality of the hardware and software of the present invention. Both implementations are described herein, and unless otherwise noted, it is intended that the peripherals 110 and the computers 120 can be in the same or different physical locations. Communication between the hardware of the system can be accomplished in a number of known ways, for example, using network connection components such as modems or Ethernet adapters. Both the peripherals / locations 110 and the computers 120 include or are attached to communication devices. Communication is intended to occur by industry standard protocols such as HTTP or HTTPS.

[0017] Each computer 120 is comprised of a central processing unit 122, a storage medium 124, a user input device 126, and a display 128. Examples of computers that may be used include commercially available personal computers, open source computing devices (e.g., Raspberry Pi), commercially available servers, and commercially available mobile devices (e.g., smartphones, smart watches, tablets). In one embodiment, each peripheral 110 and each computer 120 of the system may have software associated with the system installed. In such an embodiment, system data may be stored locally on the networked computer 120, or alternatively, on one or more remote servers 140 accessible via network 130 to either the peripheral 110 or the networked computer 120. In an alternative embodiment, the software runs as an application on the peripheral 110, and includes web-based software and iOS- and Android-based mobile applications.

[0018] FIG. 2 is a flow diagram illustrating the process of compressing a raw FastQ / A file 202 to generate a compressed FastQ / A file 236. FastA refers to a text-based format for representing nucleotide or amino acid (protein) sequences, where the nucleotides or amino acids are represented by single-letter codes. FastQ refers to a format for files that store the nucleotide base sequence, the calculated confidence of each base in the sequence, and information describing the read from its origin to its location on the flow cell of the sequencing platform. FastA files can be compressed instead of or in addition to the raw FastQ / A file 202. FastA files contain only identifiers and sequence bases. The process consists of two main stages: domain-specific compression and regular compression. The compressor aims to recognize redundant and similar patterns in the FastQ / A data and use them for lossless compression of the dataset in binary. The process starts with the FastQ file 202 being sent to the FastQ / A reader 203, where the file 202 is loaded and read. A FastQ / A file is composed of sequence bases 204, quality scores 206, and identifiers 208. The identifiers 208 contain information about the sequencing run and clusters. The bases 204 contain a sequence of base calls. The quality scores 206 contain a sequence of quality values ​​for the base sequences, typically one quality value per base 204. The FastQ / A reader 203 loads and reads the base sequences 204, quality score sequences 206, and identifiers 208 in the FastQ / A file 202 into memory and creates base sequence, quality score sequences, and identifier reads.

[0019] As shown in Table 1, in DNA / RNA, there are nucleotides other than A, C, G, T and U, e.g. N, R, W. These bases are known as irregular bases. In this method, all irregular bases related to base 204 are temporarily stored as "A" in regular bases. The system detects and stores the type and position of each irregular base 216 in the FastQ / A dataset. If there are no more irregular bases in the FastQ dataset, the file may be excluded. If a one-to-one or many-to-one mapping from quality scores 206 to irregular bases is possible in the dataset, the quality scores 206 may be used to localize the irregular bases. Otherwise, create the file. Another scenario in which the file is created is when the FastA file is compressed. [Table 1]

[0020] The system performs a template comparison which involves storing a template of consistent identifiers 208 across samples / datasets. The variables are then encoded. Since not all variables change from read to read, the template comparison first checks to determine if the variable has changed and, if so, stores the difference in the variable from the previous read.

[0021] The optimizer 210 performs statistical analysis on the distribution of each of the quality scores 206, the sequence k-mers from the bases 204, and the identifier variables 208. A k-mer is a substring of length k contained in a base sequence. In a particular embodiment, an optimization algorithm is applied separately to each of the quality scores 206, the sequence k-mers from the bases 204, and the identifier variables 208. The optimization variables are the hyperparameters of the downstream blocks. The system employs a mathematical formulation based on the encoding protocol of an objective function of the optimizer 210 that is a function of the protocol hyperparameters and the dataset distribution index. The objective function is directly related to the compression rate and is minimized by the optimizer 210. The result of the optimization process adaptively and optimally forms a protocol for each dataset.

[0022] The statistical characteristics of the quality scores, sequences and components of the identifier variables vary from one data set to another. The performance of a redundancy-based compressor depends on the frequency and distribution of the redundancy patterns and their level of imbalance. (As an example, if quality score=30 is 90% of all quality scores in the range 1-40, then the distribution of quality scores is highly imbalanced, but if all 40 possible quality scores account for 2.5% of the data set, then the distribution is perfectly balanced). A frequency-based compressor will not perform optimally unless it is customized or tuned for each FastQ or FastA data set based on its own distribution, as is the case with optimizer 210.

[0023] The bases 204 are analyzed by the system of the present technology as follows: Meta information about the characteristics of the sequence part of the dataset, such as the obtained optimal hyperparameter values, distribution characteristics and the number of different types of irregular symbols in the dataset, etc., is stored using the header 218. The regular bases 214, including the DNA bases 204, consist mainly of adenine (A), cytosine (C), guanine (G) and thymine (T), while the RNA includes adenine (A), cytosine (C), guanine (G) and uracil (U). Preferably, only two bits are required to encode each base in binary form.

[0024] The identifier 208 is sent to an identifier compression module 214 where the identifier 208 is encoded and compressed, for example using the algorithm outlined below. The encoded and compressed identifier is sent from the identifier compression module 214 to a generic compressor 234 which is responsible for generating a compressed file, preferably using a lossless generic compressor (e.g. gzip). Input: Identifier Output: Compressed Identifier (Ident_change_Index, Ident_change_value) 1. Remember the identifier of the first lead 2. For each tile, xpos, and ypos, calculate the Ident_change_Index that indicates whether or not it has changed since the previous read. (y: 1, n: 0) 3. Calculate Ident_change_value separately for tile, xpos and ypos, which indicates the amount of change (that has changed).

[0025] The quality scores 206 and sequence bases 204 are sent to a first read splitter 210 and a second read splitter 212, respectively. The first and second read splitters 210, 212 split the long reads of data into smaller reads (segments). Since the segments need to be of equal length, depending on the read_length (RL) and the number of segments required, it may be necessary to first split off a fixed number of nucleotides or quality scores (name: extra_len) from the end of the read to make the segments of equal length. The split off extra parts are called extra segments.

[0026] An exemplary embodiment of the read splitter function is described below. Input: Base or quality score sequence Output: (segmented) base sequence or quality score 1. Calculate num_seg, seg_len, and extra_len.

number

number

number

[0027] After the first read splitter 210 and the second read splitter 212 split the long reads into quality scores 206 and sequence bases 204, the split data from the quality scores 206 is sent to a first duplicate removal module 216 and the split data from the sequence bases 204 is sent to a second duplicate removal module 218. An exemplary process of duplicate removal is shown in Figure 3. The first and second duplicate removal modules 216, 218 remove duplicate data entries as shown in the following exemplary algorithm. Input: Base sequence Output: base_sequences (DR_remaining_reads), DR auxiliary information (DR_index) 1. Sort all input leads alphabetically but retain their lead numbers. 2. Divide identical reads into groups (some groups have single members, some have multiple members). 3. For each read in the input file, encode the corresponding index from the table below and call it the DR_index file. S Member of a single-member group M First member of a multi-member group Gi i-th member of a multi-member group other than the first 4. Keep only the reads with indices S and M (in the original order), call them DR_remaining_reads, and remove the others. Note: Multi-member groups are listed in alphabetical order.

[0028] Once duplicate data entries associated with the quality score 206 have been removed by the first duplicate removal module 216, the data associated with the quality score 206 is sent to a generic compressor 234, which is responsible for generating a compressed file using a lossless generic compressor (e.g., gzip).

[0029] Similarly, once duplicate data entries associated with sequence base 204 have been removed by second duplicate removal module 218, the data associated with sequence base 204 may be sent directly to a general-purpose compressor 234, which is responsible for generating a compressed file using a lossless general-purpose compressor (e.g., gzip).

[0030] In certain embodiments, data associated with sequence bases 204 is sent from the second duplicate removal module 218 to a semi-duplicate removal module 220, which removes similar data reads from the data associated with sequence bases 204. An example process for semi-duplicate removal is shown in Figure 4. An exemplary algorithm used by the semi-duplicate removal module 220 is outlined below. Input: Base sequence, default hyperparameters (SDR_diff) Output: base_sequences (SDR_remaining_reads), SDR auxiliary information (SDR_read_numbers, SDR_change_loc), SDR extra base sequence (SDR_change_seq) 1. Sort all input leads alphabetically but retain their lead numbers. 2. Group semi-identical reads, where semi-identical means that the first difference between each member of the group and the previous member is at a position equal to or less than "SDR_diff". Thus, some groups are single-membered and some are multi-membered. In an SDR block, the change is considered as a subsequence from the first difference to the end of the sequence. Example: seq1: AAAAAAAAAA, seq2: AAAAAAACAA => changed sub-seq=CAA 3. Create an empty "SDR_change_seq" string and an "SDR_change_loc list". Add the changes between each member of the multi-member set and the previous member to the "SDR_change_seq" string, and add the length of the changed sub-seq to "SDR_change_loc". 4. List the first members of the first and multi-member groups, sort them by read number, and call them SDR_remaining_reads. 5. Remove the read number of the first member of the group from read numbers and save as "SDR_read_numbers".

[0031] Once similar data reads have been removed in the semi-duplicate removal module 220, the data may be sent directly to a generic compressor 234 for lossless compression, sent to a random sequence generator module 222 to generate a random subset of the data, or sent to a dictionary subsequence removal module 226. Depending on user settings, the random sequence generator module 222 may take smaller or larger random samples of the data to conserve computing resources while maintaining the integrity of the system. Exemplary algorithms used by the random sequence generator module 222 are outlined below. Input: sequence bases, default hyperparameters (random_subset_percentage) Output:random_subset 1.random_subset_percent % of base sequences are randomly selected Note: random_subset_percent=100 gives an ideal dictionary, but for smaller samples you can get much better dictionaries in less time.

[0032] Once the random sequence generator module 222 has generated a random sample of data, it is passed through the subsequence dictionary optimizer module 224 which creates a k-mer dictionary.

[0033] The dense k-mers associated with bases 204 are listed and labeled in a binary format in the k-mer dictionary. For each k-mer in the k-mer dictionary, there is an index that labels the k-mer. These indices are coded in binary. During the data compression process, instead of the actual k-mer sequence, it is the index of the k-mer that is recorded. The optimal values ​​of some k-mers in the dictionary are specified by solving an optimization problem. In an exemplary solution of the optimization problem, an objective function is developed based on a compression algorithm to roughly estimate the compressed size of a random subset of the data set without actually compressing it. Next, optimization variables, which are hyperparameters of the compression algorithm, are specified. The values ​​of the optimization variables are obtained using a deterministic or stochastic optimization algorithm such that the objective function is minimized. Finally, the obtained optimal values ​​are used as the hyperparameter values ​​for compression of the entire data set.

[0034] Dimensionality reduction may also be performed using a k-mer dictionary and binary labels from regular bases 214 associated with bases 204. Machine learning algorithms that apply techniques other than dimensionality reduction may also be utilized, including, but not limited to, nearest neighbor analysis, Bayesian modeling, regression modeling, support vector machines, and neural networks.

[0035] In a particular embodiment, the system performs an operation called "companion of k-mers", which predicts what comes before and after a k-mer (left and right companions) and stores them in a dictionary together with the k-mer, as shown in FIG. 5. Referring to FIG. 5, during the creation of the dictionary of k-mers, the system of the present invention searches for frequent subsequences that are perfect matches in the file. However, for companions, the system of the present invention searches for frequent sufficiently similar subsequences (non-perfect matches) before or after the k-mer. Then, the k-mers and companions can be selected based on statistical features present in the example algorithm. During the creation of the dictionary, k-mers are counted if they are "perfect matches" and companions are counted if they are "predominant" and "statistically significant" at their position compared to the k-mer.

[0036] Additionally, in certain embodiments, the system performs a dynamic dictionary creation process. For each subsequence (mer) added to the dictionary, there is an internal competition between the k-mers for each k value in the k-list (e.g., for k-list=[10, 15, 20], there are three internal competitions, one for each k value). The internal competition is followed by an external competition between the winners of the internal competition (in this example, the best 10-mer, the best 15-mer, and the best 20-mer). The winners of the external competition are added to the final dictionary, an example of which is shown in FIG. 6. The length of the final dictionary is specified during the dictionary creation process based on an optimization cost function. An exemplary algorithm for dynamic dictionary creation is outlined below. Input: a random subset of the array, default hyperparameters: (k-list, existancy_threshhold, dominancy_threshhold). Output: k-mer dictionary Counting: 1. preliminary_dict is called with length len(k list ), and then create an array of k list For each k value in , the array elements are k-mer dictionaries such as [key: k-mer sequence, value: k-mer frequency] Initialization: 2. For each k value in the k list Counting the frequency of ak-mers Reorder bk-mers based on their frequency c. Count the frequency of each nucleotide (A, C, G, and T) at position 1, which is next and previous to the k-mer (RL-k) sorted to position 1 d. For every x number of the first k-mers, consider the first (y_right and y_left) nucleotides on the right and left side of the k-mer as prel_companion_right and prel_companion_left that can pass the following two thresholds: existency_ratio(k-mer(i), position(j,RL(right or left))) = (number of reads with k-mer(i) and any nucleotide at position (j) on the RL side of the k-mer) / (number of reads with k-mer(i)) dominancy_ratio(k-mer(i), position(j,RL(right or left))) = (number of reads with the dominant nucleotide at k-mer(i) and position(j) on the RL side of k-mer) / (number of reads with any nucleotide at k-mer(i) and position(j) on the RL side of k-mer(i)) 3. Calculate the (average + 2 * standard_deviation) of (y_right(k-mer(i))+y_left(k-mer(i))), which is considered as the initialized length of all k-mer companions for each k value. Competition: In the preliminary dictionary, k-mers are sorted based on their frequency, but because the length of the initialized companions of all k-mers is the same, they are actually sorted based on k-mer + initialized companion as well. 4. Create an empty list called read_list, which will be filled with the read indices of the reads that contain the final dictionary k-mers. Internal conflict: 5. For each value of k in the k list, a) Get the internal_compression_score of all k-mers.

number

number

number

number

number

number

number

number

number

number

number

number

[0037] The data generated from the subsequence dictionary optimizer module 224 is passed to a dictionary subsequence removal module 226, which is responsible for removing certain k-mers from the dictionary, as shown in Figure 7. An exemplary algorithm used by the dictionary subsequence removal module 226 is described below. Input: base sequence, k-mer dictionary. Output: base_sequences (untouched_seqs), Dr auxiliary information (k-mer_dict_exist, k-mer_dict_index, k-mer_dict_loc, comp_loc_change, comp_num_change,) k-mer removal extra sequence (k-mer_dict_remain_seqs, comp_type_change) k-mer k-mer k-mer 1. Create seven empty strings to encode information about the sequence, including the dictionary k-mers. k-mer_dict_exist (encodes the presence of a dictionary k-mer in a read) k-mer_dict_index (encodes the index of the dictionary k-mer in the read) k-mer_dict_loc (encodes the start position of a dictionary k-mer in a read) comp_num_change (encodes the number of mismatches with dictionary k-mer companions in a read) comp_loc_change (encodes the position of a mismatch in a read with a dictionary k-mer companion) comp_type_change (encodes the correct nucleotide type for a mismatch with a dictionary k-mer companion in a read) k-mer_dict_remain_seq (encodes the remainder of the read after removing k-mers and companions) untoched_seqs (sequences that did not contain any k-mer from the dictionary) 2. Encode one bit per read as k-mer_dict_exist: 0 if the read does not contain any k-mer from the final k-mer dictionary, 1 if it does. 3. If the final dictionary k-mer does not exist in the read, Encode the dictionary index of the k-mer in ak-mer_dict_index. In bk-mer_dict_loc we encode the position of the first base of the k-mer in the read. c. Find the number of mismatches between the final dictionary predicted companions and the actual companions of the k-mer in the read and encode it in comp_num_change. In d.comp_loc_change we encode the location of the difference found in the previous step. Encode the nucleotide type of the difference detected in step 3.c in e.comp_loc_type. Encode the remainder of the reads after removing k-mers and companions in fk-mer_dict_remain_seq. NOTE: In this version of the compression technique, a maximum of one k-mer per read is considered, and if there is more than one in a read, the one with a higher rank in the k-mer_dictionary is selected since it results in a higher final compression ratio.

[0038] From the dictionary subsequence removal module 226, the modified data is sent to a second semi-redundancy removal module 228, which removes similar data reads from the data associated with the sequence bases 204, as described above. In certain embodiments, the system uses a second semi-redundancy removal module 228 that has the same process as the first semi-redundancy removal module 220, but with different parameter values ​​and reversed sequence order.

[0039] Once similar data reads are removed from the data, the data is sent to a sequence regularizer module 230 which is responsible for encoding irregular nucleotides. An irregular nucleotide is one that is not A (adenosine), C (cytosine), G (guanine), or T (tyrosine). An exemplary algorithm used by the sequence regularizer module 230 is described below. Input: base sequence and / or extra sequence. Output: Sequence and / or extra sequences, sequence regularizer auxiliary information (irregular_reads_existence, irregular_nt_number, irregular_nt_location, and non_N_irregular_nt_info). Note: irregular nucleotide means any nucleotide other than A, C, G and T. 1. Create empty strings called irregular_reads_existance, irregular_nt_number irregular_nt_location and non_N_irregular_nt_info to encode the presence, number, location of irregular nucleotides and information about irregular non-N nucleotides. 2. irregular_reads_existence encodes one bit per read: 0: no nucleotides in this read; 1: at least one nucleotide is present. 3.Encode the irregular nucleotide for each irregular read in irregular_nt_number. 4. irregular_nt_location encodes the location of each irregular nucleotide in the read. 5.non_N_irregular_nt_info, by default all irregular nucleotide characters are assumed to be N (the most common irregular nucleotide) and if any irregular non-N nucleotide is observed, its information is encoded as non_N_irregular_nt_info(irregular_read_index (i-th irregular read), irregular_nt_location index (i-th irregular nucleotide in that irregular read), type). 6.non_N_irregular_nt_info=[irregular_read_index, irregular_nt_location index, type]

[0040] Once the irregular nucleotides are encoded by the sequence regularizer module 230, the data is sent to the sequence binary encoding module 232, which is responsible for binary encoding the base sequence 204. An exemplary algorithm used by the sequence binary encoding module 232 is described below. Input: base sequence. Output: binary_base sequences 1. The sequence is encoded in binary using the following hash table: Nucleotide Binary Encoded Hash Table [Table 2] NOTE: Irregular nucleotides are temporarily stored as A:

[00] because their information is recorded separately.

[0041] The encoded data from the sequence binary encoding module 232 is sent to a universal compressor 234. At the universal compressor 234, the encoded binary data set is compressed again with a lossless universal compressor (e.g., gzip). The result is a compressed FastQ / A file 236. In various embodiments, data may be sent directly to the universal compressor 234 from one or more of the first semi-duplicate removal module 220, the subsequence dictionary optimizer 224, the dictionary subsequence removal module 226, the second semi-duplicate removal module 228, and / or the sequence regularizer module 230. An exemplary algorithm for the universal compressor is described below. Input: sequence regularizer auxiliary information, k-mer removal auxiliary information, k-mer dictionary auxiliary information, SDR auxiliary information, DR auxiliary information, quality score sequence, binary base sequence, compressed identifier. Output:final_compressed_file The goal is to achieve maximum compression after the general purpose compressor, not before, so although further compression in the domain-specific compression is possible, they are not included in the process because their impact on the final file is small or worse. Hyperparameters used for optimization: k list random_subset_percentage existency_threshhold dominancy_threshhold patient_factor max_dict_len η penalty ideal_len N*(1st SDR block) N*(2nd SDR block)

[0042] The disclosed process has numerous advantages over the prior art, including:

[0043] (1) Customized Compression: The encoding protocol is not strict. Instead, the encoding protocol (and the corresponding decoding protocol) is optimally tuned based on the characteristics of each specific dataset.

[0044] (2) Compression ratio: The disclosed encoding protocol achieves a higher compression ratio than general compressors and genomic data compressors.

[0045] (3) Reference-free: The compressor is reference-free, eliminating the need to perform the computationally expensive process of aligning sequences to a reference genome.

[0046] (4) Zero loss: The entire encoding / decoding process is lossless.

[0047] (5) FastA files: The encoding algorithm can compress sequence data independently. Therefore, in addition to FastQ files, the compressor can also be used to compress FastA files.

[0048] (6) Irregular non-N bases: Irregular bases are not limited to N in the encoding process of this method.

[0049] (7) Compression Time: The disclosed encoding protocol can compress data faster than general compressors and genomic data compressors.

[0050] The above description and drawings should be considered as merely illustrative of the principles of the invention. The invention is not intended to be limited to the preferred embodiment, but can be embodied in various ways obvious to those skilled in the art. Numerous applications of the invention will be readily apparent to those skilled in the art. It is therefore not desired to limit the invention to the specific examples disclosed or to the specific construction and operation shown and described. Rather, all suitable modifications and equivalents may be used as falling within the scope of the invention. All references cited herein are incorporated by reference.

Claims

1. receiving a data file comprised of sequence bases, quality scores, and identifiers, the sequence bases including regular bases and irregular bases, the regular bases including adenine (A), cytosine (C), guanine (G), and thymine (T); applying an optimization algorithm to the quality scores, sequence k-mers from the sequence bases, and the classifier; Dividing the long reads of the sequence bases and the quality scores into smaller segments; removing duplicate and semi-duplicate reads of the sequence bases and the quality scores; performing dimensionality reduction on the sequence bases; storing a template of said identifiers that is consistent across said data files; detecting and storing the position and type of each of the irregular bases; encoding the data file in binary format such that each of the regular bases is represented by one of four two-digit binary numbers and all of the irregular bases are represented by one of the four two-digit binary numbers; compressing the encoded data file. Data compression methods for genomic data.

2. The method of claim 1 , wherein the identifier is composed of sequencing run data and cluster data.

3. The method of claim 1 , wherein the quality score comprises a sequence of quality values ​​of the sequence bases.

4. The irregular bases include N and a plurality of additional irregular bases; storing the position and type of each of the irregular bases includes storing the position of each of the irregular bases and the type of each of the additional irregular bases; The method of claim 1.

5. The method of claim 4 , wherein dimensionality reduction is performed using a k-mer dictionary and binary labels from the base sequences.

6. The method of claim 1 , wherein the data file is a FastQ file or a FastA file.

7. The method of claim 1 , wherein the optimization algorithm determines optimal values ​​for encoding algorithm hyperparameters, sequence k-mers from the bases, and the identifiers.

8. The method of claim 1 , wherein the optimization algorithm is a function of protocol hyperparameters and distribution indices of the data file.

9. The method of claim 1 , wherein the compression is reference-free.

10. The method of claim 1 , wherein the compression is adjusted based on characteristics of the data file.

11. On the computer, receiving a data file comprised of sequence bases, quality scores, and identifiers, the sequence bases including regular bases and irregular bases, the regular bases including adenine (A), cytosine (C), guanine (G), and thymine (T); applying an optimization algorithm to the quality scores, sequence k-mers from the sequence bases, and the classifier; Dividing the long reads of the sequence bases and the quality scores into smaller segments; removing duplicate and semi-duplicate reads of the sequence bases and the quality scores; performing dimensionality reduction on the sequence bases; storing a template of said identifiers that is consistent across said data files; detecting and storing the position and type of each of the irregular bases; encoding the data file in binary format such that each of the regular bases is represented by one of four two-digit binary numbers and all of the irregular bases are represented by one of the four two-digit binary numbers; a. compressing the data file, wherein the compression is lossless.

12. The non-transitory computer medium of claim 11 , wherein the identifier is comprised of sequencing run data and cluster data.

13. 12. The non-transitory computer medium of claim 11, wherein the quality score comprises a sequence of quality values ​​of the sequence bases.

14. The irregular bases include N and a plurality of irregular non-N bases; 12. The non-transitory computer medium of claim 11, wherein storing the position and type of each of the irregular bases comprises storing the position of each of the irregular bases and the type of each of the irregular non-N bases.

15. 15. The non-transitory computer medium of claim 14, wherein dimensionality reduction is performed using a k-mer dictionary and binary labels from the regular bases.

16. 12. The non-transitory computer medium of claim 11, wherein the data file is a FastQ file or a FastA file.

17. 12. The non-transitory computer medium of claim 11, wherein the optimization algorithm determines optimal hyperparameter values ​​for each of the quality scores, sequence k-mers from the bases, and the identifiers.

18. The non-transitory computer medium of claim 11 , wherein the optimization algorithm is a function of protocol hyperparameters and distribution indices of the data file.

19. The non-transitory computer medium of claim 11 , wherein the compression is reference-free.

20. The non-transitory computer medium of claim 11 , wherein the compression is adjusted based on characteristics of the data file.

Images