A parallel compression algorithm for gene sequences based on distributed engines

Abstract

This invention discloses a parallel compression algorithm for gene sequences based on a distributed engine. First, the input file is partitioned: TextInputFormat is used as the internal partitioning method for the input SAM file. Second, a single-machine compression algorithm is performed: the QUAL field is reordered to increase local redundancy; a CIGAR field is introduced for differential encoding to reduce the amount of encoding; a Bloom filter is used to construct a high-quality index and perform matching encoding. Appropriate encoding and compression tools are selected according to the type and variation characteristics of the remaining fields. Finally, the compression algorithm is parallelized: a fault-tolerant parallel compression process is designed using RDD operators. This invention employs multi-threading to process multiple jobs in parallel and utilizes a high-performance serialization mechanism to reduce the data transmission cost during the shuffle process; effectively improving the compression speed and efficiency of SAM gene sequences.

Description

Technical Field

[0001] This invention relates to the field of compression algorithms, and in particular to a parallel compression algorithm for gene sequences based on a distributed engine. Background Technology

[0002] Gene sequencing refers to the detection of the types of the four major bases (A, C, G, and T) in the DNA sequence of an organism. The development of next-generation sequencing (NGS) technology has greatly reduced the cost of genome sequencing and significantly increased the amount of data generated. Over the past decade, the cost of sequencing the entire human genome has decreased from one billion US dollars to around one thousand US dollars. Due to the continuous decline in cost and the urgent need for gene sequencing data from researchers, large-scale gene sequencing projects are emerging, such as the 1000 Genomes Project and the Human Microbiome Project.

[0003] Sequence Alignment / Map (SAM) is a gene sequence storage format that aligns sequencing results with reference sequences. Currently, the SAM format has become the standard for gene sequencing data analysis, making the compression of SAM files particularly important.

[0004] No-reference compression algorithms do not rely on external reference sequences; their compression principle is to utilize the repetition within the sequence to be compressed to reduce information. These algorithms offer greater flexibility in compression and decompression processes and are less prone to decompression errors. No-reference compression algorithms are also more adaptable to different environments, ensuring orderly compression even in complex computational settings. Specifically, no-reference algorithms can be further divided into three types: methods based on general-purpose compression tools, methods based on statistical prediction, and methods based on hypothetical references.

[0005] To fully mine and utilize the genetic data contained in sequencing data, it is necessary to ensure the long-term availability of genetic data, thus requiring persistent storage of gene sequencing data. Data compression is an effective solution to reduce the storage and transmission costs of massive genetic data. Approximately two-thirds of the human genome contains nearly repetitive segments, including not only perfect matches but also inversions and palindromes. Furthermore, there is significant similarity between different gene sequences within the same species. To achieve better compression ratios, dedicated SAM (Sequence Alignment / Map) gene compression algorithms typically leverage these biological properties of gene sequences. However, the compression ratio of existing SAM gene sequence compression algorithms still needs further optimization to reduce costs. In addition, current parallelization strategies for SAM gene compression mainly rely on single-machine multithreading, lacking distributed parallel compression algorithms. Hadoop and Spark are currently popular distributed big data platforms. Whether in local clusters or cloud computing scenarios (AWS, Azure, etc.), compression algorithms based on distributed platforms can mobilize the resources of multiple nodes to achieve efficient parallel acceleration. Distributed parallel compression algorithms offer powerful computing capabilities and have significant advantages in throughput, speed, and scalability. To provide strong support for subsequent genome analysis, there is an urgent need for more advanced SAM gene compression algorithms and distributed parallel algorithms to alleviate the problems of insufficient storage space and low transmission efficiency. Summary of the Invention

[0006] To address the aforementioned technical challenges, this invention proposes a parallel gene sequence compression algorithm based on a distributed engine. It utilizes HDFS for distributed storage of large-scale SAM gene data, overcoming the limitations of single-machine hard drive storage space; and leverages the rich parallel operators provided by Spark to achieve parallel compression of SAM gene sequences, thereby fully utilizing the storage and computing resources of the distributed cloud computing environment.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A parallel compression algorithm for gene sequences based on a distributed engine, the method specifically includes the following steps:

[0009] Step S1: Partition the SAM input file;

[0010] Step S2: Parallel compression process for the QUAL field;

[0011] Step S3: Parallel compression process 1 for SEQ related fields;

[0012] Step S4: Parallel compression process 2 for SEQ related fields;

[0013] Step S5: Parallel compression process for the remaining fields;

[0014] Step S6: Optimization of the parallel compression algorithm.

[0015] Further, the specific process of step S1 is as follows: using TextInputFormat, partition the data according to the amount of data; the expected partition data size splitSize will be the minimum of goalSize and blockSize, and the result will be the maximum of minSize; where goalSize is the ratio of the total amount of data in all input files to the minimum number of partitions, blockSize is the default block size in HDFS, and minSize is the minimum value of partition data; after calculating splitSize, traverse all files and partition each file separately according to splitSize; during the process of partitioning a file separately, if the remaining data size is greater than 1.1 times splitSize, a new partition will be created with a data size of splitSize; after continuous iteration, the final remaining data size is usually less than 1.1 times splitSize, at which point all the remaining data will be merged into a new partition.

[0016] Furthermore, step S2 specifically includes the following steps:

[0017] Step S21: The SAM file to be compressed is first partitioned by TextInputFormat to obtain RDD1; each element in RDD1 stores a sequence of a line in the SAM file to be compressed, and each line already contains the original order; the file to be compressed is divided into multiple partitions, and parallel computation will be performed between the multiple partitions before encountering the Shuffle operator.

[0018] Step S22: RDD1 uses the mapPartitionsToPair operator to calculate the frequency score and graph similarity score for each row, and concatenates the two scores using "_" as the key in the next RDD. Furthermore, when compressing the QUAL field, subsequent processes will only use the original order and QUAL fields. To avoid storing a large amount of useless data in memory, all fields other than the two mentioned above are discarded, and the result is used as the value of RDD2. The above operation is consistent with the column pruning optimization idea in Spark SQL. In the Spark SQL execution flow, rule-based optimization is performed on the logical execution plan, and column pruning is an effective optimization strategy. Column pruning first analyzes the logical execution plan tree containing metadata to identify fields that are not needed in subsequent SQL queries, and then removes the relevant fields from the tree nodes during the preceding execution process.

[0019] Step S23: RDD2 will be reordered in two levels using sortByKey; elements with the same key will flow to the same Partition and be sorted in ascending order according to the concatenated score; after sorting, RDD3 is obtained, and the element type of RDD3 remains unchanged from RDD2; then in mapPartitions, the final order stream and quality score stream are obtained from the value of RDD3, and differential encoding and adaptive encoding are performed respectively, and then sent to the ZPAQ compressor for compression; the compressed result is obtained in RDD4, where orderComArr and qualComArr represent the order compressed byte array and the quality score compressed byte array respectively;

[0020] Step S24: Write the result to HDFS using saveAsObjectFile. The compressed result will be saved in the form of SequenceFile. saveAsObjectFile is a typical action operator that will trigger the submission of the QUAL field compression job.

[0021] Furthermore, step S3 specifically includes the following steps:

[0022] Step S31: In Job1, RDD1 will be constructed into key-value RDD5 using mapPartitionsToPair; the key of RDD5 is a concatenated string of RNAME and POS, separated by "_"; the value of RDD5 only contains SEQ, CIGAR, and original order information; after sorting by sortByKey, RDD6 is obtained, and RDD6 will construct the hypothesis reference sequence in the mapPartitionsWithIndex operator; mapPartitionsWithIndex will provide the partition number during iterative operation, resulting in RDD7; RDD7 contains a large number of quintuples, each quintuple storing the partition number, hypothesis reference sequence, offset starting position list, SEQ sequence list, and other information lists; the other information includes RNAME, POS, CIGAR, and the final sorting order;

[0023] Step S32: Extract the hypothetical reference sequence, offset information, SEQ sequence, and CIGAR field information from RDD7 in Job2, and then perform differential encoding and PPMD ​​compression in mapPartitions, using byte arrays as the new element organization form; differential encoding will only be applied to the SEQ sequence available to RNAME; then write the compressed results of the hypothetical reference sequence and the SEQ sequence available to RNAME to HDFS via saveAsObjectFile; in Job3, the information to be extracted is stored in other information of RDD7; run-length encoding or differential encoding is performed in mapPartitions, and then PPMD ​​is used to obtain the compressed results of RNAME, POS, CIGAR, final sort order, etc., and finally saved to HDFS.

[0024] Furthermore, in step S4, Job4 and Job5 jointly process SEQ sequences where RNAME is unavailable. The purpose of Job4 is to obtain the number of partitions whose RNAME contains "*". First, the quintuples in RDD7 need to be filtered using the filter operator. Only when the first element in the other information list contains "*" in RNAME will the quintuple be retained. Here, the filter operation is actually performed on a quintuple-by-quintuple basis, which consumes very little time. Then, the number of partitions is counted using the count operator and assigned to the partitionCount variable in the main function. In Job5, the filter operator is used to process the SEQ sequences where RNAME is unavailable. The `partitionCount` variable is referenced, and only 5-tuples with an index less than `partitionCount+3` are retained, ensuring that RDD10 contains a portion of the hypothesis reference sequence. The `coalesce` operator is used to merge partitions, allowing multiple partitions to share the hypothesis reference sequence. In the `mapPartitions` operator, the hypothesis reference sequence and the SEQ sequence (where RNAME is unavailable) of RDD10 are first extracted. An index is constructed for the hypothesis reference sequence, and matching, encoding, and PPMD ​​compression are performed. Similarly, the compression results are saved to HDFS. The compression results of different jobs are placed in different folders, independent of each other and without affecting each other.

[0025] Furthermore, the specific process of step S5 is as follows: the remaining 7 fields are divided into 3 jobs according to the category of the final compressor using an algorithm, and the same compressor is used within the same job; Job6 is responsible for the QNAME, RNEXT, PNEXT, TLEN, and OTHER fields, Job7 is responsible for the FLAG field, and Job8 is responsible for the MAPQ field; in the mapPartitions of each job, the fields to be processed by the current job are first extracted from the input data stream of RDD1; if there are multiple fields, they need to be separated and arranged together vertically; after information extraction, encoding and compression operations are performed on a partition-by-partition basis; after obtaining the compressed byte array of each field, the final compression result is saved using saveAsObjectFile.

[0026] Furthermore, the specific process of step S6 is as follows: First, a multi-threading mechanism is introduced to achieve Job parallelization. In the user program, after generating RDD1, five sub-threads will be created. The execution bodies of sub-threads 1 to 4 are the operator processes of Job1, Job6, Job7, and Job8, respectively. The execution body of sub-thread 5 is the operator process for compressing the SEQ-related fields after generating RDD1. After generating RDD7, three more sub-threads will be created, with the execution bodies of threads 6 to 8 being the operator processes of Job2, Job3, Job4, and Job5, respectively. By creating sub-threads for independent Jobs, multiple jobWaiters will be generated simultaneously in the main thread. The Stage division, TaskSet and TaskManager generation, Task serialization, Executor scheduling, and waiting for each Job to complete all have the opportunity to be executed in parallel. Secondly, a high-performance serialization mechanism is used to register all custom types used in the SAM compression algorithm as classes using Kyro serialization.

[0027] The present invention adopts the above technical solution and has the following beneficial effects:

[0028] 1. This invention utilizes HDFS to achieve distributed storage of SAM gene sequences and Spark to achieve parallel compression of SAM gene sequences, which can make full use of the storage and computing resources of the distributed cloud computing environment and significantly improve the compression speed of large-scale gene sequences.

[0029] 2. This invention can further alleviate the problems of insufficient storage space and low transmission efficiency of large-scale gene data, and provide strong support for subsequent gene analysis work. Attached Figure Description

[0030] Figure 1 Flowchart of parallel compression for the QUAL field;

[0031] Figure 2 Parallel compression subprocess 1 for SEQ related fields;

[0032] Figure 3 Parallel compression subprocess 2 for the SEQ related fields;

[0033] Figure 4 Flowchart for parallel compression of the remaining fields. Detailed Implementation

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0035] A parallel compression algorithm for gene sequences based on a distributed engine specifically includes the following steps:

[0036] Step S1: Partition the SAM input file;

[0037] Using TextInputFormat, data is partitioned according to its size. The expected partition size, splitSize, is the minimum of goalSize and blockSize, and the result is then multiplied by minSize. GoalSize is the ratio of the total data size of all input files to the minimum number of partitions, blockSize is the default block size in HDFS, and minSize is the minimum partition size. After calculating splitSize, all files are traversed, and each file is partitioned individually according to goalSize. During the partitioning process, if the remaining data size is greater than 1.1 times splitSize, a new partition with the same size is created. After this continuous iteration, the remaining data size is usually less than 1.1 times splitSize. At this point, all remaining data is merged into a new partition to ensure that the partition size is not too small and to avoid triggering the small file problem.

[0038] Step S2: Parallel compression process for the QUAL field;

[0039] Step S21: As Figure 1 As shown, the SAM file to be compressed is first partitioned by TextInputFormat to obtain RDD1; each element in RDD1 stores a sequence of a line in the SAM file to be compressed, and each line already contains the original order; the file to be compressed is divided into multiple partitions, and parallel computation will be performed between the multiple partitions before encountering the Shuffle operator.

[0040] Step S22: RDD1 uses the mapPartitionsToPair operator to calculate the frequency score and graph similarity score for each row, and concatenates the two scores using "_" as the key in the next RDD. Furthermore, when compressing the QUAL field, subsequent processes will only use the original order and QUAL fields. To avoid storing a large amount of useless data in memory, all fields other than the two mentioned above are discarded, and the result is used as the value of RDD2. The above operation is consistent with the column pruning optimization idea in Spark SQL. In the Spark SQL execution flow, rule-based optimization is performed on the logical execution plan, and column pruning is an effective optimization strategy. Column pruning first analyzes the logical execution plan tree containing metadata to identify fields that are not needed in subsequent SQL queries, and then removes the relevant fields from the tree nodes during the preceding execution process. This strategy can effectively reduce memory usage and also reduce disk I / O and network I / O during the subsequent Shuffle process. Because RDDs have weaker data expression capabilities than SQL, automatic column pruning optimization is not currently supported; manual optimization will be performed if the optimization conditions are met.

[0041] Step S23: RDD2 will be reordered in two levels using sortByKey; elements with the same key will flow to the same Partition and be sorted in ascending order according to the concatenated score; after sorting, RDD3 is obtained, and the element type of RDD3 remains unchanged from RDD2; then in mapPartitions, the final order stream and quality score stream are obtained from the value of RDD3, differentially encoded and adaptively encoded respectively, and sent to the ZPAQ compressor for compression to obtain RDD4; the compressed result is obtained in RDD4, where orderComArr and qualComArr represent the order compressed byte array and the quality score compressed byte array respectively;

[0042] Step S24: Write the result to HDFS using saveAsObjectFile. The compressed result will be saved in the form of SequenceFile. saveAsObjectFile is a typical action operator that will trigger the submission of the QUAL field compression job.

[0043] Step S3: Parallel compression process 1 for SEQ related fields;

[0044] Step S31: In Job1, as follows Figure 2As shown, firstly, RDD1 will be processed by mapPartitionsToPair to construct RDD5 in key-value form. The key of RDD5 is a concatenated string of RNAME and POS, separated by "_". According to the column pruning concept, the value of RDD5 only contains SEQ, CIGAR, and the original order information. After sorting by sortByKey, RDD6 is obtained. RDD6 will construct the hypothesis reference sequence in the mapPartitionsWithIndex operator. mapPartitionsWithIndex will provide the partition number during the iterative operation, resulting in RDD7, which is convenient for filtering data with unavailable RNAME in subsequent processes. RDD7 contains a large number of 5-tuples. Each tuple stores the partition number, hypothesis reference sequence, offset starting position list, SEQ sequence list, and other information lists. The other information includes RNAME, POS, CIGAR, and the final sorting order.

[0045] Since RDD1 and RDD7 need to be referenced by multiple operators, the algorithm performs Cache and CheckPoint operations on both RDD1 and RDD7 simultaneously, combining the advantages of both operations. Cache data can be entirely stored in memory, entirely stored on disk, or both simultaneously. The algorithm uses a memory- and disk-compatible approach, only performing disk flushing when memory is insufficient. This maximizes memory computing performance while avoiding frequent task retries caused by memory overflow. RDDs construct lineage relationships based on a directed acyclic graph. When several partitions in a cached RDD are corrupted, the corrupted data can be quickly calculated from the parent RDD, without affecting other intact partitions. CheckPoint does not recalculate intermediate RDD data in newly created jobs; instead, it directly retrieves the required data from the cache. Even if nodes in the cluster restart, the already calculated data can be directly recovered from HDFS, and subsequent operator processes can continue.

[0046] Step S32: In Job2, extract the hypothetical reference sequence, offset information, SEQ sequence, and CIGAR field information from RDD7. Then, perform differential encoding and PPMD ​​compression in mapPartitions, and use byte arrays as the new element organization form. Differential encoding will only be applied to the SEQ sequence available to RNAME. Afterwards, write the compressed results of the hypothetical reference sequence and the SEQ sequence available to RNAME to HDFS via saveAsObjectFile. In Job3, the information to be extracted is stored in other information of RDD7. Run-length encoding or differential encoding is performed in mapPartitions, and then PPMD ​​is used to obtain the compressed results of RNAME, POS, CIGAR, final order, and other data. Finally, it is saved to HDFS.

[0047] Step S4: Parallel compression process 2 for SEQ related fields;

[0048] like Figure 3 As shown, Job4 and Job5 jointly process SEQ sequences where RNAME is unavailable. The purpose of Job4 is to obtain the number of partitions whose RNAME contains "*". First, the quintuples in RDD7 need to be filtered using the filter operator. Only quintuples whose first element in other information lists contains "*" are retained. Here, filter actually performs probing and filtering operations on a quintuple-by-quintuple basis, which consumes very little time. Then, the number of partitions is counted using the count operator and assigned to the partitionCount variable in the main function. The filter operator in Job5 further filters the partitions. The `nCount` variable is referenced, and only 5-tuples with an index less than `partitionCount+3` are retained, ensuring that RDD10 contains a portion of the hypothesis reference sequence. The `coalesce` operator is used to merge partitions, allowing multiple partitions to share the hypothesis reference sequence. In the `mapPartitions` operator, the hypothesis reference sequence and the SEQ sequence (where RNAME is unavailable) of RDD10 are first extracted. An index is constructed for the hypothesis reference sequence, and matching, encoding, and PPMD ​​compression are performed. Similarly, the compression results are saved to HDFS. The compression results of different jobs are placed in different folders, independent of each other and without affecting each other.

[0049] Step S5: Parallel compression process for the remaining fields;

[0050] The parallel compression process for the remaining fields is as follows: Figure 4As shown, the algorithm divides the remaining 7 fields into 3 jobs according to the type of the final compressor. The same compressor is used within the same job to avoid confusion during decompression. Job6 is responsible for the QNAME, RNEXT, PNEXT, TLEN, and OTHER fields, Job7 is responsible for the FLAG field, and Job8 is responsible for the MAPQ field. In the mapPartitions of each job, the fields to be processed by the current job are first extracted from the input data stream of RDD1. If there are multiple fields, they need to be separated and arranged vertically. After information extraction, encoding and compression operations are performed on a partition-by-partition basis. After obtaining the compressed byte array of each field, the final compression result is saved using saveAsObjectFile.

[0051] Step S6: Optimization of the parallel compression algorithm;

[0052] First, parallelization of jobs is achieved by introducing a multi-threading mechanism. In the user program, after generating RDD1, five sub-threads are created. The execution bodies of sub-threads 1-4 are the operator processes of Job1, Job6, Job7, and Job8, respectively, while the execution body of sub-thread 5 is the operator process for compressing the SEQ-related fields after generating RDD1. After generating RDD7, three more sub-threads are created, with the execution bodies of threads 6-8 being the operator processes of Job2, Job3, Job4, and Job5, respectively. By creating sub-threads for independent jobs, multiple jobWaiters will be generated simultaneously in the main thread. The stage division, TaskSet and TaskManager generation, Task serialization, Executor scheduling, and waiting for each job to complete all have the opportunity to be executed in parallel. Second, a high-performance serialization mechanism is used to register all custom types used in the SAM compression algorithm as classes using Kyro serialization.

[0053] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be noted that due to the limitations of textual expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A parallel compression algorithm for gene sequences based on a distributed engine, characterized in that, Specifically, the following steps are included: Step S1: Partition the SAM input file; Step S2: The QUAL field portion enters the parallel compression process; Step S3: The SEQ related fields enter the parallel compression process 1. The SAM file to be compressed will first be partitioned by TextInputFormat to obtain RDD1; Step S31: In Job1, RDD1 will be converted into key-value RDD5 using mapPartitionsToPair; the key of RDD5 is a concatenated string of RNAME and POS, with "_" as the separator. RDD5's value contains only SEQ, CIGAR, and original order information; after sorting by sortByKey, it becomes RDD6, which constructs the hypothesis reference sequence in the mapPartitionsWithIndex operator; mapPartitionsWithIndex provides the partition number during iterative computation, resulting in RDD7; RDD7 contains a large number of quintuples, each storing the partition number, hypothesis reference sequence, offset starting position list, SEQ sequence list, and other information lists; the other information includes RNAME, POS, CIGAR, and the final sort order; Step S32: Extract the hypothesis reference sequence, offset information, SEQ sequence, and CIGAR field information from RDD7 in Job2. Then, perform differential encoding and PPMD ​​compression in mapPartitions, and organize the new elements as byte arrays. Differential encoding only applies to the SEQ sequence available to RNAME. Afterward, write the compressed results of the hypothesis reference sequence and the SEQ sequence available to RNAME to HDFS via saveAsObjectFile. In Job3, save the information to be extracted in other information of RDD7. Run-length encoding or differential encoding is performed in mapPartitions, and then PPMD ​​is used to obtain the compressed results of data such as RNAME, POS, CIGAR, and final sort order, which are finally saved to HDFS. Step S4: The SEQ-related fields enter parallel compression process 2: Job4 and Job5 jointly process SEQ sequences where RNAME is unavailable; Job4 obtains the number of partitions whose RNAME contains "*"; firstly, the quintuples in RDD7 need to be filtered using the filter operator. When the first element in the other information list contains "*", the quintuple is retained; then, the number of partitions is counted using the count operator and assigned to the partitionCount variable in the main function. In Job5, the filter operator references the partitionCount variable and only retains quintuples with an index less than partitionCount+3. RDD10 contains a portion of the hypothesis reference sequence. The coalesce operator merges partitions, allowing multiple partitions to share the hypothesis reference sequence. In the mapPartitions operator, the hypothesis reference sequence and the SEQ sequence (where RNAME is unavailable) of RDD10 are first extracted. An index is constructed for the hypothesis reference sequence, and matching, encoding, and PPMD ​​compression are performed. Finally, the compressed results are saved to HDFS. The compressed results from different jobs are placed in different folders, independent of each other and without affecting one another. Step S5: The remaining fields enter the parallel compression process. The remaining fields include QNAME, RNEXT, PNEXT, TLEN, OTHER, FLAG, and MAPQ. Step S6: Optimization of the parallel compression algorithm: Parallelization of the above steps is achieved by introducing a multi-threading mechanism.

2. The gene sequence parallel compression algorithm based on a distributed engine according to claim 1, characterized in that, The specific process of step S1 is as follows: using TextInputFormat, partition the data according to the data volume; the expected partition data volume. In and Take the minimum value from the middle and compare the result with the minimum value from the middle. Compare and take the maximum value; where This is the ratio of the total data size of all input files to the minimum number of partitions. This is the default block size in HDFS. It is the minimum value of the partition data volume; Calculate Then, it will iterate through all files and process each file individually. Partitioning; during the partitioning of a file individually, if the remaining data volume is greater than... If the data size is 1.1 times that of the original partition, a new partition will be created, with a data size of [missing value]. After repeated iterations, the final amount of remaining data is usually less than [amount missing]. The value is 1.1 times that of the previous partition, at which point all remaining data is merged into a new partition.

3. The gene sequence parallel compression algorithm based on a distributed engine according to claim 1, characterized in that, The specific process of step S5 is as follows: the remaining 7 fields are divided into 3 jobs according to the category of the final compressor using an algorithm, and the same compressor is used within the same job; Job6 is responsible for the QNAME, RNEXT, PNEXT, TLEN and OTHER fields, Job7 is responsible for the FLAG field, and Job8 is responsible for the MAPQ field; in the mapPartitions of each job, the fields to be processed by the current job are first extracted from the input data stream of RDD1; if there are multiple fields, they need to be separated and arranged together vertically; after information extraction, encoding and compression operations are performed on a partition-by-partition basis; after obtaining the compressed byte array of each field, the final compression result is saved using saveAsObjectFile.

4. The gene sequence parallel compression algorithm based on a distributed engine according to claim 1, characterized in that, The specific process of step S6 is as follows: by creating child threads for independent Jobs, multiple jobWaiters will be generated simultaneously in the main thread; the stage division, taskSet and taskManager generation, task serialization, executor scheduling, and waiting for each Job to complete execution all have the opportunity to be executed in parallel; secondly, a high-performance serialization mechanism is used to register all custom types used in the SAM compression algorithm as classes using Kryo serialization.

Images