A high-throughput genomic sequencing quality score data parallel compression method

Abstract

The application relates to the technical field of data compression storage, and provides a high-throughput genome sequencing quality score data parallel compression method.The method comprises the following steps: segmenting an original gene sequencing file; performing random sampling and k-mer analysis on the sampling data to obtain statistical characteristic information and establish a parallel sequence partition model for binary classification; splicing two partition files obtained through binary classification according to splicing parameters; predicting the to-be-compressed file through a multivariate linear regression analysis prediction method to obtain compression rate gain and establish a parallel four-level run prediction mapping model for data redundancy elimination; and performing context modeling on two redundancy-eliminated subfiles through a multi-core processor cluster, and performing cascade compression in combination with arithmetic coding to obtain a final compressed file.Under the premise of significantly reducing quality score data compression time and peak memory overhead, the application also improves the quality score data compression rate, reduces the size of to-be-compressed storage files, and saves the construction cost of basic storage facilities.

Description

Technical Field

[0001] This invention relates to the field of data compression and storage technology, and in particular to a parallel compression method for high-throughput genome sequencing quality score data. Background Technology

[0002] High-throughput genome sequencing data (HTGSD) is an important type of biological big data, widely used in drug development, genome-wide association studies, virus tracing, and precision medicine. In recent years, with the rapid development of high-throughput sequencing technology, the cost of HTGSD sequencing has significantly decreased, leading to a remarkable, faster-than-Moore's Law-like growth in the scale of HTGSD data. For example, the China National Gene Bank's sequence archive system alone holds 11,372 TB of high-throughput genome sequencing data compressed using the GZIP algorithm. This growth reflects the vital role and widespread application of biotechnology in the current era, while also providing greater space and opportunities for the depth and breadth of genomics and biomedical research.

[0003] HTGSD is typically stored in FastQ file format, with Quality Scores Data (QSD) accounting for up to 70% of the space in a FastQ lossless compressed file. This means that QSD plays a crucial role in improving the FastQ file compression ratio, i.e., the ratio of the file size before compression to the file size after compression. Therefore, how to compress and store large-scale high-throughput genome sequencing quality scores data to balance the speed of genomic big data generation, reduce the cost of basic storage infrastructure construction, and lower the overhead of data sharing and transmission is a pressing issue that needs to be addressed.

[0004] With the continuous development of high-throughput sequencing technology, traditional serial compression algorithms for high-throughput genome sequencing quality score data and simple CPU parallel compression algorithms are no longer able to meet the needs of rapid real-time processing of massive data. At the same time, existing lossless compression algorithms still have a lot of room for improvement in compression ratio. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a parallel compression method for high-throughput genome sequencing quality score data.

[0006] This invention provides a parallel compression method for high-throughput genome sequencing quality score data, comprising:

[0007] S100: Segment the original gene sequencing file and obtain the quality score data as a file to be compressed;

[0008] S200: Randomly sample the file to be compressed and perform k-mer analysis on the sampled data to obtain the statistical feature information of the file to be compressed, and establish a parallel sequence partitioning model based on the statistical feature information;

[0009] S300: The file to be compressed is classified into two categories by the parallel sequence partitioning model to obtain a first partition sub-file and a second partition sub-file. The first partition sub-file and the second partition sub-file are then concatenated according to the concatenation parameters to obtain a first concatenated sub-file and a second concatenated sub-file.

[0010] S400: The compression ratio gain is obtained by predicting the file to be compressed using multiple linear regression analysis. A parallel four-level run-length prediction mapping model is established based on the compression ratio gain. Data redundancy is eliminated on the first spliced ​​sub-file and the second spliced ​​sub-file using the parallel four-level run-length prediction mapping model, respectively, to obtain the first deredundant sub-file and the second deredundant sub-file.

[0011] S500: Context modeling is performed on the first deredundancy sub-file and the second deredundancy sub-file using a multi-core processor cluster, and cascade compression is performed using arithmetic coding to obtain the final compressed file.

[0012] A parallel compression method for high-throughput genome sequencing quality score data provided by the present invention further includes:

[0013] S600: The final compressed file is decompressed in parallel using a multi-core processor cluster to obtain a lossless final output file.

[0014] According to the present invention, a parallel compression method for high-throughput genome sequencing quality score data is provided, wherein the original genome sequencing file includes sequencing description information data, sequencing DNA sequence data, sequencing additional information data, and sequencing quality score data.

[0015] According to the parallel compression method for high-throughput genome sequencing quality score data provided by the present invention, the expression of the parallel sequence partitioning model in step S200 is as follows:

[0016]

[0017] in, Let I be the partitioning result label calculated by the p-th processor core for the i-th sequence, α be the partitioning threshold (I < α, where I is 0), n be the average sequence length, k be the sliding window size used for feature extraction in k-mer analysis, M be the number of samples randomly sampled from the file to be compressed, i be the first index value, j be the second index value, and i... ′ j is the third index value. ′The fourth index value is used by NFactor to compress the file.

[0018] A k-mer sequence of length k was collected at the j-th position of the i-th sequence, q i′,j′:j′+k+1 For the i-th ′ q random sampling sequences i′ At the jth ′ The k-mer sequence is calculated from each position.

[0019] According to the parallel compression method for high-throughput genome sequencing quality score data provided by the present invention, the expression for the compression ratio gain predicted by the multiple linear regression analysis method in step S400 is as follows:

[0020]

[0021] in, Let w0 be the compression ratio gain, w1 be the second weight parameter calculated by the multiple linear regression analysis prediction method, w2 be the third weight parameter calculated by the multiple linear regression analysis prediction method, w3 be the fourth weight parameter calculated by the multiple linear regression analysis prediction method, and w4 be the fifth weight parameter calculated by the multiple linear regression analysis prediction method. Let n be the mode character calculated by the p-th processor core for the i-th sequence, n be the average sequence length, N be the number of concatenated sequences, i be the first index value, j be the second index value, v be the fifth index value, and I be an indicator function. To represent the character at the j-th position in the i-th QSD sequence of the v-th concatenated partition file for the p-th processor core, This represents the character at position j-1 in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core. Let |Θ| be the run length calculated by the p-th processor core at the j-th position of the i-th sequence, and |Θ| be the size of the character value space for the quality score data. The size of the character value space of the k-mer sequence collected by the core of the p-th processor at the j-th position of the i-th sequence.

[0022] According to the parallel compression method for high-throughput genome sequencing quality score data provided by the present invention, the expression of the parallel four-level run-length prediction mapping model in step S400 is as follows:

[0023]

[0024]

[0025]

[0026] in, Let β be the mapping result label calculated by the p-th processor core for the j-th position of the i-th sequence in the v-th file, and let β be the dynamic run-length prediction switch. This represents the character at position j+1 in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core. This represents the character at the (j+2)th position in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core.

[0027] According to the present invention, a parallel compression method for high-throughput genome sequencing quality score data is provided, wherein step S500 includes:

[0028] S510: Divide the first de-redundancy sub-file and the second de-redundancy sub-file into multiple blocks to be compressed by using a data block partitioning strategy;

[0029] S520: Multiple blocks to be compressed are modeled in parallel context using a multi-core processor cluster and cascaded compressed to obtain multiple files to be merged;

[0030] S530: Merge and package multiple files to be merged to obtain the final compressed file.

[0031] According to the parallel compression method for high-throughput genome sequencing quality score data provided by the present invention, the range of values ​​for the splicing parameters in step S300 is as follows:

[0032] 10000≤τ≤r;

[0033] Where τ is the concatenation parameter and r is the total number of string sequences in the file to be compressed.

[0034] This invention provides a parallel compression method for high-throughput genome sequencing quality score data. The overall goal is to optimize and improve the compression rate of large-scale high-throughput genome sequencing quality score data, reduce compression and decompression time, and minimize peak memory overhead. Based on the parallel sequence partitioning model, parallel four-level run-length prediction mapping model, and multi-core CPU cluster of this invention, a parallel compression method for large-scale high-throughput genome sequencing quality score data is implemented, improving the compression and storage efficiency of genome sequencing data and having significant value in industrial and academic applications.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a parallel compression method for high-throughput genome sequencing quality score data provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0039] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0041] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0042] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0043] The following is combined with Figure 1 The embodiments provided by the present invention are described below.

[0044] This invention provides a parallel compression method for high-throughput genome sequencing quality score data, comprising:

[0045] S100: Segment the original gene sequencing file and obtain the quality score data as a file to be compressed;

[0046] The original gene sequencing file includes sequencing description information data, sequencing DNA sequence data, sequencing additional information data, and sequencing quality score data.

[0047] Furthermore, the original gene sequencing file is in FastQ format. In step S100 above, the sequencing quality score data file to be compressed is obtained through data segmentation for subsequent compression and storage operations.

[0048] S200: Randomly sample the file to be compressed and perform k-mer analysis on the sampled data to obtain the statistical feature information of the file to be compressed, and establish a parallel sequence partitioning model based on the statistical feature information;

[0049] The expression for the parallel sequence partitioning model in step S200 is as follows:

[0050]

[0051] in, Let I be the partitioning result label calculated by the p-th processor core for the i-th sequence, α be the partitioning threshold (I < α, where I is 0), n be the average sequence length, k be the sliding window size used for feature extraction in k-mer analysis, M be the number of samples randomly sampled from the file to be compressed, i be the first index value, j be the second index value, and i... ′ j is the third index value. ′ The fourth index value is NFactor, which is the k-mer frequency normalization factor used to randomly sample the file to be compressed. For the p-th processor core, a k-mer sequence of length k is collected at the j-th position of the i-th sequence, q i′,j′:j′+k+1 For the i-th ′ q random sampling sequences i′ At the jth ′ The k-mer sequence is calculated from each position.

[0052] S300: The file to be compressed is classified into two categories by the parallel sequence partitioning model to obtain a first partition sub-file and a second partition sub-file. The first partition sub-file and the second partition sub-file are then concatenated according to the concatenation parameters to obtain a first concatenated sub-file and a second concatenated sub-file.

[0053] The range of values ​​for the splicing parameters in step S300 is as follows:

[0054] 10000≤τ≤r;

[0055] Where τ is the concatenation parameter and r is the total number of string sequences in the file to be compressed.

[0056] S400: The compression ratio gain is obtained by predicting the file to be compressed using multiple linear regression analysis. A parallel four-level run-length prediction mapping model is established based on the compression ratio gain. Data redundancy is eliminated on the first spliced ​​sub-file and the second spliced ​​sub-file using the parallel four-level run-length prediction mapping model, respectively, to obtain the first deredundant sub-file and the second deredundant sub-file.

[0057] The expression for the compression ratio gain predicted by the multiple linear regression analysis method in step S400 is as follows:

[0058]

[0059] in, Let w0 be the compression ratio gain, w1 be the second weight parameter calculated by the multiple linear regression analysis prediction method, w2 be the third weight parameter calculated by the multiple linear regression analysis prediction method, w3 be the fourth weight parameter calculated by the multiple linear regression analysis prediction method, and w4 be the fifth weight parameter calculated by the multiple linear regression analysis prediction method. Let n be the mode character calculated by the p-th processor core for the i-th sequence, n be the average sequence length, N be the number of concatenated sequences, i be the first index value, j be the second index value, v be the fifth index value, and I be an indicator function. To represent the character at the j-th position in the i-th QSD sequence of the v-th concatenated partition file for the p-th processor core, This represents the character at position j-1 in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core. Let |Θ| be the run length calculated by the p-th processor core at the j-th position of the i-th sequence, and |Θ| be the size of the character value space for the quality score data. The size of the character value space of the k-mer sequence collected by the core of the p-th processor at the j-th position of the i-th sequence.

[0060] Furthermore, the purpose of multiple linear regression prediction is to solve for the optimal weight vector of the parallel four-level run-length prediction mapping model. In the above formula, the parameter multiplied by the first weight is the proportion of the mode character to the total sequence characters, the parameter multiplied by the second weight parameter is the proportion of adjacent characters with a quality value greater than 7 to the total string sequence characters, the parameter multiplied by the third weight parameter is the proportion of characters with a run length greater than 3 to the total sequence characters, and the parameter multiplied by the fourth weight parameter is the proportion of the space in which the current sequence character is taken to the space in which the source character value is taken.

[0061] The expression for the parallel four-level run-length prediction mapping model in step S400 is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] in, Let β be the mapping result label calculated by the p-th processor core for the j-th position of the i-th sequence in the v-th file, and let β be the dynamic run-length prediction switch. This represents the character at position j+1 in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core. This represents the character at the (j+2)th position in the i-th QSD sequence of the v-th concatenated partition file, as computed by the p-th processor core.

[0067] Furthermore, the four parameters multiplied by the weight parameters in the expression for the compression ratio gain obtained by the multiple linear regression analysis prediction method are denoted as the sequence feature vector. By inputting the sequence feature vector, a parallel four-level run prediction mapping model can be established.

[0068] S500: Context modeling is performed on the first deredundancy sub-file and the second deredundancy sub-file using a multi-core processor cluster, and cascade compression is performed using arithmetic coding to obtain the final compressed file.

[0069] Step S500 includes:

[0070] S510: Divide the first de-redundancy sub-file and the second de-redundancy sub-file into multiple blocks to be compressed by using a data block partitioning strategy;

[0071] S520: Multiple blocks to be compressed are modeled in parallel context using a multi-core processor cluster and cascaded compressed to obtain multiple files to be merged;

[0072] S530: Merge and package multiple files to be merged to obtain the final compressed file.

[0073] This also includes:

[0074] S600: The final compressed file is decompressed in parallel using a multi-core processor cluster to obtain a lossless final output file.

[0075] Furthermore, the ZPAQ algorithm is used for both cascade compression and cascade decompression. The final compressed file is named Q.save, and the output decompressed lossless recovery file is named Q.recover.

[0076] In some embodiments, this invention uses 11 sets of real open-source data from the NCBI (National Center for Biotechnology Information) database for evaluation and verification. The total file size is 72,618,889 kilobytes, containing 219,959,565 quality score sequence data points and a total of 2,036,301,8799 quality score characters. Experimental testing evaluated the compression performance of this invention's QSD parallel compression algorithm and four benchmark methods—ZPAQ, Qscomp, LCQS, and CMIC—running on the 11 datasets using a single node and 28 CPU cores.

[0077] Experimental results were collected for normalized peak memory overhead and normalized time overhead. The lower the algorithm time and peak memory overhead, the better the algorithm performance. The experimental results show that the compression method provided by this invention achieves the best overall performance in four out of five indicators: compression ratio, compression time, compression peak memory overhead, decompression time and decompression peak memory overhead. The results indicate that the parallel compression method for large-scale high-throughput genome sequencing quality score data provided by this invention is effective in optimizing and improving the compression ratio of large-scale QSD, reducing compression and decompression time and peak memory overhead.

[0078] In some embodiments, the present invention employs multi-core CPU to accelerate sequence partitioning and four-level run-length prediction mapping calculation, and uses multi-core CPU cluster to accelerate ZPAQ for context modeling combined with arithmetic coding concatenation compression. The parallel speedup ratio of the present invention is evaluated and verified in the cases of single-node different cores and 4-core multi-node.

[0079] The experimental results collected the parallel speedup ratio, which is the ratio of parallel computing time overhead to serial computing time overhead; another parallel speedup ratio is the ratio of multi-node computing time overhead to single-node computing time overhead. The experimental results show that the parallel compression method for large-scale high-throughput genome sequencing quality score data provided by this invention exhibits an overall increasing parallel speedup ratio for compression and decompression as the data scale increases and the number of CPU cores increases. This indicates that the compression method of this invention has a significant multi-core CPU parallel acceleration effect on a single node. Further experimental results show that the parallel compression method of this invention is significantly accelerated by multi-core CPU clusters, especially the decompression parallel acceleration effect of multi-core CPU clusters.

[0080] This invention provides a parallel compression method for high-throughput genome sequencing quality score data. The overall goal is to optimize and improve QSD compression ratio, reduce time and peak memory consumption. Under the premise of significantly reducing QSD compression time and peak memory overhead, the method improves QSD compression ratio, reduces the size of the files to be compressed and stored, and saves the construction cost of basic storage facilities.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A parallel compression method for high-throughput genome sequencing quality score data, characterized in that, include: S100: Segment the original gene sequencing file and obtain the quality score data as a file to be compressed; S200: Randomly sample the file to be compressed and perform k-mer analysis on the sampled data to obtain statistical feature information of the file to be compressed. Establish a parallel sequence partitioning model based on the statistical feature information. The expression of the parallel sequence partitioning model in step S200 is: in, The first index value, The second index value, The third index value, The fourth index value, For the first The processor computing core is for the first The partitioning results obtained from the calculation of the sequence are labeled. For indicator functions, As the partition threshold, when hour The value is 0. The average sequence length, This refers to the sliding window size used for feature extraction in k-mer analysis. This represents the number of samples used to randomly sample the files to be compressed. This is the k-mer frequency normalization factor used for randomly sampling the file to be compressed. For the first The processor computing cores are in the... The first of the sequence The length collected at each location is k-mer sequences, For the first random sampling sequence In the The k-mer sequence obtained by calculating at each position; S300: The file to be compressed is classified into two categories by the parallel sequence partitioning model to obtain a first partition sub-file and a second partition sub-file. The first partition sub-file and the second partition sub-file are then concatenated according to the concatenation parameters to obtain a first concatenated sub-file and a second concatenated sub-file. S400: The compression ratio gain of the file to be compressed is predicted using multiple linear regression analysis. A parallel four-level run-length prediction mapping model is established based on the compression ratio gain. Data redundancy is then performed on the first and second concatenated sub-files using the parallel four-level run-length prediction mapping model, resulting in a first de-redundant sub-file and a second de-redundant sub-file. The expression for the compression ratio gain predicted by multiple linear regression analysis in step S400 is: in, For compression ratio gain, The first weight parameter is obtained by calculating the prediction method using multiple linear regression analysis. The second weight parameter is obtained by calculating the prediction method using multiple linear regression analysis. The third weight parameter is obtained by calculating the prediction method using multiple linear regression analysis. The fourth weight parameter is obtained by calculating the prediction method using multiple linear regression analysis. The fifth weight parameter is obtained by calculating the prediction method using multiple linear regression analysis. For the first The processor computing core is for the first The mode character obtained by calculating the sequence, The average sequence length, The number of spliced ​​sequences, The first index value, The second index value, The fifth index value, For indicator functions, To indicate the first The processor computing core is for the first The first of the concatenated partition files The first QSD sequence The character corresponding to each position Indicates the first The processor computing core is for the first The first of the concatenated partition files The first QSD sequence The character corresponding to each position For the first The processor computing cores are in the... The first of the sequence The run length calculated from each position The size of the character value space for the quality score data. For the first The processor computing cores are in the... The first of the sequence The size of the character value space of the k-mer sequence collected at each location; S500: Context modeling is performed on the first deredundancy sub-file and the second deredundancy sub-file using a multi-core processor cluster, and cascade compression is performed using arithmetic coding to obtain the final compressed file.

2. The parallel compression method for high-throughput genome sequencing quality score data according to claim 1, characterized in that, Also includes: S600: The final compressed file is decompressed in parallel using a multi-core processor cluster to obtain a lossless final output file.

3. The parallel compression method for high-throughput genome sequencing quality score data according to claim 1, characterized in that, The original gene sequencing file includes sequencing description information data, sequencing DNA sequence data, sequencing additional information data, and sequencing quality score data.

4. The parallel compression method for high-throughput genome sequencing quality score data according to claim 1, characterized in that, The expression for the parallel four-level run-length prediction mapping model described in step S400 is: in, For the first The processor computing core is for the first The first file sequence of The mapping result of the location calculation is labeled. For dynamic run prediction switch, Indicates the first The processor computing core is for the first The first of the concatenated partition files The first QSD sequence The character corresponding to each position Indicates the first The processor computing core is for the first The first of the concatenated partition files The first QSD sequence The character corresponding to each position.

5. The parallel compression method for high-throughput genome sequencing quality score data according to claim 1, characterized in that, Step S500 includes: S510: Divide the first de-redundancy sub-file and the second de-redundancy sub-file into multiple blocks to be compressed by using a data block partitioning strategy; S520: Multiple blocks to be compressed are modeled in parallel context using a multi-core processor cluster and cascaded compressed to obtain multiple files to be merged; S530: Merge and package multiple files to be merged to obtain the final compressed file.

6. The parallel compression method for high-throughput genome sequencing quality score data according to claim 1, characterized in that, The range of values ​​for the splicing parameters mentioned in step S300 is as follows: in, For splicing parameters, This represents the total number of string sequences in the file to be compressed.

Images