Method and device for compressing and decompressing FASTQ file

Abstract

The invention provides a method and device for compressing and decompressing FASTQ files. The method comprises the following steps: converting four rows of information of the same read section in the FASTQ file into one row of information, sequentially arranging the obtained one row of information of different read sections, and outputting a new text file; one row of information of the same read segment in the new text file comprises first column read segment ID number information, second column read segment sequence information and third column ASCII code information in sequence from left to right; and compressing the new text file by using the gzip to complete the compression of the FASTQ file. According to the method, the four rows of information in the FASTQ file are subjected to information reduction conversion, particularly simplification of related read sequence information, gene sequencing information with a high compression ratio can be obtained, and convenience is provided for storage and transportation of gene sequencing data.

Description

Technical Field

[0001] The present invention relates to the technical field of gene sequencing, and in particular to a method and device for compressing and decompressing a FASTQ file. Background Art

[0002] The data downloaded from the gene sequencer is a FASTQ file. FASTQ files are a text format used to store biological sequences (usually nucleic acid sequences) and their corresponding quality evaluations. This information has a very strict format: every four lines form a group, and each group carries the information of a sequencing read. In each group of 4 lines of information, the first line is the ID of the read segment, which is the unique identifier of the read segment. The ID contains the chip number and the coordinate value of the read segment on the chip; the second line is the sequence of the read segment; the third line is a placeholder line. The old version of FASTQ will repeat the read segment ID of the first line, and the new version of FASTQ will only retain a + sign for placeholder; the fourth line corresponds to the quality value of each base of the read segment sequence in the second line, and each quality value is represented by an ASCII code. This format is widely used in the fields of bioinformatics and genomics, especially in the processing and analysis of high-throughput sequencing data, and therefore occupies a large amount of storage space.

[0003] Currently, the main compression software used for FASTQ files is gzip. In addition, there are bzip2 or tar packaging and compression methods. Among them, gzip is a widely used file compression program, which can achieve a compression ratio of about 50%-60% for FASTQ files; bzip2 is another common FASTQ file compression format, which is compressed using the bzip2 tool. The compression ratio can reach about 70%; tar packaging and compression is to use tar to package multiple FASTQ files into one file for easy storage and transportation, and further compress the packaged files.

[0004] However, these compression methods all have a common disadvantage, that is, there is no algorithm designed to fully compress FASTQ files based on the characteristics of FASTQ files, and the compression ratio still has a lot of room for improvement. Therefore, how to provide a method that can improve the compression ratio and remove redundant information in FASTQ files is of great significance for the storage and transportation of genetic data. Summary of the invention

[0005] The main purpose of the present invention is to provide a method and device for compressing and decompressing FASTQ files to solve the problem of low compression of FASTQ files in the prior art.

[0006] To achieve the above-mentioned purpose, according to a first aspect of the present invention, a method for compressing a FASTQ file is provided, the method comprising: converting 4 lines of information of the same read segment in the FASTQ file into 1 line of information, arranging the obtained 1 line of information of different read segments in sequence, and outputting them as a new text file; compressing the new text file using gzip to complete the compression of the FASTQ file; wherein the 4 lines of information of the same read segment in the FASTQ file are in the following order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; the 1 line of information of the same read segment in the new text file are in the following order from left to right: the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information; The conversion of 4 lines of information of the same read segment in the FASTQ file into 1 line of information includes: converting the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information respectively, wherein the conversion of the second line of read segment sequence information includes: S1, taking a single base among A, T, C, and G as the object, using 1 to represent the position of the single base in the read segment sequence, and using 0 to represent the positions of the remaining three bases in the read segment sequence, and converting the read segment sequence information into 4 single binary character strings; S2, using the single base represented by 1 as A, T, C, G in order to splice the corresponding 4 single binary character strings into 1 total binary character string; S3, using a second-order matrix to calculate the total binary character string to obtain the second column information in the converted 1 line of information.

[0007] Further, S3 includes: setting a second-order matrix Represents 1, a second-order matrix Represents 0, and the total binary string is multiplied by a second-order matrix to obtain the final second-order matrix; the 4 numbers in the final second-order matrix are connected with "_" in sequence, and output as the second column information in the converted 1-row information.

[0008] Further, converting the second line of read sequence information includes: deleting the "@" + chip number in the first line of read ID number information of the same read in the FASTQ file, and outputting the coordinates of the read on the chip as the first column of information in the converted line of information; preferably, converting the third line of placeholders includes: deleting the third line of placeholders "+" of the same read in the FASTQ file; preferably, converting the fourth line of ASCII code includes: outputting the fourth line of ASCII code information of the same read in the FASTQ file as the third column of information in the converted line of information.

[0009] In order to achieve the above-mentioned purpose, according to a second aspect of the present invention, a method for decompressing a FASTQ compressed file is provided, the decompression method comprising: decompressing a gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing the FASTQ file according to the above-mentioned method for compressing the FASTQ file; converting one line of information of the same read segment in the initial text file into four lines of read segment information, and outputting the obtained four lines of read segment information of different read segments as a FASTQ file; wherein the one line of information of the same read segment in the initial text file is in the order from left to right: the first column of read segment ID number information, the second column of read segment sequence information, and the third column of ASCII code information; the four lines of read segment information in the FASTQ file are in the order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; converting the one line of information of the same read segment in the initial text file into four lines of read segment information comprises: converting the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information.

[0010] Further, converting the second column of read sequence information includes: performing a second-order matrix multiplication calculation to restore the second column of read sequence information to obtain a total binary string; dividing the total binary string into 4 single binary strings on average, marking the 4 single binary strings from beginning to end in the order of A, T, C, and G, the position of 1 in the single binary string is the position of the marked base in the read sequence, integrating the respective positions of the bases A, T, C, and G to obtain the second row of read sequence information in the four-row group of read information in the FASTQ file; preferably, restoring the second column of read sequence information by performing a second-order matrix multiplication calculation includes: successively restoring the four numbers of the second column of read sequence information into a second-order matrix, and converting the four numbers in the second-order matrix into a second-order matrix. The numbers are divided into a left column and a right column, and the two numbers in the same row of the two columns are compared respectively, and the result of the comparison is converted into a total binary string, and the smaller number in the same row is subtracted from the larger number, and the smaller number is retained to obtain the next second-order matrix, and subtraction is performed in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, and then the restoration of the second-order matrix multiplication calculation is terminated; wherein, the conversion of the result of the comparison into a total binary string includes: if the two numbers in the left column are greater than the two numbers in the right column, a 0 is added to the leftmost side of the binary string, and if the two numbers in the left column are less than the two numbers in the right column, a 1 is added to the leftmost side of the binary string, and the second-order matrices before the restoration are converted into binary strings in sequence to obtain a total binary string.

[0011] Further, converting the first column of read ID number information includes: adding "@" + chip number before the first column of read ID number information, and outputting it as the first line of read ID number information in the four lines of read information in the FASTQ file; preferably, converting the third column of ASCII code information includes: outputting the third column of ASCII code information as the fourth line of ASCII code information in the four lines of read information in the FASTQ file; preferably, converting one line of information of the same read in the initial text file into four lines of read information includes: adding a third line of placeholder information, including: outputting "+" as the third line of placeholder information in the four lines of read information in the FASTQ file.

[0012] In order to achieve the above-mentioned purpose, according to a third aspect of the present invention, a device for compressing a FASTQ file is provided, the device comprising: a conversion module, configured to convert 4 lines of information of the same read segment in the FASTQ file into 1 line of information, and the obtained 1 line of information of different read segments is arranged in sequence and output as a new text file; a compression module, configured to compress the new text file using gzip to complete the compression of the FASTQ file; wherein the 4 lines of information of the same read segment in the FASTQ file are in the following order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; the 1 line of information of the same read segment in the new text file is in the following order from left to right: the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information; wherein the conversion module is configured to convert 4 lines of information of the same read segment in the FASTQ file into 1 line of information, and the obtained 1 line of information of different read segments is arranged in sequence and output as a new text file; and the compression module is configured to compress the new text file using gzip to complete the compression of the FASTQ file; wherein the 4 lines of information of the same read segment in the FASTQ file are in the following order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; and the 1 line of information of the same read segment in the new text file are in the following order from left to right: the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information. The conversion module includes: a first-line read ID number information conversion unit, a second-line read sequence information conversion unit, a third-line placeholder conversion unit and a fourth-line ASCII code information conversion unit; wherein the second-line read sequence information conversion unit includes: a single binary string conversion unit, which is configured to take a single base among A, T, C, and G as an object, 1 represents the position of the single base in the read sequence, and 0 represents the position of the remaining three bases in the read sequence, and converts the read sequence information into 4 single binary strings; a total binary string concatenation unit, which is configured to concatenate the corresponding 4 single binary strings into 1 total binary string in the order of A, T, C, and G represented by the single base represented by 1; and a calculation unit, which is configured to calculate the total binary string using a second-order matrix to obtain the second column information in the converted 1-line information.

[0013] Further, the calculation unit includes: a matrix calculation unit, configured to set a second-order matrix Represents 1, a second-order matrix represents 0, and the total binary string is multiplied by a second-order matrix to obtain a final second-order matrix; the second column information output unit is set to connect the 4 numbers in the final second-order matrix with "_" in sequence, and output them as the second column information in the converted 1-row information.

[0014] Further, the first line of read ID number information conversion unit is configured to delete the "@" + chip number in the first line of read ID number information of the same read segment in the FASTQ file, and output the coordinates of the read segment on the chip as the first column of information in the converted line of information; preferably, the third line of placeholder conversion unit is configured to delete the third line of placeholder "+" of the same read segment in the FASTQ file; preferably, the fourth line of ASCII code information conversion unit is configured to output the fourth line of ASCII code information of the same read segment in the FASTQ file as the third column of information in the converted line of information.

[0015] In order to achieve the above-mentioned purpose, according to a fourth aspect of the present invention, a decompression device for a FASTQ compressed file is provided, the decompression device comprising: a decompression module, configured to decompress a gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing the FASTQ file according to the above-mentioned method for compressing the FASTQ file; a FASTQ file output module, configured to convert one line of information of the same read segment in the initial text file into four lines of read segment information, and output the obtained four lines of read segment information of different read segments as a FASTQ file; wherein the one line of information of the same read segment in the initial text file is in the order from left to right: a first column of read segment ID number information, a second column of read segment sequence information, and a third column of ASCII code information; the four lines of read segment information in the FASTQ file are in the order from top to bottom: a first line of read segment ID number information, a second line of read segment sequence information, a third line of placeholders, and a fourth line of ASCII code information; the FASTQ file output module comprises: a first column of read segment ID number information conversion unit, a second column of read segment sequence information conversion unit, and a third column of ASCII code information conversion unit.

[0016] Further, the second column read sequence information conversion unit includes: a total binary string conversion unit, which is configured to restore the second column read sequence information by second-order matrix multiplication calculation to obtain a total binary string; a second row read sequence information integration unit, which is configured to evenly divide the total binary string into 4 single binary strings, mark the 4 single binary strings from beginning to end in the order of A, T, C, G, the position of 1 in the single binary string is the position of the marked base in the read sequence, and integrate the respective positions of the bases A, T, C, and G to obtain the second row of read sequence information in the four-row group of read information in the FASTQ file; preferably, the total binary string conversion unit includes: a second-order matrix conversion unit, which is configured to restore the four numbers of the second column read sequence information into one in sequence A second-order matrix is ​​obtained, and four numbers in the second-order matrix are divided into a left column and a right column. Two numbers in the same row of the two columns are compared respectively, and the result of the comparison is converted into a total binary string, and the smaller number in the same row is subtracted from the larger number, and the smaller number is retained to obtain the next second-order matrix, and subtraction is performed in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, and then the restoration of the second-order matrix multiplication calculation is terminated; the second-order matrix conversion unit includes: a total binary string generation unit, which is set to add a 0 to the leftmost side of the binary string if the two numbers in the left column are greater than the two numbers in the right column, and add a 1 to the leftmost side of the binary string if the two numbers in the left column are less than the two numbers in the right column, and the binary string conversion is performed on the second-order matrix before the restoration is terminated in sequence to obtain a total binary string.

[0017] Furthermore, the first column read ID number information conversion unit is configured to add "@" + chip number before the first column read ID number information, and output it as the first line of read ID number information in the four lines of read information in the FASTQ file; preferably, the third column ASCII code information conversion unit is configured to output the third column ASCII code information as the fourth line of ASCII code information in the four lines of read information in the FASTQ file; preferably, the FASTQ file output module also includes: an information adding unit, which is configured to output "+" as the third line of placeholder information in the four lines of read information in the FASTQ file.

[0018] In order to achieve the above-mentioned purpose, according to the fifth aspect of the present invention, a computer-readable storage medium is provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned method or the above-mentioned decompression method.

[0019] In order to achieve the above-mentioned object, according to a sixth aspect of the present invention, a processor is provided, and the processor is used to run a program, wherein the program runs the above-mentioned method or the above-mentioned decompression method.

[0020] By applying the technical solution of the present invention, the four lines of information in the FASTQ file are transformed to reduce the information, especially the simplification of the read sequence information, so that highly compressed gene sequencing information can be obtained, which facilitates the storage and transportation of gene sequencing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 The figure shows the compression ratio results of FASTQ file compression using the compression method of the present application, gzip software and bzip2 software. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Terminology explanation:

[0025] gzip software: Created by Jean-loup Gailly and Mark Adler, it is a file compression software used for UN Ⅸ system. After compression, the compressed file is in .gz format.

[0026] Compression ratio: refers to the ratio of data volume reduction by encoding data through compression algorithms during data storage or transmission. The higher the compression ratio, the smaller the amount of compressed data is relative to the original data, thus saving storage space and transmission time. When decompressing, the same decompression algorithm can be used to restore the same data content as the original file.

[0027] As mentioned in the background technology, the compression obtained by the method for compressing FASTQ files in the prior art is relatively low, and it is not a specific method for compressing according to the characteristics of the FASTQ file format. Therefore, the present application intends to provide a compression method that can improve the compression ratio of FASTQ files, thereby facilitating the storage and transportation of gene sequencing information.

[0028] In a first typical embodiment of the present application, a method for compressing a FASTQ file is provided, the method comprising: converting 4 lines of information of the same read segment in the FASTQ file into 1 line of information, arranging the obtained 1 line of information of different read segments in sequence, and outputting the same line of information into a new text file; compressing the new text file using gzip to complete the compression of the FASTQ file; wherein the 4 lines of information of the same read segment in the FASTQ file are in the following order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders, and the fourth line of ASCII code information; the 1 line of information of the same read segment in the new text file is in the following order from left to right: the first column of read segment ID number information, the second column of read segment sequence information, and the third column of ASCII code information; The conversion of 4 lines of information of the same read segment in the ASTQ file into 1 line of information includes: converting the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information, wherein the conversion of the second line of read segment sequence information includes: S1, taking a single base among A, T, C, and G as the object, using 1 to represent the position of the single base in the read segment sequence, and using 0 to represent the positions of the remaining three bases in the read segment sequence, and converting the read segment sequence information into 4 single binary character strings; S2, using the single base represented by 1 as A, T, C, G in order to concatenate the corresponding 4 single binary character strings into 1 total binary character string; S3, using a second-order matrix to calculate the total binary character string to obtain the second column information in the converted 1 line of information.

[0029] By using the above-mentioned FASTQ file compression method, especially the binary simplification processing of the read sequence information, a new text file in the .USCI.FQ format is obtained, which can increase the reading speed of the simplified data, reduce data redundancy, and improve storage efficiency. It is particularly suitable for the storage and transmission of large-scale genomic data, such as in genomic research, disease diagnosis and personalized medicine.

[0030] The binary string is further compressed into a more compact digital string by mathematical operations. In a preferred embodiment, S3 includes: setting a second-order matrix Represents 1, a second-order matrix Represents 0, and the total binary string is multiplied by the second-order matrix to obtain the final second-order matrix; the 4 numbers in the final second-order matrix are connected in sequence with "_" and output as the second column information in the converted 1-row information. Among them, the 4 numbers in the final second-order matrix are connected in sequence in the order of the positions of a, b, c, and d in the matrix to obtain the second column information a_b_c_d.

[0031] Since hundreds of reads in the same chip all carry identification information of the same chip, there is a large amount of information redundancy. In a preferred embodiment, converting the second line of read sequence information includes: deleting the "@" + chip number in the first line of read ID information of the same read in the FASTQ file, and outputting the coordinates of the read on the chip as the first column of information in the converted line of information;

[0032] Since placeholders are often used for standardization of file formats and do not contain substantial sequencing data, in a preferred embodiment, converting the third row of placeholders includes: deleting the third row of placeholders "+" for the same read segment in the FASTQ file; since the sequencing quality score information of the read segment is important for subsequent genetic research, in a preferred embodiment, converting the fourth row of ASCII codes includes: outputting the fourth row of ASCII code information for the same read segment in the FASTQ file as the third column of information in the converted 1 row of information.

[0033] In a second typical embodiment of the present application, a method for decompressing a FASTQ compressed file is provided, the decompression method comprising: decompressing a gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing the FASTQ file according to the above-mentioned method for compressing the FASTQ file; converting one line of information of the same read segment in the initial text file into four lines of read segment information, and the obtained four lines of read segment information of different read segments are output as a FASTQ file; wherein the one line of information of the same read segment in the initial text file is in the order from left to right: the first column of read segment ID number information, the second column of read segment sequence information, and the third column of ASCII code information; the four lines of read segment information in the FASTQ file are in the order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; converting the one line of information of the same read segment in the initial text file into four lines of read segment information comprises: converting the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information.

[0034] The above-mentioned decompression method is used to decompress the compressed files of this application. The operation is simple and can greatly improve the speed of storage and transportation processing of large-scale data sets. On the basis of ensuring data integrity and accuracy, it realizes the restoration of highly compressed data and subsequent genetic analysis.

[0035] In a preferred embodiment, transforming the second column of read sequence information includes: restoring the second column of read sequence information by a second-order matrix multiplication calculation to obtain a total binary string; dividing the total binary string into 4 single binary strings on average, marking the 4 single binary strings from beginning to end in the order of A, T, C, and G, the position of 1 in the single binary string is the position of the marked base in the read sequence, integrating the respective positions of the bases A, T, C, and G to obtain the second line of read sequence information in the four-line read information in the FASTQ file.

[0036] The digital string compressed by matrix calculation is restored to a binary string. In a preferred embodiment, the restoration of the second column read sequence information by second-order matrix multiplication calculation includes: restoring the four numbers of the second column read sequence information into a second-order matrix in sequence, dividing the four numbers in the second-order matrix into a left column and a right column, comparing the two numbers in the same row in the two columns respectively, converting the result of the comparison into a total binary string, and subtracting the smaller number in the same row from the larger number, retaining the smaller number, to obtain the next second-order matrix, and performing subtraction in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, then the restoration of the second-order matrix multiplication calculation is terminated; wherein, the four numbers of the second column read sequence information are used as the numbers of positions a, b, c, and d in the matrix in order from left to right to restore the second-order matrix.

[0037] Among them, converting the result of the size comparison into a total binary string includes: if the two numbers in the left column are greater than the two numbers in the right column, then adding a 0 to the leftmost side of the binary string; if the two numbers in the left column are less than the two numbers in the right column, then adding a 1 to the leftmost side of the binary string, and converting the binary string of the second-order matrix before the end of the restoration in turn to obtain a total binary string.

[0038] In order to further restore the original chip identification of the read segment, in a preferred embodiment, converting the first column of read segment ID number information includes: adding "@" + chip number before the first column of read segment ID number information, and outputting it as the first line of read segment ID number information in the four-line read segment information in the FASTQ file; in a preferred embodiment, converting the third column of ASCII code information includes: outputting the third column of ASCII code information as the fourth line of ASCII code information in the four-line read segment information in the FASTQ file; in order to restore the standardized format of the FASTQ file, in a preferred embodiment, converting one line of information of the same read segment in the initial text file into four lines of read segment information includes: adding a third line of placeholder information, including: outputting "+" as the third line of placeholder information in the four-line read segment information in the FASTQ file.

[0039] In a third typical embodiment of the present application, a device for compressing a FASTQ file is provided, the device comprising: a conversion module, configured to convert 4 lines of information of the same read segment in the FASTQ file into 1 line of information, and the obtained 1 line of information of different read segments is arranged in sequence and output as a new text file; a compression module, configured to compress the new text file using gzip to complete the compression of the FASTQ file; wherein the 4 lines of information of the same read segment in the FASTQ file are in the following order from top to bottom: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; the 1 line of information of the same read segment in the new text file is in the following order from left to right: the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information; wherein the conversion module The block includes: a first row of read segment ID number information conversion unit, a second row of read segment sequence information conversion unit, a third row of placeholder conversion unit and a fourth row of ASCII code information conversion unit; wherein the second row of read segment sequence information conversion unit includes: a single binary string conversion unit, which is set to take a single base among A, T, C, and G as the object, 1 represents the position of the single base in the read segment sequence, and 0 represents the position of the remaining three bases in the read segment sequence, and converts the read segment sequence information into 4 single binary strings; a total binary string concatenation unit, which is set to concatenate the corresponding 4 single binary strings into 1 total binary string in the order of A, T, C, and G represented by the single base represented by 1; a calculation unit, which is set to calculate the total binary string using a second-order matrix to obtain the second column information in the converted 1 row of information.

[0040] Utilizing the above-mentioned FASTQ file compression device, especially the binary simplification processing of the read sequence information in the single binary string conversion unit, the reading speed of the simplified data can be increased, the redundancy of the data can be reduced, and the storage efficiency can be improved. It is particularly suitable for the storage and transmission of large-scale genomic data, such as in genomic research, disease diagnosis and personalized medicine.

[0041] The binary string is further compressed into a more compact digital string by mathematical operations. In a preferred embodiment, the calculation unit includes: a matrix calculation unit, which is configured to set a second-order matrix Represents 1, a second-order matrix represents 0, and the total binary string is multiplied by the second-order matrix to obtain the final second-order matrix; the second column information output unit is configured to connect the 4 numbers in the final second-order matrix with "_" in sequence, and output as the second column information in the converted 1-row information. Among them, the 4 numbers in the final second-order matrix are connected in sequence in the order of the positions of a, b, c, and d in the matrix.

[0042] Since hundreds of reads in the same chip carry identification information of the same chip, there is a large amount of information redundancy. In a preferred embodiment, the first-line read ID number information conversion unit is configured to delete the "@" + chip number in the first-line read ID number information of the same read in the FASTQ file, and output the coordinates of the read on the chip as the first column of information in the converted line of information; since placeholders are often used for standardization of file formats and do not contain substantial sequencing data, in a preferred embodiment, the third-line placeholder conversion unit is configured to delete the third-line placeholder "+" of the same read in the FASTQ file; since the sequencing quality score information of the read is more important for subsequent genetic research, in a preferred embodiment, the fourth-line ASCII code information conversion unit is configured to output the fourth-line ASCII code information of the same read in the FASTQ file as the third column of information in the converted line of information.

[0043] In a fourth typical embodiment of the present application, a decompression device for a FASTQ compressed file is provided, the decompression device comprising: a decompression module, configured to decompress a gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing the FASTQ file according to the above-mentioned method for compressing the FASTQ file; a FASTQ file output module, configured to convert one line of information of the same read segment in the initial text file into four lines of read segment information, and output the obtained four lines of read segment information of different read segments as a FASTQ file; wherein the one line of information of the same read segment in the initial text file is in the order from left to right: a first column of read segment ID number information, a second column of read segment sequence information, and a third column of ASCII code information; the four lines of read segment information in the FASTQ file are in the order from top to bottom: a first line of read segment ID number information, a second line of read segment sequence information, a third line of placeholders, and a fourth line of ASCII code information; the FASTQ file output module comprises: a first column of read segment ID number information conversion unit, a second column of read segment sequence information conversion unit, and a third column of ASCII code information conversion unit.

[0044] The above-mentioned decompression device is used to decompress the compressed files of the present application. The operation is simple and can greatly improve the speed of storage and transportation processing of large-scale data sets. On the basis of ensuring data integrity and accuracy, it realizes the restoration of highly compressed data and subsequent genetic analysis.

[0045] In a preferred embodiment, the second column read sequence information conversion unit includes: a total binary string conversion unit, which is configured to restore the second column read sequence information by performing a second-order matrix multiplication calculation to obtain a total binary string; a second row read sequence information integration unit, which is configured to divide the total binary string into 4 single binary strings on average, mark the 4 single binary strings from beginning to end in the order of A, T, C, G, the position of 1 in the single binary string is the position of the marked base in the read sequence, and integrate the respective positions of the bases A, T, C, and G to obtain the second row of read sequence information in the four-row read information in the FASTQ file.

[0046] The digital string compressed by matrix calculation is restored to a binary string. In a preferred embodiment, the total binary string conversion unit includes: a second-order matrix conversion unit, which is configured to restore the four numbers of the second column of read sequence information to a second-order matrix in sequence, divide the four numbers in the second-order matrix into a left column and a right column, compare the two numbers in the same row in the two columns respectively, convert the result of the comparison into a total binary string, and subtract the smaller number in the same row from the larger number, and retain the smaller number to obtain the next second-order matrix, and perform subtraction in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, then the restoration of the second-order matrix multiplication calculation is terminated;

[0047] The second-order matrix conversion unit includes: a total binary string generation unit, which is set to add a 0 to the leftmost side of the binary string if the two numbers in the left column are greater than the two numbers in the right column, and add a 1 to the leftmost side of the binary string if the two numbers in the left column are less than the two numbers in the right column, and perform binary string conversion on the second-order matrix before the restoration is completed in sequence to obtain a total binary string.

[0048] In order to further restore the original chip identification of the read segment, in a preferred embodiment, the first column read segment ID number information conversion unit is configured to add "@" + chip number before the first column read segment ID number information, and output it as the first line of read segment ID number information in the four-line read segment information in the FASTQ file; in order to restore the standardized format of the FASTQ file, in a preferred embodiment, the FASTQ file output module also includes: an information addition unit, which is configured to output "+" as the third line of placeholder information in the four-line read segment information in the FASTQ file; the original sequencing quality data is directly restored and output, and in a preferred embodiment, the third column ASCII code information conversion unit is configured to output the third column ASCII code information as the fourth line of ASCII code information in the four-line read segment information in the FASTQ file.

[0049] In a fifth typical implementation of the present application, a computer-readable storage medium is provided, the storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above method or the above decompression method.

[0050] In a sixth typical implementation of the present application, a processor is provided, and the processor is used to run a program, wherein the program runs the above method or the above decompression method.

[0051] The present application is further described in detail below in conjunction with specific embodiments. These embodiments should not be construed as limiting the scope of protection claimed in the present application.

[0052] Example 1

[0053] The sequencing information of one of the reads in the compressed FASTQ file is as follows:

[0054] First line: @S250092100L1C001R00100000172 / 1

[0055] Second row: ATCAGC

[0056] Third row: +

[0057] Fourth row: EEEBDB

[0058] The above four lines of information are converted according to the present application to obtain the following converted information in sequence:

[0059] 1. Delete the "@" and chip number in the first row above to get the first column of information after conversion: L1C001R00100000172 / 1;

[0060] 2. Calculate the second-order matrix of the read sequence in the second row and convert it into the second column information. The specific steps are as follows:

[0061] Sequence ATCAGC, convert this sequence into 4 single binary strings according to the positions of single bases A, T, C, and G:

[0062] A: 100100

[0063] T: 010000

[0064] C:001001

[0065] G:000010

[0066] Concatenate the four single binary strings into one total binary string in the order of single bases A, T, C, G:

[0067] 100100010000001001000010,

[0068] Using the matrix Represents 1, using the matrix Represents 0, convert the total binary string mentioned above into a second-order matrix, and perform second-order matrix multiplication to obtain a final second-order matrix. Connect the four numbers in the final second-order matrix with “_” in order of their positions to obtain the second column of information: 6007_3290_4424_2423.

[0069] 3. Delete the placeholder in the third line above.

[0070] 4. Keep the ASCII code information in the fourth row as the third column information: EEEBDB.

[0071] In summary, we get one row and three columns of information after conversion: L1C001R00100000172 / 1 6007_3290_4424_2423

[0072] EEEBDB, and then continue to use gzip software to compress the converted information.

[0073] Detection experiment

[0074] 50 FASTQ files were compressed using gzip software, bzip2 software, and the method in Example 1 of the present application, respectively. After compression, the compression ratios of different software were calculated, such as Figure 1 As shown, it can be seen that the compression ratio obtained by the compression method of the present application is about 90%, which is greatly improved compared with the compression effect of gzip software and bzip2 software.

[0075] From the above description, it can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: by performing information reduction conversion on the 4 lines of information in the FASTQ file, especially by simplifying the read sequence information, highly compressed gene sequencing information can be obtained, which facilitates the storage and transportation of gene sequencing data.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for compressing a FASTQ file, characterized in that: The method comprises: Converting the four lines of information of the same read segment in the FASTQ file into one line of information, arranging the obtained one line of information of different read segments in sequence, and outputting them into a new text file; Compressing the new text file using gzip to complete the compression of the FASTQ file; The order of the four lines of information of the same read segment in the FASTQ file from top to bottom is: the first line of read segment ID number information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; The information of one line of the same read segment in the new text file is as follows from left to right: the first column is the read segment ID number information, the second column is the read segment sequence information, and the third column is the ASCII code information; Converting the four lines of information of the same read segment in the FASTQ file into one line of information comprises: converting the first line of read segment ID information, the second line of read segment sequence information, the third line of placeholders, and the fourth line of ASCII code information, respectively; The converting of the second line of read sequence information includes: S1, taking a single base among A, T, C, and G as the object, using 1 to represent the position of the single base in the read sequence, and using 0 to represent the positions of the other three bases in the read sequence, converting the read sequence information into 4 single binary strings; S2, concatenating the corresponding four single binary character strings into one total binary character string in the order of A, T, C, and G represented by the single base 1; S3, using a second-order matrix to calculate the total binary string to obtain the second column information in the converted 1 row of information.

2. The method according to claim 1, characterized in that The S3 includes: Set up a second-order matrix Represents 1, a second-order matrix represents 0, performing a second-order matrix multiplication calculation on the total binary string to obtain a final second-order matrix; The four numbers in the final second-order matrix are connected in sequence with "_" and output as the second column information in the converted 1 row of information.

3. The method according to claim 1, characterized in that Converting the second line of read sequence information includes: The "@" + chip number in the first row of the read ID number information of the same read in the FASTQ file is deleted, and the coordinates of the read on the chip are output as the first column of the converted 1 row of information; Preferably, converting the third row placeholder comprises: deleting the third row placeholder "+" of the same read segment in the FASTQ file; Preferably, converting the fourth line of ASCII code includes: outputting the fourth line of ASCII code information of the same read segment in the FASTQ file as the third column of information in the converted 1 line of information.

4. A method for decompressing a FASTQ compressed file, characterized in that: The decompression method comprises: Decompressing the gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing the FASTQ file according to the method for compressing a FASTQ file according to any one of claims 1 to 3; Converting one line of information of the same read segment in the initial text file into four lines of read segment information, and outputting the obtained four lines of read segment information of different read segments as the FASTQ file; The information of one line of the same read segment in the initial text file is as follows from left to right: the first column is the read segment ID number information, the second column is the read segment sequence information, and the third column is the ASCII code information; The order of the four lines of read information in the FASTQ file from top to bottom is: the first line of read ID information, the second line of read sequence information, the third line of placeholders and the fourth line of ASCII code information; Converting one line of information of the same read segment in the initial text file into four lines of read segment information includes: converting the first column of read segment ID number information, the second column of read segment sequence information and the third column of ASCII code information.

5. The decompression method according to claim 4, characterized in that: Converting the second column of read sequence information includes: Performing a second-order matrix multiplication calculation on the read sequence information of the second column to restore the sequence information, and obtaining a total binary string; The total binary string is evenly divided into 4 single binary strings, and the 4 single binary strings are marked from beginning to end in the order of A, T, C, and G, and the position of 1 in the single binary string is the position of the marked base in the read sequence, and the positions of the bases A, T, C, and G are integrated to obtain the second line of read sequence information in the four-line group of read information in the FASTQ file; Preferably, performing a second-order matrix multiplication calculation to restore the second column of read sequence information comprises: The four numbers of the second column of read sequence information are restored to a second-order matrix in sequence, and the four numbers in the second-order matrix are divided into a left column and a right column, and the two numbers in the same row in the two columns are compared in size respectively, and the result of the comparison is converted into a total binary string, and the smaller number in the same row is subtracted from the larger number, and the smaller number is retained to obtain the next second-order matrix, and subtraction is performed in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, and then the restoration of the second-order matrix multiplication calculation is terminated; The step of converting the result of the size comparison into a total binary string includes: If the two numbers in the left column are greater than the two numbers in the right column, add a 0 to the leftmost side of the binary string. If the two numbers in the left column are less than the two numbers in the right column, add a 1 to the leftmost side of the binary string. The binary character string is converted sequentially on the second-order matrix before the restoration is completed to obtain one total binary character string.

6. The decompression method according to claim 4, characterized in that: Converting the first column of read segment ID number information includes: Add "@" + chip number before the first column of read ID information, and output it as the first line of read ID information in the four-line group of read information in the FASTQ file; Preferably, converting the third column of ASCII code information comprises: outputting the third column of ASCII code information as the fourth line of ASCII code information in the four-line group of read segment information in the FASTQ file; Preferably, converting one line of information of the same read segment in the initial text file into four lines of read segment information includes: adding a third line of placeholder information, including: outputting "+" as the third line of placeholder information in the four lines of read segment information in the FASTQ file.

7. A device for compressing a FASTQ file, characterized in that: The device comprises: A conversion module is configured to convert four lines of information of the same read segment in the FASTQ file into one line of information, and the obtained one line of information of different read segments is arranged in sequence and output as a new text file; A compression module is configured to compress the new text file using gzip to complete the compression of the FASTQ file; The order of the four lines of information of the same read segment in the FASTQ file from top to bottom is: the first line of read segment ID information, the second line of read segment sequence information, the third line of placeholders and the fourth line of ASCII code information; The information of one line of the same read segment in the new text file is as follows from left to right: the first column is the read segment ID number information, the second column is the read segment sequence information, and the third column is the ASCII code information; The conversion module includes: a first line read segment ID number information conversion unit, a second line read segment sequence information conversion unit, a third line placeholder conversion unit and a fourth line ASCII code information conversion unit; Wherein, the second row read sequence information conversion unit includes: A single binary string conversion unit is configured to take a single base among A, T, C, and G as an object, use 1 to represent the position of the single base in the read sequence, and use 0 to represent the positions of the other three bases in the read sequence, and convert the read sequence information into four single binary strings; The total binary string concatenation unit is configured to concatenate the corresponding four single binary strings into one total binary string in the order of A, T, C, and G represented by the single bases 1; The calculation unit is configured to calculate the total binary string using a second-order matrix to obtain the second column information in the converted 1 row of information.

8. The device according to claim 7, characterized in that The computing unit comprises: The matrix calculation unit is set to set the second-order matrix Represents 1, a second-order matrix represents 0, performing a second-order matrix multiplication calculation on the total binary string to obtain a final second-order matrix; The second column information output unit is configured to connect the four numbers in the final second-order matrix with "_" in sequence and output them as the second column information in the converted 1 row information.

9. The device according to claim 7, characterized in that The first line of read segment ID number information conversion unit is configured to delete the "@" + chip number in the first line of read segment ID number information of the same read segment in the FASTQ file, and output the coordinates of the read segment on the chip as the first column of information in the converted 1 line of information; Preferably, the third-row placeholder conversion unit is configured to delete the third-row placeholder "+" of the same read segment in the FASTQ file; Preferably, the fourth line ASCII code information conversion unit is configured to output the fourth line ASCII code information of the same read segment in the FASTQ file as the third column information in the converted 1 line of information.

10. A decompression device for a FASTQ compressed file, characterized in that: The decompression device comprises: a decompression module, configured to decompress a gzip file to obtain an initial text file, wherein the gzip file is obtained by compressing a FASTQ file according to the method for compressing a FASTQ file according to any one of claims 1 to 3; an output FASTQ file module, configured to convert one line of information of the same read segment in the initial text file into four lines of read segment information, and output the obtained four lines of read segment information of different read segments as the FASTQ file; The information of one line of the same read segment in the initial text file is as follows from left to right: the first column is the read segment ID number information, the second column is the read segment sequence information, and the third column is the ASCII code information; The order of the four lines of read information in the FASTQ file from top to bottom is: the first line of read ID information, the second line of read sequence information, the third line of placeholders and the fourth line of ASCII code information; The FASTQ file output module includes: a first column read ID number information conversion unit, a second column read sequence information conversion unit and a third column ASCII code information conversion unit.

11. The decompression device according to claim 10, characterized in that: The second column read sequence information conversion unit comprises: a total binary string conversion unit, configured to restore the second column of read sequence information by performing a second-order matrix multiplication calculation to obtain a total binary string; The second-line read sequence information integration unit is configured to divide the total binary string into four single binary strings on average, mark the four single binary strings from beginning to end in the order of A, T, C, and G, the positions of 1 in the single binary strings are the positions of the marked bases in the read sequence, and integrate the respective positions of the bases A, T, C, and G to obtain the second-line read sequence information in the four-line read information group in the FASTQ file; Preferably, the total binary string conversion unit comprises: A second-order matrix conversion unit is configured to restore the four numbers of the second column of read sequence information into a second-order matrix in sequence, divide the four numbers in the second-order matrix into a left column and a right column, compare the two numbers in the same row in the two columns respectively, convert the result of the comparison into a total binary string, and subtract the smaller number in the same row from the larger number, retain the smaller number, and obtain the next second-order matrix, and perform subtraction in sequence until the two columns of numbers in the same row in a certain second-order matrix are equal, then the restoration of the second-order matrix multiplication calculation is terminated; The second-order matrix conversion unit comprises: Generate a total binary string unit, which is set to add a 0 to the leftmost side of the binary string if the two numbers in the left column are greater than the two numbers in the right column, and add a 1 to the leftmost side of the binary string if the two numbers in the left column are less than the two numbers in the right column. The binary character string is converted sequentially on the second-order matrix before the restoration is completed to obtain one total binary character string.

12. The decompression device according to claim 10, characterized in that: The first column read segment ID number information conversion unit is configured to add “@” + chip number before the first column read segment ID number information, and output it as the first row of read segment ID number information in the four rows of read segment information in the FASTQ file; Preferably, the third column ASCII code information conversion unit is configured to output the third column ASCII code information as the fourth line of ASCII code information in the four lines of read information in the FASTQ file; Preferably, the FASTQ file output module further comprises: an adding information unit, which is configured to output "+" as the third line of placeholder information in the four-line group of read information in the FASTQ file.

13. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 3 or the decompression method according to any one of claims 4 to 6.

14. A processor, characterized in that: The processor is used to run a program, wherein the program runs the method according to any one of claims 1 to 3 or the decompression method according to any one of claims 4 to 6.

Images