Genetic data lossless compression system

Abstract

The invention discloses a gene data lossless compression system, comprising: a data stream shunting module, a data compression module and a bit width conversion module; the data stream shunting module receives a gene data file, the gene data file comprises a plurality of reads, each read comprises four parts of ID, sequence, plus and quality, the data stream shunting module divides the input data stream into four parts of ID data stream, sequence data stream, plus data stream and quality data stream, and transmits the data stream to the data compression module; the data compression module compresses the received data stream by different compression methods; the bit width conversion module performs bit width conversion on the compression result of the data compression module. The gene data lossless compression system of the invention has a high throughput rate.

Description

Technical Field

[0001] The invention belongs to the technical field of data compression, and in particular relates to a gene data lossless compression system. Background Art

[0002] The second generation of gene sequencing technology has significantly reduced the cost of gene sequencing, and the size of gene data files has increased accordingly. Computational tasks based on gene data have become increasingly complex, and may require the collaboration of multiple computing nodes to complete data processing tasks such as sequencing, sorting, and gene matching. Data compression technology can reduce the space occupied by files, alleviate the bandwidth and capacity pressure brought by large-scale gene data files to networks and storage devices, and thus improve the overall computing efficiency of computing node systems.

[0003] In order to completely preserve the effective information in the genetic data file, lossless compression methods are usually used. Compared with lossy compression, the result of lossless compression can restore the effective information in the source file without distortion through decompression methods. Indicators for measuring the performance of lossless compression methods include compression ratio and data throughput. The compression ratio refers to the ratio of the size of the file before compression to the size after compression. For example, a 100M file is compressed to 50M, and the compression ratio is 100 / 50=2. The larger the compression ratio, the greater the space saved by lossless compression. The data throughput refers to the size of the data that can be processed per second. For example, if the data throughput is 100MB / s, it means that the lossless compression method can process 100MB of data per second.

[0004] Most existing lossless compression algorithms are based on CPU implementation, and their design goal is to achieve a high compression ratio. Their disadvantage is that the data processing capability is limited by the CPU performance and quantity. When faced with massive amounts of genetic data, their data throughput cannot be well matched with the data throughput of computing nodes and network transmission, which reduces the overall efficiency of the computing system.

[0005] The compression algorithm implemented on the existing FPGA platform mainly compares the gene data file to be compressed with the existing sequence (reference sequence) and uses the position coordinates in the reference sequence to replace the gene sequence. The shortcomings are that the compression effect depends on the selection of the reference sequence, and the hardware implementation of the compression algorithm occupies more storage devices and logic resources, which is not conducive to embedding the compression algorithm into the computing node and realizing the computing in the network, and it is difficult to parallelize. Summary of the invention

[0006] The present invention provides a gene data lossless compression system to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0007] A gene data lossless compression system, comprising: a data stream splitting module, a data compression module and a bit width conversion module;

[0008] The data stream splitting module receives a gene data file, the gene data file includes a plurality of reads, each read includes four parts: ID, sequence, plus and quality, the data stream splitting module divides the input data stream into four parts: ID data stream, sequence data stream, plus data stream and quality data stream, and transmits the data stream to the data compression module;

[0009] The data compression module compresses the received data stream using different compression methods;

[0010] The bit width conversion module performs bit width conversion on the compression result of the data compression module.

[0011] Furthermore, the data compression module includes an ID compression module, a sequence compression module and a quality compression module;

[0012] After dividing the input data stream into four parts, namely, ID data stream, sequence data stream, plus data stream and quality data stream, the data stream splitting module discards the plus data stream and transmits the ID data stream, sequence data stream and quality data stream to the ID compression module, the sequence compression module and the quality compression module respectively.

[0013] Further, the ID compression module compresses the ID information according to the relationship between the ID information of different reads of the ID data stream;

[0014] The sequence compression module uses a 2-bit binary number to represent the base information, and converts four 2-bit binary numbers into one 8-bit binary number;

[0015] The quality compression module compresses characters in the quality data stream by using MTF coding and fixed dictionary Huffman coding.

[0016] Furthermore, the ID information in each read of the ID data stream is divided into several partitions according to special symbols, and the ID compression module adopts a different compression method for each type of partition.

[0017] Further, the ID compression module includes an ASCII code partition module, an ASCII code conversion module and an ID compression encoding module;

[0018] The ASCII code partitioning module is used to filter out information of a part of the partitions of the ID information and transmit it to the ASCII code conversion module;

[0019] The ASCII code conversion module is used to convert the received ID partition information represented in the form of ASCII code into binary digital representation;

[0020] The ID compression encoding module is used to encode the ID partition information represented in binary form.

[0021] Furthermore, the ID compression module further compresses the processed data using an RLE encoding method.

[0022] Furthermore, the quality compression module includes a Huffman codeword generation module, an MTF encoding module and a Huffman encoding module with a fixed dictionary;

[0023] The Huffman codeword generation module is used to receive the Huffman codeword length configured externally, and generate a Huffman code table, i.e., a dictionary, according to the conversion rules specified in the paradigm Huffman coding;

[0024] The MTF coding is used to perform MTF coding on the input quality data stream;

[0025] The Huffman encoding module is used to store the dictionary generated by the Huffman codeword generation module, and perform Huffman encoding on the encoding result of the MTF encoding module according to the dictionary.

[0026] Furthermore, the bit width conversion module converts the compression result of the data compression module into a data stream with a bit width of 64 bits.

[0027] Furthermore, the bit width conversion module includes an ID compression result bit width conversion module, a sequence compression result bit width conversion module and a quality compression result bit width conversion module;

[0028] The ID compression result bit width conversion module extracts valid information of the ID compression result and converts it into a 64-bit bit width;

[0029] The sequence compression result bit width conversion module converts eight valid 8-bit binary numbers into one 64-bit binary number through a register;

[0030] The quality compression result bit width conversion module extracts the effective number of bits of the input quality compression result and converts it into a 16-bit output signal, and then converts every 4 effective 16-bit signals into a 64-bit output signal through a register.

[0031] The benefit of the present invention lies in that the provided gene data lossless compression system has the characteristics of high throughput and saving FPGA hardware resources, can be embedded in various computing nodes and can be further expanded to support more types of FASTQ files. The compression algorithm in the present invention is a pipeline structure based on data flow, which can maximize the throughput of hardware implementation. The present invention belongs to an algorithm without reference sequence, and does not require storage resources to store reference sequences; the compression algorithm structure design of the present invention is suitable for FPGA implementation, for example, the Huffman algorithm based on a fixed dictionary is adopted to save logic resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 A schematic diagram of a FASTQ file fragment used in the compression algorithm provided in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the content classification of FASTQ;

[0035] Figure 3 An operation flow chart of the gene data lossless compression system provided by an embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the data flow diversion module;

[0037] Figure 5 Schematic diagram of content partitioning of ID information;

[0038] Figure 6 It is a schematic diagram of the ID compression module;

[0039] Figure 7 Schematic diagram of the input and output correspondence of RLE encoding;

[0040] Figure 8 It is a schematic diagram of the sequence compression method;

[0041] Fig. 9 This is a schematic diagram of the quality compression module;

[0042] Fig.10 The schematic diagram is a working principle of the bit width conversion module taking 8-bit output bit width as an example. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The present application discloses a gene data lossless compression system for compressing gene data files. The input gene data files are stored in the form of FASTQ files. Figure 1 The following is a partial schematic diagram of a gene data file, which contains 5 reads. Figure 2 The following is the structure of a read in a FASTQ file. According to the nature of the content, the read can be divided into four parts: ID, sequence, plus, and quality.

[0045] The gene data lossless compression system of the present application comprises: a data stream splitting module, a data compression module and a bit width conversion module.

[0046] like Figure 3 The flowchart of the gene data lossless compression system of the present application is shown. The data stream splitting module receives the gene data file, and the data stream splitting module divides the input data stream into four parts: ID data stream, sequence data stream, plus data stream and quality data stream according to the composition of the gene data file, and transmits the data stream to the data compression module. The data compression module compresses the received data stream by using different compression methods. The bit width conversion module performs bit width conversion on the compression result of the data compression module.

[0047] Specifically, if Figure 4 As shown in the figure, the data in the FASTQ file is input into the data stream splitting module in the form of 8-bit ASCII code data stream, and then divided into four parts: ID data stream, sequence data stream, plus data stream and quality data stream of 8-bit ASCII code data stream. Among them, the data stream of the plus part does not contain valid information and is directly discarded. The data stream of the remaining part is output and transmitted to the corresponding data compression module.

[0048] In the implementation manner of the present application, the data compression module includes an ID compression module, a sequence compression module and a quality compression module. After the data stream splitting module divides the input data stream into four parts, namely, an ID data stream, a sequence data stream, a plus data stream and a quality data stream, the plus data stream is discarded, and the ID data stream, the sequence data stream and the quality data stream are transmitted to the ID compression module, the sequence compression module and the quality compression module respectively.

[0049] In an embodiment of the present application, the ID compression module compresses the ID information according to the relationship between the ID information of different reads of the ID data stream. The ID information in each read of the ID data stream ID is divided into several partitions according to special characters, such as Figure 5 In the FASTQ file, special characters are punctuation marks, such as "@", ".", ":", and " / ". The function of the ID compression module is to compress the ID information according to these partitions, which specifically includes an ASCII code partition module, an ASCII code conversion module, and an ID compression encoding module, as shown in Figure 6 shown.

[0050] The ASCII code partition module recognizes the ASCII codes of these punctuation marks through a finite state machine, thereby determining which partition the current input data stream belongs to and performing corresponding processing, including generating a header file of the compression result, and filtering out partitions 4, 5, 6, and 7 of the ID information and transmitting them to the ASCII code conversion module.

[0051] The ASCII code conversion module is used to convert the received information of the ID partitions 4, 5, 6, and 7 from ASCII codes into binary numbers. The conversion result is encoded by the ID compression encoding module.

[0052] The encoding principles for different ID partitions in the ID compression module are summarized as follows:

[0053] First, the values ​​of each partition of the ID information of the first read are directly stored in the header file of the compression result by the ASCII code partition module for decompression.

[0054] According to the relationship between the contents of different partitions of reads in the same FASTQ file, the partitions of IDs are further divided into four categories, and different compression methods are used for each category of partitions:

[0055] (1) In the same FASTQ file, the corresponding partition contents of all reads are exactly the same, such as partition 1, partition 3, and partition 8. The ASCII code partitioning module stores the information of these partitions in the header of the compression result, and subsequent encoding no longer involves these partitions. In the original FASTQ file, the contents of these partitions appear once for each read. After compression, the contents of these three partitions appear only once in the header, thus achieving the compression effect.

[0056] (2) and the previous read partitions show a fixed pattern, such as partition 2, which shows a pattern of increasing by 1. Since its change pattern is fixed, the same compression method as (1) is adopted, that is, only the value of partition 2 of the first read is stored in the header, and the subsequent encoding process does not involve partition 2.

[0057] (3) For partitions with similar contents to the previous read, such as partition 4, partition 5, and partition 6, the difference between the partition and the corresponding partition of the previous read is taken as the compression result. Since the ID part of the first read is fully reflected in the header, its compression result is all 0.

[0058] (4) If the partition content is very different from the previous read and has no obvious pattern, such as partition 7, the original value is used as the compression result.

[0059] After processing according to the above encoding principles, the result obtained will have several consecutive 0s at the beginning. The ID compression module uses RLE (run-length encoding) to further compress the encoding result. The compression strategy is: if there are consecutive 0s at the beginning, then these consecutive 0s are replaced by the number of times 0 appears. The specific corresponding relationship is as follows Figure 7 As shown in FIG. 1 , “X” represents a non-zero value. The output signal of the ID compression module also includes a bits signal recording the number of valid bits of the compression result.

[0060] For the sequence data stream, since the base of the sequence can only have four possibilities, "A", "C", "G", and "T", a 2-bit binary number can be used to represent one base. In the implementation of the present application, the sequence compression module uses a 2-bit binary number to represent the base information, and first converts "A", "C", "G", and "T" into binary "00", "01", "10", and "11" respectively. And converts the four 2-bit binary numbers into an 8-bit binary number, such as Figure 8 shown.

[0061] The types of characters that may appear in the Quality data stream are limited. Based on this feature, the quality compression module compresses the characters in the quality data stream through MTF (move-to-front) coding and fixed-dictionary Huffman coding. "Fixed-dictionary Huffman coding" means that the Huffman tree is not built based on scanning all data, but is configured by external codeword length, and the Huffman code table, i.e., "dictionary", is formed according to the rules specified by the paradigm Huffman coding. The dictionary does not change during the compression process.

[0062] like Fig. 9 As shown, in an embodiment of the present application, the quality compression module includes a Huffman codeword generation module, an MTF encoding module and a Huffman encoding module with a fixed dictionary.

[0063] The Huffman codeword generation module is used to receive the externally configured Huffman codeword length and generate the Huffman code table, i.e., the dictionary, according to the conversion rules specified in the paradigm Huffman coding. MTF coding is used to perform MTF coding on the input quality data stream. The Huffman coding module is used to store the dictionary generated by the Huffman codeword generation module and perform Huffman coding on the coding result of the MTF coding module according to the dictionary. After the quality data stream is MTF-coded, the Huffman code table is queried and Huffman coding is performed. In hardware implementation, the Huffman code table is stored in RAM, and the table lookup operation is implemented in the form of reading RAM.

[0064] The function of the bit width conversion module is to pack the effective number of bits of the processing results of the three compression modules into an output signal of a specified bit width. In the embodiment of the present application, the bit width conversion module converts the compression result of the data compression module into a data stream with a bit width of 64 bits. Fig.10 As shown, the case where the output bit width is 8 bits is taken as an example. Fig.10 This is an example diagram of converting input containing less than or equal to 8 bits of valid data into 8-bit output. The bit width conversion uses 8 groups of FIFOs with a bit width of 1. start = 3, indicating that data writing starts from FIFO3; bits = 4, indicating that the number of valid data bits of the input signal is 4, that is, the 4 bits of data starting from the 3rd bit are to be written into the FIFO. When the status of the 8 groups of FIFOs are all "non-empty", reading these 8 groups of FIFOs at the same time can obtain data with a bit width of 8 and only valid information. In the actual compression process, the specified output bit width is 64 bits.

[0065] Specifically, the bit width conversion module includes an ID compression result bit width conversion module, a sequence compression result bit width conversion module and a quality compression result bit width conversion module.

[0066] The ID compression result bit width conversion module extracts the valid information of the ID compression result and converts it into a 64-bit bit width. For the sequence data stream part, the compressed data bit width is 8 bits, so the register can be used directly to convert 8 valid 8-bit data into 1 64-bit data.

[0067] For the ID data stream and the quality data stream, the conversion method is more complicated than the sequence data stream because the number of bits of valid data contained in the output data will change. The bits containing valid information in the input data are cached by using parallel FIFOs combined with the information recorded in the register. When the status of all FIFOs is displayed as "not empty", the output signal of the specified bit width can be obtained by reading these FIFOs at the same time.

[0068] Specifically, the sequence compression result bit width conversion module converts 8 valid 8-bit binary numbers into 1 64-bit binary number through a register. The quality compression result bit width conversion module extracts the valid number of bits of the input quality compression result and converts it into a 16-bit output signal, and then converts every 4 valid 16-bit signals into a 64-bit output signal through a register.

[0069] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A gene data lossless compression system, characterized in that: Contains: data stream splitting module, data compression module and bit width conversion module; The data stream splitting module receives a gene data file, the gene data file includes a plurality of reads, each read includes four parts: ID, sequence, plus and quality, the data stream splitting module divides the input data stream into four parts: ID data stream, sequence data stream, plus data stream and quality data stream, and transmits the data stream to the data compression module; The data compression module compresses the received data stream using different compression methods; The bit width conversion module performs bit width conversion on the compression result of the data compression module; The data compression module includes an ID compression module, a sequence compression module and a quality compression module; After the data stream splitting module divides the input data stream into four parts, namely, an ID data stream, a sequence data stream, a plus data stream and a quality data stream, the plus data stream is discarded, and the ID data stream, the sequence data stream and the quality data stream are transmitted to the ID compression module, the sequence compression module and the quality compression module respectively; The ID compression module compresses the ID information according to the relationship between the ID information of different reads of the ID data stream; The sequence compression module uses a 2-bit binary number to represent the base information, and converts four 2-bit binary numbers into one 8-bit binary number; The quality compression module compresses the characters in the quality data stream by using MTF coding and Huffman coding with a fixed dictionary; The ID compression module divides the ID information in each read of the ID data stream into several partitions according to the special characters in the ID information, stores the values ​​of each partition of the ID information of the first read in the header file of the compression result, and adopts different compression methods for each type of partition; The ID compression module includes an ASCII code partition module, an ASCII code conversion module and an ID compression encoding module; The ASCII code partitioning module is used to filter out the information of some partitions of the ID information and transmit it to the ASCII code conversion module, while completing the encoding work of the remaining partitions; The ASCII code conversion module is used to convert the received ID partition information represented in the form of ASCII code into binary digital representation; The ID compression encoding module is used to encode the ID partition information represented in binary form.

2. The method for lossless compression of genetic data according to claim 1, characterized in that: The ID compression module further compresses the processed data using the RLE encoding method.

3. The method for lossless compression of genetic data according to claim 1, characterized in that: The quality compression module includes a Huffman codeword generation module, an MTF encoding module and a Huffman encoding module with a fixed dictionary; The Huffman codeword generation module is used to receive the Huffman codeword length configured externally, and generate a Huffman code table, i.e., a dictionary, according to the conversion rules specified in the paradigm Huffman coding; The MTF coding is used to perform MTF coding on the input quality data stream; The Huffman encoding module is used to store the dictionary generated by the Huffman codeword generation module, and perform Huffman encoding on the encoding result of the MTF encoding module according to the dictionary.

4. The method for lossless compression of genetic data according to claim 1, characterized in that: The bit width conversion module converts the compression result of the data compression module into a data stream with a bit width of 64 bits.

5. The method for lossless compression of genetic data according to claim 4, characterized in that: The bit width conversion module includes an ID compression result bit width conversion module, a sequence compression result bit width conversion module and a quality compression result bit width conversion module; The ID compression result bit width conversion module extracts valid information of the ID compression result and converts it into a 64-bit bit width; The sequence compression result bit width conversion module converts eight valid 8-bit binary numbers into one 64-bit binary number through a register; The quality compression result bit width conversion module extracts the effective number of bits of the input quality compression result and converts it into a 16-bit output signal, and then converts every 4 effective 16-bit signals into a 64-bit output signal through a register.

Images