Gene sequencing data processing method and system and readable storage medium

Abstract

The invention discloses a gene sequencing data processing method and system and a readable storage medium. The processing method comprises the following steps: dividing received gene sequencing data to obtain at least one data block, wherein each data block comprises at least one combination of an identifier, a base sequence and a quality value; and in response to the compression signal, performing compression processing on the identifier through a first compression algorithm, performing compression processing on the base sequence through a second compression algorithm, and performing compression processing on the quality value through a third compression algorithm to obtain a data compression block. According to the processing method, the gene sequencing data can be divided into the data blocks, and the data with different characteristics in each data block is correspondingly compressed without additional storage space, so that the effect of receiving, compressing and storing at the same time can be realized, and the processing and storage efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of gene sequencing technology, and in particular to a gene sequencing data processing method, a processing system and a readable storage medium. Background Art

[0002] The large amount of data obtained through gene sequencing poses a great challenge to the storage system, which requires the gene sequencing data to be compressed. Gene sequencing data contains data with different characteristics. In related technologies, in order to achieve rapid data extraction and processing, data with different characteristics are compressed and decompressed separately, which requires dividing multiple memory spaces for separation and compression and decompression processing, which requires a large amount of memory and dump processing, reducing the efficiency of processing and storage. Summary of the invention

[0003] The present invention provides a method, a processing system and a readable storage medium for processing gene sequencing data to solve at least one of the above-mentioned technical problems.

[0004] A method for processing gene sequencing data of the present invention comprises:

[0005] Dividing the received gene sequencing data to obtain at least one data block, each of the data blocks including at least one combination of an identifier, a base sequence, and a quality value;

[0006] In response to the compression signal, the identifier is compressed using a first compression algorithm, the base sequence is compressed using a second compression algorithm, and the quality value is compressed using a third compression algorithm to obtain a data compression block.

[0007] With the above processing method, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0008] The received gene sequencing data is divided and processed to obtain at least one data block, including:

[0009] The gene sequencing data is divided and processed so that the size of the data blocks obtained by the division is less than or equal to the preset data amount. This is conducive to improving compression efficiency and reducing the frequent occupation of computing resources.

[0010] The processing method comprises:

[0011] When it is determined that the data volume of a data block obtained through the division process is greater than or equal to the preset data volume, the determined data block is compressed, which is beneficial to reducing the overall compression time of the gene sequencing data.

[0012] The identifier is compressed by a first compression algorithm, the base sequence is compressed by a second compression algorithm, and the quality value is compressed by a third compression algorithm, including:

[0013] The identifier, the comparison result and the quality value are compressed in parallel, and the data compression block is obtained according to the result of the parallel compression process, wherein the comparison result is obtained by comparing the base sequence with a preset reference genome. In this way, the compression time can be reduced.

[0014] The first compression algorithm includes a dictionary tree algorithm; and / or, the second compression algorithm includes a range coding algorithm; and / or, the third compression algorithm includes a range coding algorithm.

[0015] A method for processing gene sequencing data of the present invention comprises:

[0016] Reading a data compression block to obtain compression results corresponding to the identifier, the base sequence, and the quality value, respectively, wherein the data compression block is obtained by compressing the identifier using a first compression algorithm, compressing the base sequence using a second compression algorithm, and compressing the quality value using a third compression algorithm;

[0017] The compression results corresponding to the identifier, the base sequence and the quality value are decompressed, and the data block is obtained according to the results after the decompression, the data block includes the identifier, the base sequence and the quality value, and the data block is obtained by dividing the received gene sequencing data.

[0018] With the above processing method, gene sequencing data can be divided into data blocks, and by performing corresponding compression processing on data with different characteristics in each data block, the compressed gene sequencing data can be easily read.

[0019] The processing method comprises:

[0020] When at least one data block is obtained, the gene sequencing data is obtained by combining at least one data block according to a data index, wherein the data index is generated when the data compression block is obtained. In this way, the gene sequencing data can be obtained quickly and conveniently.

[0021] Decompressing the compression results corresponding to the identifier, the base sequence, and the quality value respectively includes:

[0022] The compression results corresponding to the identifier, the base sequence and the quality value are decompressed in parallel, which is beneficial to improving the decompression efficiency.

[0023] A gene sequencing data processing system of the present invention includes a first host module and a first processing module, wherein the first host module is used to:

[0024] Dividing the received gene sequencing data to obtain at least one data block, each of the data blocks including at least one combination of an identifier, a base sequence, and a quality value;

[0025] The first processing module is used for:

[0026] In response to the compression signal, the identifier is compressed using a first compression algorithm, the base sequence is compressed using a second compression algorithm, and the quality value is compressed using a third compression algorithm to obtain a data compression block.

[0027] With the above processing method, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0028] A processing system of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the processing method described in at least one of the above-mentioned embodiments can be implemented.

[0029] With the above processing method, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0030] A computer-readable storage medium of the present invention stores a computer program thereon, and when the computer program is executed by a processor, the steps of the processing method described in at least one of the above-mentioned embodiments can be implemented.

[0031] With the above processing method, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0032] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0034] Figure 1 is a schematic flow chart of a processing method according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a module of a processing system according to an embodiment of the present invention;

[0036] Figure 3 is another schematic flow chart of a processing method according to an embodiment of the present invention;

[0037] Figure 4 is another module schematic diagram of a processing system according to an embodiment of the present invention;

[0038] Figure 5 is another module schematic diagram of a processing system according to an embodiment of the present invention;

[0039] Figure 6 It is another module schematic diagram of the processing system according to the embodiment of the present invention.

[0040] Description of main component symbols:

[0041] Processing system 10;

[0042] A first host module 11 , a first processing module 12 , a second host module 13 , a second processing module 14 , a memory 15 , and a processor 16 . DETAILED DESCRIPTION

[0043] In the description of the present invention, examples of some disclosed embodiments have been shown in the accompanying drawings accordingly, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] In the description of the present invention, many different embodiments or examples are disclosed to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention.

[0045] With the increasing maturity and widespread application of high-throughput testing technology, the time to obtain biological gene sequences has been greatly reduced, and the resulting storage capacity of gene sequencing data has also exploded, posing a huge challenge to existing storage devices. Taking DNBSEQ-T7 as an example, the system generates 10 to 40 billion 150-nucleotide read sequences in a single 24-30 hour operation, and requires up to 6TB of storage space to store gene sequence data. The demand for improving the capacity and performance of storage devices has become increasingly urgent.

[0046] Therefore, the application of compression technology based on computational storage is particularly important in the fields of gene sequencing and bioinformatics analysis. The use of compression can reduce data redundancy, thereby reducing the consumption of storage space and network transmission bandwidth; computational storage can unload the compression / decompression load of the host, release the host's computing resources and memory resources, and improve the overall performance of the system.

[0047] The earliest compression method was to use general compression tools such as gzip and bzip2. However, these tools only compress the basic features of the text and do not fully utilize the characteristics of genetic sequencing data in FASTQ format, so the compression rate is not ideal.

[0048] Subsequently, compression algorithms for gene sequencing data in FASTQ format emerged. Based on the characteristics of gene sequencing data in FASTQ format, it is usually divided into three parts for separate compression, namely identifiers, base sequences, and quality value data. Compression of base sequences is divided into two types of compression algorithms: non-parameterized and parameterized. The parameterized compression is based on the reference genome sequence, and an index is established to compare with the file to be compressed. After the comparison is successful, the position, deviation and other information in the reference genome are recorded, while the non-parameterized compression directly compresses the bases.

[0049] In the above compression algorithm, in order to realize the separate compression of identifier, base sequence and quality value data, three memory spaces are required for separation and compression processing. In order to improve the efficiency of reference genome sequence comparison, the reference genome sequence segment will be indexed. This process also consumes a lot of memory, and in the process of comparison and file writing, a large number of computing tasks will be generated, which will consume the computing resources of the host.

[0050] Based on the foregoing content, the present invention can solve at least one of the above-mentioned technical problems.

[0051] Please refer to Figure 1 A method for processing gene sequencing data of the present invention may include:

[0052] 01: Divide the received gene sequencing data to obtain at least one data block, each data block including at least one combination of an identifier, a base sequence, and a quality value;

[0053] 02: In response to the compression signal, the identifier is compressed by a first compression algorithm, the base sequence is compressed by a second compression algorithm, and the quality value is compressed by a third compression algorithm to obtain a data compression block.

[0054] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 2 The processing system 10 may include a first host module 11 and a first processing module 12. The first host module 11 may be used to: divide the received gene sequencing data to obtain at least one data block, each data block including an identifier, a base sequence and a quality value. The first processing module 12 is used to: respond to the compression signal, compress the identifier using a first compression algorithm, compress the base sequence using a second compression algorithm, and compress the quality value using a third compression algorithm to obtain a data compression block.

[0055] With the above-mentioned processing method and processing system 10, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0056] Moreover, without the need for dumping, it can avoid consuming a large amount of memory and computing processing resources, which is beneficial to improving the overall performance of the system.

[0057] Gene sequencing data can be obtained when a gene sequencer sequences a gene sample. Gene sequencing data can represent information related to the gene sequence of a gene sample. The gene information of the gene sample can be determined based on the gene sequencing data, which is convenient for further gene research. The processing system 10 can be a gene sequencer, so that when the gene sequencing data of the gene sample is obtained, the gene sequencing data can be processed and stored in real time, thereby reducing the storage pressure of the gene sequencer and avoiding affecting the performance of the gene sequencer during sequencing when the gene sequencing data is processed.

[0058] The gene sequencing data may be stored in a corresponding space when acquired. The processing system 10 may store the gene sequencing data in a corresponding storage space of the processing system 10 .

[0059] Gene sequencing data may have a large number of data groups, each of which may consist of an identifier, a base sequence and a quality value. In other words, in the present invention, the minimum complete data of gene sequencing data should be in the form of a data group.

[0060] When a data block is obtained by division, the position where the gene sequencing data is divided may be the starting position and the ending position of the data corresponding to the data block. When more than one data block is obtained by division, the position where the gene sequencing data is divided may be the position between two data groups of the gene sequencing data. Based on the above, the data groups in each data block can be complete, avoiding data missing and inability to perform subsequent compression processing.

[0061] According to the different characteristics of the data in gene sequencing data, it can be classified into identifiers, base sequences and quality values.

[0062] The base sequences in the data group are obtained by sequencing the corresponding positions of the gene sample. The identifiers in the data group can represent the identification information of the base sequences in the same data group. The quality values ​​in the data group can reflect the sequencing quality of the base sequences in the same data group. Each data block can have at least one data group.

[0063] The compression signal can be generated and responded by the processing system 10 according to specific circumstances, or can be generated manually. In the case of generating a compression signal, it means that the received gene sequencing data can be compressed.

[0064] After the data with different characteristics in the data block are compressed accordingly, it means that the data block is completely compressed, and the result of the internal data after compression processing can form a data compression block.

[0065] Different types of compression algorithms can be used for compression according to the different characteristics of the data. Compared with using the same algorithm to compress all the data in a data block, it can adapt to the characteristics of the data and achieve a certain compression rate and compression efficiency, which is conducive to achieving good compression effects.

[0066] In addition, in the processing method of the present invention, when the amount of data is small, there may be a situation where only one data block can be obtained after the division processing.

[0067] In the processing method of the present invention, the gene sequencing data can be uncompressed internal data stored locally, or external data directly received through TCP / IP.

[0068] Step 01 (dividing the received gene sequencing data to obtain at least one data block) may include:

[0069] The gene sequencing data is divided and processed so that the size of the data blocks obtained by the division is less than or equal to the preset data amount.

[0070] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 2 The first host module 11 is used to divide and process the gene sequencing data according to the order in which the data is received, and make the size of the data blocks obtained by the division less than or equal to the preset data amount.

[0071] This is beneficial for both improving compression efficiency and reducing the frequent occupation of computing resources.

[0072] The gene sequencing data may be divided according to the order in which the gene sequencing data are received, or may be divided according to the order in which the gene sequencing data are stored in the corresponding storage space.

[0073] In some cases, gene sequencing data may be generated in real time and need to be processed and stored immediately, so that the gene sequencing data can be presented in the form of a data stream, or in other words, the reception of gene sequencing data is actually the continuous reception of multiple data groups. The order of receiving gene sequencing data can be the order of receiving the contents of the gene sequencing data according to the reception time.

[0074] In some cases, the storage space used to store gene sequencing data may not be continuous, so it may be necessary to store continuous data in the gene sequencing data in separate storage spaces, or in other words, the data received first may be stored at the back of the storage space, and the data received later may be stored at the front of the storage space. The storage order of the gene sequencing data in the corresponding storage space may be the front-to-back order of the storage space where the contents of the gene sequencing data are located.

[0075] When gene sequencing data is obtained, data blocks can be divided according to a certain data size. The size of the divided data blocks will be less than or equal to the preset size, so that the data blocks are not too large and affect the compression efficiency, and the data blocks are not too small and the device frequently calculates for compression processing.

[0076] Treatment options may include:

[0077] When it is determined that the data amount of a data block obtained through the division process is greater than or equal to the preset data amount, compression processing is performed on the determined data block.

[0078] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 2 The first host module 11 is used to: when it is determined that the data amount of a data block obtained through the division process is greater than or equal to the preset data amount, compress the determined data block.

[0079] This will help reduce the overall compression time of genetic sequencing data.

[0080] When receiving gene sequencing data, the gene sequencing data can be continuously divided and processed to obtain different data blocks in sequence. When the amount of data in the divided data blocks is greater than or equal to the preset amount of data, it can be determined that the data blocks have sufficient data amount, and then the determined data blocks can be compressed.

[0081] Since the amount of data to be compressed and the compression time are basically positively correlated, the compression process can be started when the obtained data block has a large enough amount of data. During the compression process, since a corresponding time is required, the remaining part of the gene sequencing data can still be divided and processed within this time, which can improve the processing efficiency.

[0082] Moreover, when the data volume of a data block is greater than or equal to a preset data volume, compression processing is performed. Compared with dividing the entire gene sequencing data and then compressing all of it, this can avoid the situation where the computing resource occupancy rate is too low and the processing efficiency is reduced, and the occupancy rate is too high and the processing pressure is increased, thereby helping to speed up the processing speed.

[0083] Step 02 (compressing the identifier using a first compression algorithm, compressing the base sequence using a second compression algorithm, and compressing the quality value using a third compression algorithm) may include:

[0084] The identifier, the alignment result and the quality value are compressed in parallel, and a data compression block is obtained according to the result of the parallel compression process. The alignment result is obtained by comparing the base sequence with a preset reference genome.

[0085] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 2 The first processing module 12 is used to: perform parallel compression processing on the identifier, the comparison result and the quality value, and obtain a data compression block according to the result of the parallel compression processing. The comparison result is obtained by comparing the base sequence with the preset reference genome.

[0086] This can reduce the compression time.

[0087] When the identifier, the comparison result and the quality value can be compressed according to different compression algorithms, the different compression algorithms will be executed in parallel, which can reduce the total compression time compared to serial execution.

[0088] Different compression algorithms are executed in parallel, and the first compression algorithm, the second compression algorithm and the third compression algorithm may be executed synchronously, rather than executing one before executing the next one.

[0089] Among them, the identifier can be processed according to the first compression algorithm to obtain a compression prefix, the comparison result can be processed according to the second compression algorithm to obtain a compression comparison result, the quality value can be processed according to the third compression algorithm to obtain compression quality data, and then the compression prefix, compression comparison result and compression quality data can be combined to obtain a data compression block.

[0090] The algorithm for compressing the identifier may include a dictionary tree algorithm. The dictionary tree algorithm can facilitate fast retrieval. In the case where the identifier is used as a prefix of a data group, it can be retrieved in the dictionary tree with a high retrieval speed.

[0091] The algorithm for compressing the comparison result may include a range encoding algorithm. The range encoding algorithm can encode data with a larger character string length, and the encoded result can have a smaller character length, thereby achieving a compression effect. Moreover, the result of decompressing the encoded result can be exactly the same as the data before encoding, which can achieve lossless compression of data.

[0092] The algorithm for compressing the quality value may include a range coding algorithm. Generally, the quality value may have the same length as the base sequence, and the range coding algorithm may ensure lossless compression of the quality value.

[0093] Please refer to Figure 3 A method for processing gene sequencing data of the present invention may include:

[0094] 03: Read the data compression block to obtain compression results corresponding to the identifier, the base sequence and the quality value respectively, the data compression block is obtained by compressing the identifier using the first compression algorithm, compressing the base sequence using the second compression algorithm, and compressing the quality value using the third compression algorithm;

[0095] 04: Decompress the compression results corresponding to the identifier, base sequence and quality value respectively, and obtain data blocks according to the decompression results respectively. The data blocks include the identifier, base sequence and quality value. The data blocks are obtained by dividing the received gene sequencing data.

[0096] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 4, the processing system 10 may include a second processing module 14. The second processing module 14 may be used to: read the data compression block to obtain compression results corresponding to the identifier, the base sequence and the quality value respectively, the data compression block is obtained by compressing the identifier through a first compression algorithm, compressing the base sequence through a second compression algorithm, and compressing the quality value through a third compression algorithm; decompressing the compression results corresponding to the identifier, the base sequence and the quality value respectively, and obtaining a data block according to the results after the decompression processing, the data block includes the identifier, the base sequence and the quality value, and the data block is obtained by dividing the received gene sequencing data.

[0097] In the above-mentioned processing method and processing system 10, the gene sequencing data can be divided into data blocks, and the compressed gene sequencing data can be conveniently read by performing corresponding decompression processing on the data with different characteristics in each data block.

[0098] After the gene sequencing data is compressed, although the compression rate can be improved and the computing resource usage can be reduced, the compressed data cannot be used directly. When the gene sequencing data needs to be used, it needs to be decompressed.

[0099] When the data compression block is read to obtain the compressed identifier, base sequence and quality value respectively, since the data compression block is obtained by combining data, the compression results corresponding to the identifier, base sequence and quality value respectively can be decompressed separately to obtain the identifier, base sequence and quality value before compression.

[0100] According to the positions of the identifiers, base sequences and quality values ​​in the data set, the identifiers, base sequences and quality values ​​can be spliced ​​to obtain corresponding data blocks. In the case of splicing multiple data sets, the obtained data blocks can be combined to obtain gene sequencing data before compression.

[0101] In addition, Figure 4 In the embodiment, the processing system 10 may further include a second host module 13. When the data compression block needs to be decompressed, the second host module 13 may send a decompression signal to the second processing module 14, so that the second processing module 14 may determine to decompress the corresponding data compression block according to the received decompression signal. The second host module 13 may also receive the data block obtained by the second processing module 14 after decompression processing.

[0102] Treatment options may include:

[0103] When at least one data block is obtained, the at least one data block is combined according to a data index to obtain gene sequencing data, and the data index is generated when the data compression block is obtained.

[0104] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 4 The second processing module 14 can be used to: when at least one data block is obtained, combine the at least one data block according to the data index to obtain gene sequencing data, and the data index is generated when the data compression block is obtained.

[0105] In this way, gene sequencing data can be obtained quickly and easily.

[0106] When a data block is compressed to obtain a data compression block, a data index can be generated according to the position of the data block in the gene sequencing data. When the compressed gene sequencing data needs to be decompressed, the position of the data block obtained after the data compression block is decompressed in the original gene sequencing data can be determined according to the data index corresponding to the data compression block, so that the data blocks obtained after the decompression can be conveniently combined in sequence according to the data index, and the data finally obtained can be the original gene sequencing data.

[0107] Step 04 (decompressing the compressed results corresponding to the identifier, base sequence and quality value respectively) may include:

[0108] The compression results corresponding to the identifier, base sequence and quality value are decompressed in parallel.

[0109] The processing method of the present invention can be implemented by the processing system 10 of the present invention. Figure 4 The second processing module 14 can be used to: perform parallel decompression processing on the compression results corresponding to the identifier, the base sequence and the quality value respectively.

[0110] This can help improve decompression efficiency.

[0111] When decompressing the compression results corresponding to the identifier, base sequence and quality value respectively, different decompression operations can be executed in parallel, which can reduce the total decompression time compared to serial execution, thereby improving the decompression efficiency.

[0112] Different decompression operations can be executed in parallel, and the first decompression algorithm, the second decompression algorithm and the third decompression algorithm can be executed synchronously, rather than executing one of them before executing the next one. The first decompression algorithm is used to decompress the compression result of the corresponding identifier, the second decompression algorithm is used to decompress the compression result of the corresponding base sequence, and the third decompression algorithm is used to decompress the compression result of the corresponding quality value.

[0113] In addition, the inventive concept of the present invention can also be embodied through the following situations.

[0114] exist Figure 5In the embodiment, the processing system 10 may include a Host and a CSD (computational storage drive) computing storage module.

[0115] The host may include a CPU (Central Processing Unit) and a DRAM (Dynamic Random Access Memory).

[0116] The CSD computing storage module may include an FPGA (Field Programmable Gate Array) unit, an SSD (Solid State Drive) controller, and a NAND (NotAnd) medium.

[0117] The host can communicate with the FPGA unit and the SSD controller through different PCIe (PCI-Express, peripheral component interconnect express, high-speed serial computer expansion bus standard) channels. The SSD controller can communicate with the NAND medium through the ONFI (Open NAND Flash Interface, open NAND flash memory interface) channel.

[0118] The gene sequencing data can be transmitted to the processing system 10 via TCP (Transmission Control Protocol).

[0119] The host can implement the control logic of the gene sequencing data. When the processing system 10 receives the gene sequencing data, the host can divide and process the gene sequencing data to obtain data blocks. The data blocks divided by the host can be transmitted to the CSD computing and storage module through the PCIe channel.

[0120] When the gene sequencing data needs to be compressed, the Host will transfer the data block to the FPGA unit. The FPGA unit can compress the data block through the built-in compression algorithm. The compressed data block can be transmitted to the SSD controller, and the SSD controller can store the compressed data block so that the compressed data block can be written to the NAND medium through the ONFI channel, and finally the compression and storage of the gene sequencing data can be realized.

[0121] When transmitting data blocks to the CSD computing and storage module, the host can send a compression signal to the CSD computing and storage module, so that the CSD computing and storage module can determine the need to compress the data blocks according to the compression signal.

[0122] When the compressed gene sequencing data needs to be decompressed, the compressed gene sequencing data in the NAND medium can be read by the SSD controller and transmitted to the FPGA unit for decompression. After the decompression is completed, the decompressed gene sequencing data can be transmitted to the Host or other terminal devices that can communicate with the processing system 10 via the PCIe channel or Ethernet according to the command setting of the Host.

[0123] The host can send a decompression signal to the CSD computing and storage module, so that the CSD computing and storage module can determine the need to decompress the compressed gene sequencing data according to the decompression signal.

[0124] The host may be the first host module 11 or the second host module 13. The CSD computing storage module may be the first processing module 12 or the second processing module 14.

[0125] In addition, in some cases, the FPGA unit and the SSD controller can be integrated into the same device. The FPGA unit and the SSD controller can also be set up independently, and the FPGA unit can be used as an external external component of the sequencer or a storage device including the SSD controller, or can be integrated into the sequencer as a built-in compression and decompression acceleration component.

[0126] In summary, the technical solution of the present invention can achieve the following technical effects:

[0127] 1. In the compression part, real-time compression can be performed. The offline data of the gene sequencer (i.e. Fastq format data) can be compressed and stored while being transmitted, without the need for additional storage space, thus avoiding the space and time loss caused by dump processing;

[0128] 2. In the decompression part, according to the decompression instructions, the data is read from the computing storage module and decompressed on the spot, which also avoids the space and time loss caused by the dump processing;

[0129] 3. During the compression / decompression process, the host does not need to participate in the specific compression / decompression operations, which avoids the consumption of the host's computing resources and memory resources by the compression / decompression itself, releases the host resources, and is conducive to improving the overall performance of the system.

[0130] Please refer to Figure 6A processing system 10 of the present invention includes a memory 15 and a processor 16. The memory 15 may store a computer program. When the processor 16 executes the computer program, the steps of the processing method in at least one of the above cases can be implemented.

[0131] For example, when the computer program of the present invention is executed, the following steps of the processing method can be implemented:

[0132] 01: Divide the received gene sequencing data to obtain at least one data block, each data block including at least one combination of an identifier, a base sequence, and a quality value;

[0133] 02: In response to the compression signal, the identifier is compressed by a first compression algorithm, the base sequence is compressed by a second compression algorithm, and the quality value is compressed by a third compression algorithm to obtain a data compression block.

[0134] In the above-mentioned processing system 10, gene sequencing data can be divided into data blocks. By performing corresponding compression processing on data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0135] When the computer program of the present invention is executed, the following steps of the processing method can also be implemented:

[0136] 03: Read the data compression block to obtain compression results corresponding to the identifier, the base sequence and the quality value respectively, the data compression block is obtained by compressing the identifier using the first compression algorithm, compressing the base sequence using the second compression algorithm, and compressing the quality value using the third compression algorithm;

[0137] 04: Decompress the compression results corresponding to the identifier, base sequence and quality value respectively, and obtain data blocks according to the decompression results respectively. The data blocks include the identifier, base sequence and quality value. The data blocks are obtained by dividing the received gene sequencing data.

[0138] With the above processing method, gene sequencing data can be divided into data blocks, and by performing corresponding decompression processing on data with different characteristics in each data block, the compressed gene sequencing data can be easily read.

[0139] A computer-readable storage medium of the present invention stores a computer program, which, when executed by the processor 16, can implement the steps of the processing method in at least one of the above situations.

[0140] For example, when the computer program of the present invention is executed, the following steps of the processing method can be implemented:

[0141] 01: Divide the received gene sequencing data to obtain at least one data block, each data block including at least one combination of an identifier, a base sequence, and a quality value;

[0142] 02: In response to the compression signal, the identifier is compressed by a first compression algorithm, the base sequence is compressed by a second compression algorithm, and the quality value is compressed by a third compression algorithm to obtain a data compression block.

[0143] The above-mentioned computer-readable storage medium can divide the gene sequencing data into data blocks. By performing corresponding compression processing on the data with different characteristics in each data block, no additional storage space is required, which is conducive to achieving the effect of receiving, compressing and storing at the same time, thereby improving the efficiency of processing and storage.

[0144] When the computer program of the present invention is executed, the following steps of the processing method can also be implemented:

[0145] 03: Read the data compression block to obtain compression results corresponding to the identifier, the base sequence and the quality value respectively, the data compression block is obtained by compressing the identifier using the first compression algorithm, compressing the base sequence using the second compression algorithm, and compressing the quality value using the third compression algorithm;

[0146] 04: Decompress the compression results corresponding to the identifier, base sequence and quality value respectively, and obtain data blocks according to the decompression results respectively. The data blocks include the identifier, base sequence and quality value. The data blocks are obtained by dividing the received gene sequencing data.

[0147] With the above processing method, gene sequencing data can be divided into data blocks, and by performing corresponding decompression processing on data with different characteristics in each data block, the compressed gene sequencing data can be easily read.

[0148] The computer-readable storage medium may be provided in the processing system 10 or in other terminals. The processing system 10 may communicate with other terminals to obtain corresponding programs.

[0149] It is understood that computer-readable storage media may include: any entity or device capable of carrying a computer program, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROMs), random access memories (RAMs), and software distribution media, etc. Computer programs include computer program codes. Computer program codes may be in source code form, object code form, executable files, or some intermediate form, etc.

[0150] In certain embodiments of the present invention, the processor 16 may be a central processing unit (CPU), a graphics processing unit (GPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0151] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0153] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments of the present invention without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for processing gene sequencing data, characterized in that: include: Dividing the received gene sequencing data to obtain at least one data block, each of the data blocks including at least one combination of an identifier, a base sequence, and a quality value; In response to the compression signal, the identifier is compressed using a first compression algorithm, the base sequence is compressed using a second compression algorithm, and the quality value is compressed using a third compression algorithm to obtain a data compression block.

2. The processing method according to claim 1, characterized in that: The received gene sequencing data is divided and processed to obtain at least one data block, including: The gene sequencing data is divided and processed so that the size of the data blocks obtained by the division is less than or equal to the preset data amount.

3. The processing method according to claim 1, characterized in that: The processing method comprises: When it is determined that the data amount of a data block obtained through the division process is greater than or equal to the preset data amount, compression processing is performed on the determined data block.

4. The processing method according to claim 1, characterized in that: The identifier is compressed by a first compression algorithm, the base sequence is compressed by a second compression algorithm, and the quality value is compressed by a third compression algorithm, including: The identifier, the comparison result and the quality value are compressed in parallel, and the data compression block is obtained according to the result of the parallel compression processing, wherein the comparison result is obtained by comparing the base sequence with a preset reference genome.

5. The processing method according to claim 4, characterized in that: The first compression algorithm includes a dictionary tree algorithm; and / or, The second compression algorithm comprises a range encoding algorithm; and / or, The third compression algorithm comprises a range encoding algorithm.

6. A method for processing gene sequencing data, characterized in that: include: Reading a data compression block to obtain compression results corresponding to the identifier, the base sequence, and the quality value, respectively, wherein the data compression block is obtained by compressing the identifier using a first compression algorithm, compressing the base sequence using a second compression algorithm, and compressing the quality value using a third compression algorithm; The compression results corresponding to the identifier, the base sequence and the quality value are decompressed, and the data block is obtained according to the results after the decompression, the data block includes the identifier, the base sequence and the quality value, and the data block is obtained by dividing the received gene sequencing data.

7. The processing method according to claim 6, characterized in that: The processing method comprises: When at least one of the data blocks is obtained, the gene sequencing data is obtained by combining at least one of the data blocks according to a data index, and the data index is generated when the data compression block is obtained.

8. The processing method according to claim 6, characterized in that: Decompressing the compression results corresponding to the identifier, the base sequence, and the quality value respectively includes: The compression results corresponding to the identifier, the base sequence and the quality value are decompressed in parallel.

9. A gene sequencing data processing system, characterized in that: The system comprises a first host module and a first processing module, wherein the first host module is used for: Dividing the received gene sequencing data to obtain at least one data block, each of the data blocks including at least one combination of an identifier, a base sequence, and a quality value; The first processing module is used for: In response to the compression signal, the identifier is compressed using a first compression algorithm, the base sequence is compressed using a second compression algorithm, and the quality value is compressed using a third compression algorithm to obtain a data compression block.

10. A gene sequencing data processing system, characterized in that: A second processing module is included, wherein the second processing module is used for: Reading a data compression block to obtain compression results corresponding to the identifier, the base sequence, and the quality value, respectively, wherein the data compression block is obtained by compressing the identifier using a first compression algorithm, compressing the base sequence using a second compression algorithm, and compressing the quality value using a third compression algorithm; The compression results corresponding to the identifier, the base sequence and the quality value are decompressed, and the data block is obtained according to the results after the decompression, the data block includes the identifier, the base sequence and the quality value, and the data block is obtained by dividing the received gene sequencing data.

11. A gene sequencing data processing system, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it can implement the steps of the processing method described in any one of claims 1 to 5, and / or can implement the steps of the processing method described in any one of claims 6 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the steps of the processing method described in any one of claims 1 to 5, and / or it can implement the steps of the processing method described in any one of claims 6 to 8.

Images