A gene sequencing data sorting method, integrated circuit and sorting device
By using in-store computing hierarchical sorting method and lossless compression technology in the sorting of gene sequencing data, the problems of processor burden and I/O overhead are solved, and efficient sorting and performance improvement of gene sequencing data are achieved.
Patent Information
- Application Number
- CN202110632396.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-07
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-06-07
AI Technical Summary
In the existing gene sequencing data sorting methods, the processor has a large burden and the I/O overhead between the CPU and the external memory is large, resulting in insufficient performance.
Using in-store calculation method, the internal memory or internal and external memory is determined according to the size of the data to be sorted. Quick sorting algorithm and double-tuning sorting algorithm are used to avoid the processor's direct software sorting and lossless compression. Improve bandwidth utilization of external memory.
It improves the speed and performance of gene sequencing data sorting, reduces processor burden and I/O overhead, and is suitable for efficient sorting of large-capacity gene sequencing data.
Smart Images

Figure CN113257352B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer system structure design and data sorting technology, and in particular to a gene sequencing data sorting method, an integrated circuit and a sorting device. Background Art
[0002] With the rapid development of bioinformatics, gene analysis has become a technical means widely used in scientific research and industry, and has been successfully applied in species identification, disease diagnosis, etc. The basis of gene analysis is gene sequencing technology, and the second-generation sequencing technology is currently widely used.
[0003] The cost of second-generation sequencing is constantly decreasing, which has led to the rapid growth of gene sequencing data. This effect will become more and more obvious, and gene sequencing data will reach an astonishing level in the future. In order to process the massive amount of gene sequencing data, humans need to use modern computing systems to complete a set of gene analysis processes, in which sorting after comparing the gene sequencing data with the reference sequence is an important step.
[0004] Since the gene data to be sorted may be relatively large, and even difficult to read all of it into the memory for calculation, an external sorting method is needed. The currently widely used solution is software external sorting, which uses the processor as the control and calculation unit for sorting, moves the intermediate data between the memory and the hard disk, and merges them to obtain the final sorting result. However, this external sorting solution uses the processor as the processing unit during the sorting process, which will increase the burden on the CPU; and in the traditional architecture, a large amount of I / O overhead will be generated between the processor and the external memory, resulting in performance deficiencies. Summary of the invention
[0005] The technical problem solved by the present invention is to provide a gene sequencing data sorting method, an integrated circuit and a sorting device, aiming to solve the problems of software sorting CPU burden and a large amount of I / O overhead generated between the processor and the external memory, thereby improving the performance of gene sequencing data sorting.
[0006] The basic solution provided by the present invention is:
[0007] A method for sorting gene sequencing data, comprising:
[0008] Reading the storage capacity required for the data to be sequenced to determine the size of the data to be sequenced;
[0009] If the size of the data to be sequenced does not exceed a preset threshold, the data to be sequenced is sorted in the internal memory; if the size of the data to be sequenced exceeds the preset threshold, the data to be sequenced is hierarchically sorted in the internal memory and the external memory;
[0010] After the data to be sequenced are sequenced, the sequencing results of the data to be sequenced are written back to the external memory for storage.
[0011] The principles of the basic solution are:
[0012] In this scheme, the storage capacity of the data to be sequenced is read to determine the size of the data to be sequenced; and the data to be sequenced is compared with a preset threshold. When the size of the data to be sequenced does not exceed the preset threshold, the data to be sequenced is sorted in the internal memory, and the sorting result of the internal sorting is stored in the external memory; when the size of the data to be sequenced exceeds the preset threshold, the data to be sequenced is sorted in the internal memory and the external memory, and the sorting result of the external sorting is stored in the external memory.
[0013] Hierarchical sorting is performed in the internal memory and the external memory, that is, when the storage capacity of the data to be sequenced is large and exceeds the capacity of the internal memory, the data to be sequenced is sorted using a hardware sorting network that interacts between the internal memory and the external memory.
[0014] The beneficial effects of the basic solution are:
[0015] (1) In this scheme, for smaller data to be sequenced, internal storage is used to sort the gene sequencing data, which can improve the speed; for larger data to be sequenced, the storage capacity occupied by it is greater than the internal storage, and the internal storage and external storage are used to hierarchically sort the gene sequencing data. The external storage has a large storage capacity and can store larger gene sequencing data.
[0016] (2) In this solution, since the gene sequencing data sorting engine is based on the calculation in the storage, the gene sequencing data sorting is completed through the hardware network of internal memory, external memory, etc., avoiding the software burden problem caused by software sorting through the processor CPU; at the same time, it also avoids the large amount of I / O overhead caused by the transmission of the sequenced data back and forth between the processor and the memory, thereby improving the performance of gene sequencing data sorting.
[0017] Furthermore, if the size of the data to be sequenced does not exceed a preset threshold, the step of sorting the data to be sequenced in the internal memory includes: sorting the data to be sequenced in the internal memory using a quick sorting algorithm.
[0018] In this solution, when the size of the data to be sequenced does not exceed the preset threshold, a combination of a traditional processor (CPU) and internal memory (DRAM) is used to complete the gene sequencing data sorting, thereby improving the sorting speed of small-capacity gene sequencing data.
[0019] Furthermore, if the size of the data to be sequenced exceeds a preset threshold, the step of sorting the data to be sequenced in the internal memory and the external memory includes: equally dividing the data to be sequenced into a plurality of small blocks of gene sequencing data.
[0020] In this solution, when the size of the data to be sequenced exceeds a preset threshold, the data to be sequenced is equally divided into multiple small blocks of gene sequencing data, and each small block of gene sequencing data is sorted separately, thereby avoiding the problem that the internal storage cannot be read and sorted.
[0021] Furthermore, the step of equally dividing the data to be sequenced into a plurality of small blocks of gene sequencing data is specifically as follows:
[0022] The data to be sequenced has N read pairs, and the N read pairs are equally divided into T parts. Then, the number of read pairs in each equally divided small block of gene sequencing data is N / T.
[0023] For the data to be sequenced, what needs to be sorted are the multiple read pairs contained therein. The data to be sequenced is equally divided according to the read pairs it has, so as to be divided into multiple small blocks of gene sequencing data, so as to facilitate the hierarchical sorting of the multiple small blocks of gene sequencing data in the internal memory and the external memory step by step, so that the sorting process of larger data to be sequenced can be completed on the storage end, thereby improving the performance of gene sequencing data sorting.
[0024] Further, after the step of equally dividing the data to be sequenced into a plurality of small blocks of gene sequencing data, the method includes:
[0025] Lossless compression is performed on each small block of gene sequencing data in the data to be sequenced.
[0026] By losslessly compressing each small block of gene sequencing data, it is possible to achieve that larger data to be sequenced can be directly processed on the compressed data after being equally divided, avoiding the storage bandwidth limitations of internal and external memories when sorting the sequencing data, and improving the bandwidth utilization of the external memory.
[0027] Further, after the step of losslessly compressing each small block of gene sequencing data in the data to be sequenced, the method further comprises:
[0028] Perform bitonal sorting on each small block of gene sequencing data after lossless compression.
[0029] Small blocks of gene sequencing data after lossless compression are sorted through bitonic sorting, which is suitable for hardware implementation, that is, data sorting is completed directly through hardware networks such as internal memory and external memory, avoiding the software burden problem caused by software sorting through processor CPU, etc.; at the same time, it also avoids the large amount of I / O overhead caused by the transmission of the data to be sequenced back and forth between the processor and the memory, thereby improving the performance of gene sequencing data sorting.
[0030] Furthermore, the step of performing bitonal sorting on each small block of gene sequencing data after lossless compression includes:
[0031] Sorting each small block of gene sequencing data, and merging the sorted small block of gene sequencing data into a plurality of medium block gene sequencing data;
[0032] Sorting each of the middle-block gene sequencing data, and merging the sorted middle-block gene sequencing data into a plurality of large-block gene sequencing data;
[0033] The large blocks of gene sequencing data are sorted, and the sorted large blocks of gene sequencing data are merged for output.
[0034] Due to the use of the bitonic sorting algorithm, the gene sequencing data is divided into blocks from small to large, and the purpose of sorting all the gene sequencing data is achieved layer by layer. The bandwidth of storage at all levels is used more effectively, and the situation where sorting cannot be caused by insufficient storage space is avoided, saving storage space. At the same time, the merged large blocks of gene sequencing data are ordered, which is conducive to the successful application of gene sequencing data in species identification, disease diagnosis and other aspects.
[0035] Furthermore, the lossless compression is specifically to encode the repeated information of each small block of gene sequencing data according to a directed acyclic graph.
[0036] In this solution, the repeated information of each small block of gene sequencing data is encoded based on a directed acyclic graph, so that it can be sorted directly on the file in this compressed format, while improving the bandwidth utilization of the external memory.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes an integrated circuit, the integrated circuit comprising: an internal / external sorting judge, an external memory chip, a bitonal sorter, a data merger, and a fast sorting processor connected to the internal / external sorting judge, the external memory chip having a data blocker and a lossless compressor;
[0038] The input end of the data blocker is the input end of the external memory chip, the output end of the data blocker is connected to the input end of the lossless compressor, and the output end of the lossless compressor is the output end of the external memory chip;
[0039] The quick sort processor is connected to an internal memory, the dual-tone sorter is connected to an on-chip buffer and the internal memory, and the data merger is connected to an external memory.
[0040] Since the flash memory chip in the integrated circuit has a data blocker and a lossless compressor, this configuration allows the storage computing unit in the integrated circuit of the present solution to offload the compression step to the programmable hardware logic unit, so that the gene sequencing data is compressed during the storage process, achieving the overlap of data input / output (I / O) and calculation, and reducing the time overhead of switching between gene sequencing data steps; in addition, since the compression process is offloaded by hardware, high-concurrency processing of the compression process can be achieved, further reducing the time overhead caused by data compression and decompression during the switching of traditional gene sequencing processes.
[0041] The present invention further provides a sorting device, the sorting device comprising:
[0042] A memory, a processor, and a gene sequencing data sorting program stored in the memory and executable on the processor, wherein the gene sequencing data sorting program implements the steps of the gene sequencing data sorting method described above when executed by the processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a schematic diagram of a flow chart of an embodiment of a method for sorting gene sequencing data of the present invention;
[0044] Figure 2 A schematic diagram of a gene comparison and analysis module of an embodiment of a gene sequencing data sorting method of the present invention;
[0045] Figure 3 This is a schematic diagram of the flow of an internal / external sorting algorithm involved in an embodiment of a gene sequencing data sorting method of the present invention;
[0046] Figure 4 This is a schematic diagram of the structure of an integrated circuit involved in an embodiment of a gene sequencing data sorting method of the present invention;
[0047] Figure 5 A schematic diagram of the structure of an embodiment of a gene sequencing data sorting engine for in-memory calculation according to the present invention. DETAILED DESCRIPTION
[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0049] The following is further described in detail through specific implementation methods:
[0050] The sorting device in this solution is a terminal device, which is a structure of a hardware operating environment. The sorting device in the embodiment of the present invention can be a terminal device such as a PC, a portable computer, etc.
[0051] The terminal device may include: a processor, a communication bus, a user interface, a network interface, and a memory. The communication bus is used to realize the connection and communication between the processor, the user interface, the network interface, and the memory. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), a handwriting board, a stylus, etc., and the optional user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface (such as an RJ45 interface) and a wireless interface (such as a WIFI interface).
[0052] In the sequencing device of this solution, the user interface is mainly used to communicate data with each terminal; the network interface is mainly used to connect to the backend server and communicate data with the backend server; and the processor can be used to call the gene sequencing data sequencing program stored in the memory and execute the following steps: Figure 1 The following operations are shown:
[0053] Step S100, obtaining the data to be sequenced generated after aligning the Fastq file with the reference sequence;
[0054] Step S200, reading the storage capacity required for the data to be sequenced to determine the size of the data to be sequenced;
[0055] Step S300, if the size of the data to be sequenced does not exceed a preset threshold, the data to be sequenced is sorted in the internal memory; if the size of the data to be sequenced exceeds the preset threshold, the data to be sequenced is hierarchically sorted in the internal memory and the external memory;
[0056] Step S400, after sorting the data to be sequenced, write the sorting results of the data to be sequenced back to the external memory for storage.
[0057] Note: In this embodiment, the data to be sequenced is generated by aligning the Fastq file with the reference sequence, which is referred to as intermediate data in this embodiment.
[0058] In this solution, the gene sequencing data sorting method is a gene sequencing data sorting engine based on in-storage calculation, which performs orderly sorting according to the name or coordinates in the gene sequencing data; specifically, a Fastq file is compared with a reference sequence, or two Fastq files are compared with a reference sequence to generate SAM data to be sequenced. The gene sequencing data in the SAM data to be sequenced is unordered and needs to be sorted in order according to the gene fragment name of the SAM data to be sequenced or the position of the gene fragment on the reference sequence.
[0059] It should be noted that the Fastq file contains many broken gene fragments. The process of aligning the Fastq file with the reference sequence is to find out the positions of these gene fragments on the existing reference sequence in order to obtain the complete sequenced gene sequence.
[0060] In this embodiment, since what is obtained is the data to be sequenced generated after the Fastq file is aligned with the reference sequence, the data to be sequenced is a complete sequenced gene sequence.
[0061] In this implementation, the gene sequencing data sorting method is an adaptive method of internal sorting and external sorting, specifically, whether to use internal sorting or external sorting will be determined according to the size of the data to be sorted. The internal memory in this solution refers to the internal storage DRAM in the traditional architecture, and the external memory refers to the external storage Flash in the hard disk. This design is due to the fact that the storage capacity of gene sequencing data varies, and the intermediate data (SAM file) generated after aligning the gene sequencing data with the reference sequence also varies accordingly. The internal memory used in this solution is relatively small and fast, and the external memory is relatively large and slow. When the storage capacity of the gene sequencing data is small, internal sorting is used, and when the storage capacity of the gene sequencing data is large, external sorting is used. This adaptive method is beneficial to leveraging strengths and avoiding weaknesses.
[0062] Furthermore, the gene sequencing data sorting method is based on the gene sequencing data sorting engine calculated in the storage, and the gene sequencing data is sorted in order according to the name or coordinate in the gene sequencing data. The gene sequencing data sorting method includes internal / external sorting judgment, quick sorting, external sorting data blocking, lossless compression, bitonic sorting and external sorting merging. Since this scheme is the second-generation sequencing (NGS) gene preprocessing, many broken gene fragments are stored in the Fastq file. Before the internal / external sorting judgment, a Fastq file needs to be compared with the reference sequence, or two Fastq files need to be compared with the reference sequence to generate SAM intermediate data. The gene sequencing data in the SAM intermediate data is disordered and needs to be sorted in order according to the gene fragment name of the SAM intermediate data or the position of the gene fragment on the reference sequence. The comparison process is to obtain the position of these broken gene fragments on the existing reference sequence to obtain a complete sequenced gene sequence.
[0063] After obtaining the intermediate data after comparison, the storage capacity of the intermediate data is read to determine the size of the intermediate data. The internal / external sorting judgment determines whether to use internal sorting or external sorting according to the size of the intermediate data. A preset threshold is set dynamically here. When the intermediate data exceeds the preset threshold, external sorting is used. When the intermediate data does not exceed the preset threshold, internal sorting is used. The fast sort is used for internal sorting, and the data sorting is completed by using the traditional processor CPU and internal memory DRAM. The external sorting data block refers to the equal division of the intermediate data with a large amount of data, and the sorting of each small block of gene sequencing data that is equally divided; the bitonic sorting is used for the external sorting. The sorting algorithm is suitable for hardware implementation, so the data sorting is completed by using the hardware sorting network such as internal memory and external memory. The external sorting merge refers to the merging of the sorted small blocks of gene sequencing data into ordered large blocks of gene sequencing data in the case of external sorting. The lossless compression refers to the lossless compression of the small blocks of gene sequencing data in the external memory in the case of external sorting to improve the bandwidth utilization of the external memory. At the same time, the lossless compression algorithm allows sorting to be performed directly on the compressed data.
[0064] In this embodiment, the intermediate data with a large amount of data is equally divided. Specifically, for gene sequencing data (SAM file / Fastq file), what needs to be sorted are read pairs. Assuming that there are N read pairs in an intermediate data, which are equally divided into T parts, the number of read pairs in each small block of gene sequencing data is N / T. Each small block of gene sequencing data is sorted specifically by using a hardware sorting tree for internal sorting of the small block of gene sequencing data, and a hardware merging tree for merging the small blocks of gene sequencing data.
[0065] It should be noted that Fastq is a text format that stores biological sequences (usually nucleic acid sequences) and corresponding quality evaluations, all encoded in ASCII, which is almost the standard format for high-throughput gene sequencing. Internal sorting is sorting in the internal memory, and external sorting is sorting in combination with the internal memory and the external memory, and there is interaction between the internal memory and the external memory. The dynamically set preset threshold can be adaptively set according to the size of the internal memory, so as to distinguish whether to use internal sorting or external sorting according to the size of the intermediate data.
[0066] In one embodiment, the application scenario of this solution is the sorting of gene sequencing data, such as Figure 2As shown in the figure, in the gene analysis process of double-end sequencing, the two Fastq files will be compared with the reference sequence first, which usually generates larger intermediate data (SAM file). At this time, the SAM file is disordered and needs to be sorted according to the gene sequencing data name in the SAM file or the coordinates of the reference sequence to facilitate the analysis of the gene sequencing data. The size of the SAM file to be sorted is positively correlated with the size of the input gene data file. The storage capacity of these gene data files is smaller, such as 2GB, 8GB, 16GB, etc., and the storage capacity is larger, such as 128GB, 256GB, etc.
[0067] The algorithm flow of this scheme is as follows Figure 3 As shown, first, choose whether to use internal sorting or external sorting according to the size of the SAM file generated by the comparison and the size of the actual internal memory of the computing system. For example, when the size of the SAM file to be sorted is 2GB and the size of the internal memory of the computing system is 16GB, it is only necessary to use the internal sorting method for the SAM file to be sorted. At this time, the sorting of the SAM file to be sorted will be directly regarded as an ordinary sorting task, that is, the task is completed by running the fast sorting algorithm on the traditional processor-internal memory (CPU-DRAM) system. When the size of the SAM file to be sorted is 200GB, and the size of the internal memory of the computing system is only 16GB, it is necessary to use the external sorting scheme for the SAM file to be sorted. At this time, the external sorting is an I / O intensive task, and the external memory needs to participate in the sorting process. The double-tuning sorting algorithm is used to divide the SAM file to be sorted into blocks, from small to large, and the SAM file to be sorted is sorted layer by layer.
[0068] In one embodiment, the SAM data to be sorted is stored in an external memory. After equal division and lossless compression, the gene sequencing data blocks generated after equal division in the external memory are sorted. The sorting results will be further merged by the internal memory. Each merger of the internal memory can merge multiple equally divided and sorted gene sequencing data together. There are multiple layers of mergers in the internal memory to merge and sort iteratively; that is, through the dual-tuning sorting including static random access memory SRAM, dynamic random access memory DRAM and external memory Flash, iteratively merge and sort step by step, and hierarchical sorting is performed using memories of different storage capacities, and finally the merged gene sequencing data is written back to the external memory, so as to complete the sorting of disordered gene sequencing data in an external memory. It can be understood that through the step-by-step merging of the tree structure, the complete sorted gene sequencing data is finally obtained on the external memory, so as to improve the bandwidth utilization and save the storage space of the external memory.
[0069] In addition, to achieve the above-mentioned purpose, the present invention also provides an integrated circuit, referring to Figure 4 As shown, the integrated circuit is based on a gene sequencing data sorting engine calculated in the storage, and performs orderly sorting according to the names or coordinates in the gene sequencing data. The integrated circuit includes: an internal / external sorting judge, an external memory chip, a bitonic sorter, a data merger, and a fast sorting processor connected to the internal / external sorting judge in sequence, and the external memory chip has a data blocker and a lossless compressor;
[0070] The input end of the data blocker is the input end of the external memory chip, the output end of the data blocker is connected to the input end of the lossless compressor, and the output end of the lossless compressor is the output end of the external memory chip;
[0071] The quick sort processor is connected to an internal memory, the dual-tone sorter is connected to an on-chip buffer and the internal memory, and the data merger is connected to an external memory.
[0072] The gene sequencing data sorting method described above in the present invention may be run in a sorting device, and the sorting device may include: a memory, a processor, a communication bus, and a gene sequencing data sorting program stored in the memory:
[0073] The communication bus is used to realize the connection and communication between the processor and the memory;
[0074] The processor is used to execute the gene sequencing data sorting program to achieve normal operation of the gene sequencing data sorting system.
[0075] In this embodiment, the integrated circuit is implemented Figure 3 The carrier of the gene data sorting algorithm process shown in . The internal / external sorting judge will select the fast sorting processor or the bitonic sorting device according to the preset threshold and the size of the input SAM file to be sorted; the data blocker and lossless compressor in the external memory chip can block and compress the SAM file to be sorted in the external memory; the blocked and compressed gene sequencing data will be sent to the bitonic sorting device for preliminary sorting, and then the data merger will merge the preliminary sorting results into a larger sorting result, iterating repeatedly, and finally the data merger will send the result to the external memory.
[0076] It should be noted that, referring to Figure 5As shown, the gene sequencing data sorting engine based on in-storage calculation performs orderly sorting according to the name or coordinates in the gene sequencing data. The gene sequencing data sorting engine based on in-storage calculation has a flash memory controller and a flash memory conversion layer. The flash memory controller controls the reading and writing of the external memory Flash, and the flash memory conversion layer processes the conversion of logical addresses and physical addresses and the scheduling of flash memory access; the configurator and the scheduler connected to the flash memory conversion layer, the configurator will receive the size of the SAM file, and write the configuration information obtained by analysis into the integrated circuit, and the scheduler will receive the information that the data blocker divides the SAM file equally, and cooperate with the flash memory conversion layer to control the operation of the gene sequencing data sorting engine based on in-storage calculation; there are hardware execution units for blocking and compression in the flash memory chip; the integrated circuit for gene sequencing data sorting is responsible for completing the actual sorting task.
[0077] Among them, the steps implemented when the gene sequencing data sorting program running on the processor is executed can refer to the embodiment of the gene sequencing data sorting method in the present invention, and will not be repeated here.
[0078] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field are aware of all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known systems should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for technicians in this field, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
Claims
1. A method for sorting gene sequencing data, characterized in that: include: Reading the storage capacity required for the data to be sequenced to determine the size of the data to be sequenced; If the size of the data to be sequenced does not exceed a preset threshold, the data to be sequenced is sorted in the internal memory; if the size of the data to be sequenced exceeds the preset threshold, the data to be sequenced is sorted in the internal memory and the external memory; After sorting the data to be sequenced, writing the sorting results of the data to be sequenced back to the external memory for storage; The step of sorting in the internal memory includes: sorting the data to be sequenced in the internal memory using a quick sorting algorithm; The sorting in the internal memory and the external memory is to sort the data to be sequenced using a hardware sorting network that interacts between the internal memory and the external memory; If the size of the data to be sequenced exceeds a preset threshold, the step of sorting the data to be sequenced in the internal memory and the external memory includes: equally dividing the data to be sequenced into a plurality of small blocks of gene sequencing data.
2. The gene sequencing data sorting method according to claim 1, characterized in that: The specific steps of equally dividing the data to be sequenced into multiple small blocks of gene sequencing data are as follows: The data to be sequenced has N read pairs, and the N read pairs are equally divided into T parts. Then, the number of read pairs in each equally divided small block of gene sequencing data is N / T.
3. The gene sequencing data sorting method according to claim 1, characterized in that: After the step of equally dividing the data to be sequenced into a plurality of small blocks of gene sequencing data, the method further includes: losslessly compressing each small block of gene sequencing data in the data to be sequenced.
4. The gene sequencing data sorting method according to claim 3, characterized in that: After the step of losslessly compressing each small block of gene sequencing data in the data to be sequenced, the following steps are included: Perform bitonal sorting on each small block of gene sequencing data after lossless compression.
5. The gene sequencing data sorting method according to claim 4, characterized in that: The step of performing bitonal sorting on each small block of gene sequencing data after lossless compression comprises: Sorting each small block of gene sequencing data, and merging the sorted small block of gene sequencing data into a plurality of medium block gene sequencing data; Sorting each of the middle-block gene sequencing data, and merging the sorted middle-block gene sequencing data into a plurality of large-block gene sequencing data; The large blocks of gene sequencing data are sorted, and the sorted large blocks of gene sequencing data are merged for output.
6. The gene sequencing data sorting method according to claim 3, characterized in that: The lossless compression specifically encodes the repetitive information of each small block of gene sequencing data according to a directed acyclic graph.
7. An integrated circuit, characterized in that: include: An internal / external sorting judge, an external memory chip, a bitonic sorter, a data merger, and a fast sorting processor connected to the internal / external sorting judge, wherein the external memory chip has a data blocker and a lossless compressor; The input end of the data blocker is the input end of the external memory chip, the output end of the data blocker is connected to the input end of the lossless compressor, and the output end of the lossless compressor is the output end of the external memory chip; The quick sort processor is connected to an internal memory, the dual-tone sorter is connected to an on-chip buffer and the internal memory, and the data merger is connected to an external memory.
8. A sorting device, characterized in that: The sorting device comprises: A memory, a processor, and a gene sequencing data sorting program stored in the memory and executable on the processor, wherein the gene sequencing data sorting program, when executed by the processor, implements the steps of the gene sequencing data sorting method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Memory and hard-disk distributary storage method of data of fast DNA sequencing data analysis platform
CN108197433A
