Gene sequencing data external sorting method and device based on different storage levels

By adopting multi-level memory hierarchical sorting and hardware network processing in the sorting of gene sequencing data, the problem of low efficiency in the prior art is solved, and efficient massive data processing and performance improvement are achieved.

CN113257356BActive Publication Date: 2024-11-29CHINA SCI & TECH WESTERN RES INST OF COMPUTING TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202110633578.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-07
Publication Date
2024-11-29
Estimated Expiration
2041-06-07

AI Technical Summary

Technical Problem

The prior art is inefficient and has a heavy CPU burden in the process of sorting gene sequencing data, and cannot effectively process massive gene sequencing data.

Method used

Using the gene sequencing data out-ordering method based on different storage levels, multi-level hierarchical sorting is performed through the first memory, the second memory and the third memory, and the data sorting is completed using the hardware network to avoid the burden on the processor software, and through lossless compression and dual Adjust the sorting to improve efficiency.

Benefits of technology

It improves the efficiency and performance of gene sequencing data sorting, reduces I/O overhead, and is suitable for large-scale gene sequencing data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113257356B_ABST
    Figure CN113257356B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for external sorting of gene sequencing data based on different storage levels, which belongs to the technical field of computer system structure design and data sorting, and proposes the following scheme: the method for external sorting of gene sequencing data based on different storage levels includes: reading the storage capacity required for the data to be sorted to determine the size of the data to be sorted; if the size of the data to be sorted exceeds a preset threshold, the data to be sorted is hierarchically sorted through a first memory, a second memory, and a third memory; after the data to be sorted is sorted, the sorting result of the data to be sorted is written back to the external memory for storage. The technical scheme of the present invention ensures the performance of gene sequencing data sorting, and at the same time improves the efficiency of gene sequencing data sorting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer system structure design and data sorting technology, and in particular to a gene sequencing data external sorting method and device based on different storage levels. Background Art

[0002] With the rapid development of bioinformatics, gene analysis has become a technical means widely used in scientific research and industry, and has been successfully applied in species identification, disease diagnosis, etc. The basis of gene analysis is gene sequencing technology, and the second-generation sequencing technology is currently widely used.

[0003] The cost of second-generation sequencing is constantly decreasing, which has led to the rapid growth of gene sequencing data. This effect will become more and more obvious, and gene sequencing data will reach an astonishing level in the future. In order to process the massive amount of gene sequencing data, humans need to use modern computing systems to complete a set of gene analysis processes, in which sorting after comparing the gene sequencing data with the reference sequence is an important step.

[0004] Since the gene data to be sorted may be relatively large, and even difficult to read all into the memory for calculation, an external sorting method is needed. The currently widely used solution is software external sorting, which uses the processor as the control and calculation unit of the sorting, moves the intermediate data between the memory and the hard disk, and merges them to obtain the final sorting result. However, this external sorting solution uses the processor as the processing unit during the sorting process, which will increase the burden on the CPU, and the efficiency of this sorting is low. Summary of the invention

[0005] The technical problem solved by the present invention is to provide a method and device for external sorting of gene sequencing data based on different storage levels, thereby ensuring the performance of gene sequencing data sorting and improving the efficiency of gene sequencing data sorting.

[0006] The basic solution provided by the present invention is:

[0007] Gene sequencing data external sorting methods based on different storage levels include:

[0008] Reading the storage capacity required for the data to be sorted to determine the size of the data to be sorted;

[0009] If the size of the data to be sorted exceeds a preset threshold, the data to be sorted is hierarchically sorted through the first memory, the second memory and the third memory;

[0010] After the data to be sorted are sorted, the sorting result of the data to be sorted is written back to the external memory for storage.

[0011] The principles of the basic solution are:

[0012] In this scheme, the external sorting method of gene sequencing data based on different storage levels is a gene sequencing data sorting engine based on internal storage calculation, which performs orderly sorting according to the name or coordinates in the gene sequencing data; specifically, the storage capacity of the data to be sorted is read to determine the size of the data to be sorted; when the size of the data to be sorted exceeds a preset threshold, the data to be sorted is sorted in a multi-level hierarchical manner through the first memory, the second memory and the third memory, and the sorting result of the external sorting is stored in the external memory.

[0013] Hierarchical sorting is performed through the first memory, the second memory and the third memory, that is, when the storage capacity of the data to be sorted is large and exceeds the capacity of the internal memory, the data to be sorted is sorted using a hardware sorting network that interacts between the internal memory and the external memory.

[0014] The beneficial effects of the basic solution are:

[0015] (1) In this scheme, for larger data to be sorted, the storage capacity occupied by which is greater than the internal storage, the first memory, the second memory and the third memory are used to perform multi-level hierarchical sorting. Compared with directly sorting the larger data to be sorted, the multi-level sorting can divide the larger data to be sorted into smaller data and sort them separately. The multiple levels can run in parallel and time-division multiplexing to improve the sorting efficiency of the external sorting. At the same time, due to the large storage capacity of the external storage, it can store larger gene sequencing data, and at the same time improve the performance of the external sorting.

[0016] (2) In this solution, since the gene sequencing data is sorted through a hardware network such as the first memory, the second memory and the third memory, the software burden problem caused by software sorting through a processor is avoided.

[0017] (3) In this solution, since the gene sequencing data is sorted through a hardware network such as the first memory, the second memory, and the third memory, a large amount of I / O overhead caused by the transmission of the data to be sorted back and forth between the processor and the memory is avoided, thereby improving the performance of gene sequencing data sorting.

[0018] Furthermore, the storage capacity of the first memory is smaller than the storage capacity of the second memory, and the storage capacity of the second memory is smaller than the storage capacity of the third memory.

[0019] In this solution, the data to be sorted is hierarchically sorted through multi-level memories with different storage capacities, thereby improving the performance of sorting gene sequencing data.

[0020] Furthermore, the first memory is a static random access memory, the second memory is a dynamic random access memory, and the third memory is an external memory.

[0021] In this solution, memories of different capacities are further limited to static random access memory, dynamic random access memory and external memory, so as to better divide the sorting data into hierarchical levels.

[0022] Further, if the size of the data to be sorted exceeds a preset threshold, the step of hierarchically sorting the data to be sorted through the first memory, the second memory and the third memory includes:

[0023] The data to be sorted is equally divided into a plurality of first gene sequencing data via a first memory;

[0024] Performing lossless compression on each first gene sequencing data in the data to be sequenced respectively;

[0025] The first gene sequencing data after lossless compression are sorted in a bitonal manner.

[0026] In this solution, when the size of the data to be sorted exceeds a preset threshold, the data to be sorted is equally divided into multiple first gene sequencing data through the first memory, that is, the data to be sorted is equally divided into multiple first gene sequencing data through static random access storage, and each first gene sequencing data is losslessly compressed, and the losslessly compressed first gene sequencing data is sorted through bitonic sorting, and bitonic sorting is suitable for hardware implementation, that is, the sorting of the first gene sequencing data is completed directly through the first memory, which improves the speed of sorting the gene sequencing data after blocking, avoids the storage bandwidth limitation of the memory when sorting the data to be sorted, and improves the bandwidth utilization of the external memory.

[0027] Furthermore, the step of performing bitonal sorting on each first gene sequencing data after lossless compression includes:

[0028] The sorted first gene sequencing data are combined into a plurality of second gene sequencing data according to a tree structure through a second memory;

[0029] The merged second gene sequencing data are sorted by bitonic order.

[0030] After the step of performing bitonal sorting on the merged second gene sequencing data, the method further includes:

[0031] The sorted second gene sequencing data are combined into a plurality of third gene sequencing data according to a tree structure via a third memory;

[0032] The merged third gene sequencing data are sorted by bitune.

[0033] After the step of performing bitonal sorting on the merged third gene sequencing data, the following steps are performed:

[0034] The sorted third gene sequencing data are merged into ordered final gene sequencing data according to the tree structure and written back to the external memory.

[0035] In this solution, the sorted first gene sequencing data are merged according to the tree structure through the dynamic random access memory, and after the merger, the multiple second gene sequencing data are bi-tonically sorted based on the hardware network. The sorted second gene sequencing data are further merged according to the tree structure through the external memory, and after the merger, the multiple third gene sequencing data are bi-tonically sorted based on the hardware network. Compared with directly sorting the blocked gene sequencing data through the external memory, the hardware overhead is less, and it is convenient to work in parallel, thereby improving the efficiency of gene sequencing data sorting.

[0036] Furthermore, the step of equally dividing the data to be sorted into a plurality of first gene sequencing data is specifically as follows:

[0037] The data to be sorted has N read pairs, and the N read pairs are equally divided into T parts. Then, the number of read pairs in each equally divided first gene sequencing data is N / T.

[0038] For the data to be sorted, there are multiple read pairs in the data to be sorted, and the data to be sorted is evenly divided according to the read pairs it has, so as to be divided into multiple first gene sequencing data, so as to facilitate hierarchical external sorting of the multiple first gene sequencing data, so that the sorting process of larger data to be sorted can be completed on the storage end, thereby improving the performance of gene sequencing data sorting.

[0039] Furthermore, the lossless compression is specifically to encode the repeated information of each first gene sequencing data according to a directed acyclic graph.

[0040] Based on the directed acyclic graph, the repeated information of each first gene sequencing data is encoded, that is, the repeated information in the first gene sequencing data is compressed, thereby reducing the size of the first gene sequencing data. At the same time, this method can be used to directly sort the files in this compressed format, thereby improving the bandwidth utilization of the external memory.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a gene sequencing data external sorting device based on different storage levels, and the gene sequencing data sorting device comprises:

[0042] A memory, a processor, and a program for external sorting of gene sequencing data based on different storage levels stored in the memory and executable on the processor, wherein the program for external sorting of gene sequencing data based on different storage levels, when executed by the processor, implements the steps of the method for external sorting of gene sequencing data based on different storage levels as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of an embodiment of a method for external sorting of gene sequencing data based on different storage levels of the present invention;

[0044] Figure 2 A schematic diagram of a module for gene comparison analysis of an embodiment of a gene sequencing data external sorting method based on different storage levels of the present invention;

[0045] Figure 3 A schematic diagram of a circuit for internal / external sorting according to an embodiment of the method for external sorting of gene sequencing data based on different storage levels of the present invention;

[0046] Figure 4 It is a schematic diagram of the external sorting hierarchical sorting structure involved in an embodiment of the external sorting method of gene sequencing data based on different storage levels of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of block compression in external sorting according to an embodiment of the external sorting method of gene sequencing data based on different storage levels of the present invention;

[0048] Figure 6 A schematic diagram of the structure of an embodiment of a gene sequencing data sorting engine for in-memory calculation according to the present invention. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] The following is further described in detail through specific implementation methods:

[0051] The gene sequencing data external sorting device based on different storage levels in this solution can be a terminal device, which is a structure of a hardware operating environment. The gene sequencing data external sorting device based on different storage levels in the embodiment of the present invention can be a terminal device such as a PC or a portable computer.

[0052] The terminal device may include: a processor, a communication bus, a user interface, a network interface, and a memory. The communication bus is used to realize the connection and communication between the processor, the user interface, the network interface, and the memory. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), a handwriting board, a stylus, etc., and the optional user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface (such as an RJ45 interface) and a wireless interface (such as a WIFI interface).

[0053] In the sequencing device of this solution, the user interface is mainly used for data communication with each terminal; the network interface is mainly used for connecting to the backend server and communicating with the backend server; and the processor can be used to call the gene sequencing data external sequencing program based on different storage levels stored in the memory and execute the following steps: Figure 1 The following operations are shown:

[0054] Step S100, obtaining the data to be sequenced generated after aligning the Fastq file with the reference sequence;

[0055] That is, refer to Figure 2 As shown, before sorting the data to be sorted, it is necessary to compare a Fastq file with a reference sequence, or compare multiple Fastq files with a reference sequence to generate SAM intermediate data. The gene sequencing data in the SAM intermediate data is disordered and needs to be sorted in order according to the gene fragment name in the SAM intermediate data or the position of the gene fragment on the reference sequence. It should be noted that the Fastq file contains many broken gene fragments. The process of comparing the Fastq file with the reference sequence is to obtain the position of these gene fragments on the existing reference sequence to obtain the complete sequenced gene sequence.

[0056] Furthermore, in the gene analysis process of double-end sequencing in this scheme, the two Fastq files will be compared with the reference sequence first, which usually generates larger data to be sorted (SAM file). At this time, the SAM file is disordered and needs to be sorted according to the gene sequencing data name in the SAM file or the coordinates of the reference sequence to facilitate the analysis of the gene sequencing data. The size of the SAM file to be sorted is positively correlated with the size of the input gene data file. The storage capacity of these gene data files is smaller, such as 2GB, 8GB, 16GB, etc., and the storage capacity is larger, such as 128GB, 256GB, etc.

[0057] Step S200, reading the storage capacity required for the data to be sorted to determine the size of the data to be sorted;

[0058] Step S300, if the size of the data to be sorted exceeds a preset threshold, the data to be sorted is hierarchically sorted through the first memory, the second memory and the third memory;

[0059] In this implementation, when the size of the data to be sorted does not exceed a preset threshold, a combination of a traditional processor (CPU) and an internal memory (DRAM) is used to complete the gene sequencing data sorting; if the size of the data to be sorted exceeds the preset threshold, the data to be sorted is hierarchically sorted through a first memory, a second memory, and a third memory.

[0060] Since the storage capacity of gene sequencing data varies, the data to be sorted (SAM files) generated after comparing the gene sequencing data with the reference sequence also varies accordingly. The internal memory used in this solution is relatively small and fast, while the external memory is relatively large and slow. When the storage capacity of gene sequencing data is small, internal sorting is used, and when the storage capacity of gene sequencing data is large, external sorting is used. That is, when the storage capacity of gene sequencing data is large, the first memory, the second memory and the third memory are used for hierarchical sorting. This adaptive method is beneficial to leveraging strengths and avoiding weaknesses.

[0061] The circuit structure of this scheme is as follows Figure 3 As shown, first, choose whether to use internal sorting or external sorting according to the size of the SAM file generated by the comparison and the size of the actual internal memory of the computing system. For example, when the size of the SAM file to be sorted is 2GB and the size of the internal memory of the computing system is 16GB, it is only necessary to use the internal sorting method for the SAM file to be sorted. At this time, the sorting of the SAM file to be sorted will be directly regarded as an ordinary sorting task, that is, the task is completed by running the fast sorting algorithm on the traditional processor-internal memory (CPU-DRAM) system. When the size of the SAM file to be sorted is 200GB, and the size of the internal memory of the computing system is only 16GB, it is necessary to use the external sorting scheme for the SAM file to be sorted. At this time, the external sorting is an I / O intensive task, and the external memory needs to participate in the sorting process. The double-tuning sorting algorithm is used to divide the SAM file to be sorted into blocks, from small to large, and the SAM file to be sorted is sorted layer by layer.

[0062] Furthermore, the storage capacity of the first memory is smaller than the storage capacity of the second memory, and the storage capacity of the second memory is smaller than the storage capacity of the third memory. Specifically, the first memory is a static random access memory, the second memory is a dynamic random access memory, and the third memory is an external memory. By using memories of different storage capacities for hierarchical sorting in this solution, it is easier to better divide the data to be sorted into hierarchical sorting, and at the same time improve the performance of sorting gene sequencing data.

[0063] Step S400: after sorting the data to be sorted, write the sorting result of the data to be sorted back to the external memory for storage.

[0064] Based on the above embodiments, in order to fully explore the performance of external sorting, refer to Figure 4 He Ru Figure 5 As shown in the figure, this scheme uses three storage levels to sort large gene sequencing data externally. The three storage levels are SRAM level, DRAM level and Flash level; the first storage level is the on-chip cache (SRAM), which is also the static random access storage in this scheme; the second storage level is the internal memory (DRAM), which is also the dynamic random access storage in this scheme; the third storage level is the flash memory (Flash), which is also the external memory in this scheme, and the best possible external sorting performance is obtained through a layer-by-layer progressive approach.

[0065] Furthermore, the data to be sorted is initially stored in the Flash. After segmentation and compression, the sorter in the SRAM layer will sort the data blocks generated after the segmentation of the data to be sorted in the Flash. The sorting results will be sent to the merger in the DRAM layer. A merger can merge sorted data blocks in multiple SRAM layers together. There are multiple layers of mergers in the DRAM layer to iteratively merge sorted blocks, and finally write the merged blocks into the Flash, thus completing the sorting of an unordered Flash data block. Figure 5 The figure shows the sorting at the Flash level, where multiple data blocks sorted at the DRAM level are merged step by step through a tree structure, and finally complete sorted gene sequencing data is obtained at the Flash level. It should be noted that the lossless compression involved improves the bandwidth utilization and saves the storage space of the Flash memory.

[0066] In the above implementation, the external sorting method of gene sequencing data based on different storage levels is an adaptive method of internal sorting and external sorting, which specifically refers to whether to use internal sorting or external sorting according to the size of the data to be sorted. The internal memory in this scheme refers to the internal storage DRAM in the traditional architecture, and the external memory refers to the external storage Flash in the hard disk.

[0067] Further, the external sorting method of gene sequencing data based on different storage levels is based on the gene sequencing data sorting engine calculated in the storage, and the gene sequencing data is sorted in order according to the name or coordinate in the gene sequencing data. The external sorting method of gene sequencing data based on different storage levels includes internal / external sorting judgment, quick sorting, external sorting data block, lossless compression, bitonic sorting and external sorting merging. Since this scheme is the second generation sequencing (NGS) gene preprocessing, many broken gene fragments are stored in the Fastq file. Before the internal / external sorting judgment, a Fastq file needs to be compared with the reference sequence, or two Fastq files need to be compared with the reference sequence to generate SAM data to be sorted. The gene sequencing data in the SAM data to be sorted is disordered and needs to be sorted in order according to the gene fragment name of the SAM data to be sorted or the position of the gene fragment on the reference sequence. The comparison process is to obtain the position of these broken gene fragments on the existing reference sequence to obtain a complete sequenced gene sequence.

[0068] After obtaining the aligned data to be sorted, the storage capacity of the data to be sorted is read to determine the size of the data to be sorted, wherein the internal / external sorting judgment determines whether to use internal sorting or external sorting according to the size of the data to be sorted, and a preset threshold is set dynamically here. When the data to be sorted exceeds the preset threshold, external sorting is adopted, and when the data to be sorted does not exceed the preset threshold, internal sorting is adopted; wherein the quick sorting is used for the internal sorting, and the data sorting is completed by the traditional processor CPU and internal memory DRAM. wherein the external sorting data block refers to the equal division of the data to be sorted with a large amount of data, and the sorting of each first gene sequencing data that is equally divided; wherein the bitonic sorting is used for the external sorting, and the sorting algorithm is suitable for hardware implementation, so the data sorting is completed by the hardware sorting network such as internal memory and external memory. wherein the external sorting merge refers to the merging of the sorted first gene sequencing data into the ordered final gene sequencing data in the case of external sorting. wherein the lossless compression refers to the lossless compression of the first gene sequencing data in the external memory in the case of external sorting, so as to improve the bandwidth utilization of the external memory, and the lossless compression algorithm allows the sorting to be performed directly on the compressed data.

[0069] In this embodiment, the data to be sorted with a large amount of data is equally divided. Specifically, for gene sequencing data (SAM file / Fastq file), what needs to be sorted are read pairs (reads). Assuming that there are N read pairs in a data to be sorted, which are equally divided into T parts, the number of read pairs in each first gene sequencing data is N / T. Each first gene sequencing data is sorted specifically by using a hardware sorting tree for internal sorting of the first gene sequencing data, and a hardware merging tree for merging each first gene sequencing data.

[0070] It should be noted that Fastq is a text format that stores biological sequences (usually nucleic acid sequences) and corresponding quality evaluations, all encoded in ASCII, which is almost the standard format for high-throughput gene sequencing. Internal sorting is sorting in the internal memory, and external sorting is sorting in combination with the internal memory and the external memory, with interaction between the internal memory and the external memory. The dynamically set preset threshold can be adaptively set according to the size of the internal memory, so as to distinguish whether to use internal sorting or external sorting according to the size of the data to be sorted.

[0071] In one embodiment, referring to Figure 3 As shown, the gene sequencing data external sorting device based on different storage levels performs orderly sorting according to the name or coordinate in the gene sequencing data according to the gene sequencing data sorting engine calculated in the storage, which can be a dedicated integrated circuit, including: an internal / external sorting judge, an external memory chip, a bitonic sorter, a data merger, and a fast sorting processor connected to the internal / external sorting judge, wherein the external memory chip has a data blocker and a lossless compressor;

[0072] The input end of the data blocker is the input end of the external memory chip, the output end of the data blocker is connected to the input end of the lossless compressor, and the output end of the lossless compressor is the output end of the external memory chip;

[0073] The quick sort processor is connected to an internal memory, the dual-tone sorter is connected to an on-chip buffer and the internal memory, and the data merger is connected to an external memory.

[0074] In this embodiment, the external memory chip, i.e. Figure 3 The flash memory chip in the embodiment has a data blocker and a lossless compressor. This configuration enables the storage computing unit in the integrated circuit of the present invention to offload the compression step to the programmable hardware logic unit, so that the gene sequencing data is compressed during the storage process, thereby achieving the overlap of data input / output (I / O) and computing, and reducing the time overhead of switching between gene sequencing data steps. In addition, since the compression process is offloaded by hardware, high-concurrency processing of the compression process can be achieved, further reducing the time overhead caused by data compression and decompression during the switching of traditional gene sequencing processes.

[0075] The above-mentioned gene sequencing data external sorting method based on different storage levels in the present invention may be run in a gene sequencing data external sorting device based on different storage levels. The gene sequencing data external sorting device based on different storage levels may include: a memory, a processor, a communication bus, and a gene sequencing data external sorting program based on different storage levels stored in the memory:

[0076] The communication bus is used to realize the connection and communication between the processor and the memory;

[0077] The processor is used to execute the external sorting program of gene sequencing data based on different storage levels to achieve the normal operation of the above-mentioned external sorting device of gene sequencing data based on different storage levels.

[0078] In this embodiment, the ASIC is implemented Figure 2 The carrier of the gene data sorting algorithm process shown in . The internal / external sorting judge will select the fast sorting processor or the bitonic sorting device according to the preset threshold and the size of the input SAM file to be sorted; the data blocker and lossless compressor in the external memory chip can block and compress the SAM file to be sorted in the external memory; the blocked and compressed gene sequencing data will be sent to the bitonic sorting device for preliminary sorting, and then the data merger will merge the preliminary sorting results into a larger sorting result, iterating repeatedly, and finally the data merger will send the result to the external memory.

[0079] It should be noted that, referring to Figure 6 As shown, the gene sequencing data sorting engine based on in-storage calculation performs orderly sorting according to the name or coordinates in the gene sequencing data. The gene sequencing data sorting engine based on in-storage calculation has a flash memory controller and a flash memory conversion layer. The flash memory controller controls the reading and writing of the external memory Flash, and the flash memory conversion layer processes the conversion of logical addresses and physical addresses and the scheduling of flash memory access; the configurator and the scheduler connected to the flash memory conversion layer, the configurator will receive the size of the SAM file, and write the configuration information obtained by analysis into the integrated circuit, and the scheduler will receive the information that the data blocker divides the SAM file equally, and cooperate with the flash memory conversion layer to control the operation of the gene sequencing data sorting engine based on in-storage calculation; there are hardware execution units for blocking and compression in the flash memory chip; the integrated circuit for gene sequencing data sorting is responsible for completing the actual sorting task.

[0080] Among them, the steps implemented when the gene sequencing data sorting program running on the processor is executed can refer to the embodiment of the gene sequencing data external sorting method based on different storage levels in the present invention, and will not be repeated here.

[0081] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field are aware of all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known systems should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for technicians in this field, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for sorting gene sequencing data based on different storage levels, characterized in that: include: Reading the storage capacity required for the data to be sorted to determine the size of the data to be sorted; If the size of the data to be sorted exceeds a preset threshold, the data to be sorted is hierarchically sorted through the first memory, the second memory and the third memory; The gene sequencing data is sorted by using a hardware network, wherein the hardware network includes a first memory, a second memory, and a third memory. Specifically, the gene sequencing data is sorted by using the first memory, the second memory, and the third memory; After sorting the data to be sorted, writing the sorting result of the data to be sorted back to the external memory for storage; the storage capacity of the first memory is smaller than the storage capacity of the second memory, and the storage capacity of the second memory is smaller than the storage capacity of the third memory; The first memory is a static random access memory, the second memory is a dynamic random access memory, and the third memory is an external memory; If the size of the data to be sorted exceeds a preset threshold, the step of hierarchically sorting the data to be sorted through the first memory, the second memory and the third memory includes: The data to be sorted is equally divided into a plurality of first gene sequencing data via a first memory; Performing lossless compression on each first gene sequencing data in the data to be sequenced respectively; performing bitonal sorting on each first gene sequencing data after lossless compression; The step of performing bitonal sorting on each first gene sequencing data after lossless compression includes: The sorted first gene sequencing data are combined into a plurality of second gene sequencing data according to a tree structure through a second memory; Performing bitonal sorting on each merged second gene sequencing data, specifically, performing bitonal sorting on the plurality of second gene sequencing data based on the hardware network after the merging; After the step of performing bitonal sorting on the merged second gene sequencing data, the method further includes: The sorted second gene sequencing data are combined into a plurality of third gene sequencing data according to a tree structure via a third memory; The merged third gene sequencing data are subjected to bitonal sorting. Specifically, after the merging, the plurality of third gene sequencing data are subjected to bitonal sorting based on the hardware network.

2. The method for external sorting of gene sequencing data based on different storage levels according to claim 1, characterized in that: After the step of performing bitonal sorting on the merged third gene sequencing data, the following steps are performed: The sorted third gene sequencing data are merged into ordered final gene sequencing data according to the tree structure and written back to the external memory.

3. The method for external sorting of gene sequencing data based on different storage levels according to claim 1, characterized in that: The step of equally dividing the data to be sorted into a plurality of first gene sequencing data is specifically as follows: The data to be sorted has N read pairs, and the N read pairs are equally divided into T parts. Then, the number of read pairs in each equally divided first gene sequencing data is N / T.

4. The method for external sorting of gene sequencing data based on different storage levels according to claim 1, characterized in that: The lossless compression specifically encodes the repeated information of each first gene sequencing data according to a directed acyclic graph.

5. The method for external sorting of gene sequencing data based on different storage levels according to claim 1, characterized in that: The external memory is a flash memory.

6. A gene sequencing data external sorting device based on different storage levels, characterized in that: The gene sequencing data sorting device comprises: A memory, a processor, and a program for external sorting of gene sequencing data based on different storage levels stored in the memory and executable on the processor, wherein the program for external sorting of gene sequencing data based on different storage levels, when executed by the processor, implements the steps of the method for external sorting of gene sequencing data based on different storage levels as described in any one of claims 1 to 5.

7. The device for externally sorting gene sequencing data based on different storage levels according to claim 6, characterized in that: The gene sequencing data sorting engine is based on the calculation in the storage, and the gene sequencing data is sorted in order according to the names in the gene sequencing data. It is a dedicated integrated circuit, including: an internal / external sorting judge, an external memory chip, a bitonic sorter, a data merger, and a fast sorting processor connected to the internal / external sorting judge.

8. The device for external sorting of gene sequencing data based on different storage levels according to claim 7, characterized in that: The external memory chip has a data blocker and a lossless compressor.

9. The device for external sorting of gene sequencing data based on different storage levels according to claim 6, characterized in that: The gene sequencing data sorting engine with in-storage computing has a flash memory controller and a flash memory conversion layer. The flash memory controller reads and writes the external memory Flash, and the flash memory conversion layer handles the conversion of logical addresses and physical addresses and the scheduling of flash memory access.

10. The device for external sorting of gene sequencing data based on different storage levels according to claim 9, characterized in that: The configurator and scheduler are connected to the flash conversion layer. The configurator receives the size of the SAM file and writes the configuration information obtained through analysis into the integrated circuit. The scheduler receives the information that the data blocker divides the SAM file equally and cooperates with the flash conversion layer to control the operation of the gene sequencing data sorting engine calculated in the storage. The flash memory chip has a hardware execution unit for blocking and compression. The integrated circuit for sorting gene sequencing data is responsible for completing the actual sorting task.

Citation Information

Patent Citations

  • Memory and hard-disk distributary storage method of data of fast DNA sequencing data analysis platform

    CN108197433A

  • Sequencing data storage method, device and equipment and computer readable storage medium

    CN111261227A

  • Data restructuring in multi-level memory hierarchies

    US20110055232A1

  • Method and system for merging alignment and sorting to optimize

    WO2020182175A1