Methods and systems for generating target fastq files from indeterminate fastq files generated from ngs

Abstract

The application provides a method and system for generating a target FastQ file from an NGS generated undetermined FastQ file, and a computer readable storage medium and terminal. The method comprises the following steps: S1, preparing an original file and Index information to obtain the NGS generated undetermined FastQ file; S2, generating and correcting a segmentation point; S3, decompressing, filtering and compressing; and S4, merging. The system comprises a database unit for storing original data generated by NGS, sample information and data generated after analysis; a processing unit comprising a segmentation point generating and correcting module, a decompression filtering and compression module and a merging module; and the database unit communicates with the processing unit. The application can quickly and accurately filter a FastQ file matched with a barcode provided by a customer from an NGS generated undetermined FastQ file, and is very practical.

Description

TECHNICAL FIELD

[0001] The present application relates to a method and system for processing next-generation sequencing (NGS) data, in particular to a method and system for generating a target FastQ file from an undetermined FastQ file generated by NGS BACKGROUND

[0002] With the maturity and popularity of next-generation sequencing (NGS) technology, the cost of sequencing has been greatly reduced, while the amount of data generated is very large. For example, the Nova seq 6000 sequencer can generate 3.6Tb x 2 data at a time, which is equivalent to 2000 times the human genome. In recent years, more and more biological macromolecules are sequenced by NGS in the field of biomedicine. Generally, biological macromolecules, genomes or metagenomes are provided by biological pharmaceutical companies, universities or research institutes to professional sequencing companies for detection by NGS.

[0003] In order to facilitate the publication and sharing of sequencing data, the bases and their quality score data measured by NGS are mostly stored in FASTQ format (corresponding file FastQ file). FASTQ format is a common gene sequencing data, stored in units of reads, carrying quality information of base sequencing, and is the most common format for downstream analysis software.

[0004] The undetermined FastQ file generated by NGS is a sequencing result that the sequence tag (called barcode in NGS) in the first split result of NGS cannot match the barcode in the samplesheet. Some customers want to obtain this part of data. However, due to the design of the current splitting process, directly copying the Undertermined FastQ file from the splitting result is not "clean", which may export other customers' data information, causing risks to the enterprise.

[0005] Therefore, before delivery, the file needs to be filtered to exclude two types of data:

[0006] 1) Exclude known barcodes.

[0007] 2) In the case of non-pipeline (called Lane in NGS), exclude other customers' barcodes.

[0008] There are many cases where the barcode cannot be matched, such as when the customer extracts data and misses 1 barcode, resulting in all sequences marked by the barcode being attributed to the NGS generated Undetermined FastQ file. Due to the large size of the Undetermined FastQ file, the time required for direct filtering is not within an acceptable range. Therefore, it is necessary to split the Undetermined FastQ file into several small fragments before filtering. The present application describes the process required to filter this file.

[0009] The amount of data generated by NGS is very large, and usually sequencing companies will store sequencing data on public clouds (Ali, AWS, Tencent, etc.) for customers to download themselves; without deploying a box, the customer needs a dedicated computer and installs a special download tool corresponding to the public cloud, with a large bandwidth network access, manual confirmation of whether sequencing data has been generated, and manual addition of the download process after the data generation is complete. If the transmission service is interrupted due to network or other reasons, it needs to be restarted manually, which involves a lot of manpower, and the integrity of the data cannot be protected. Due to the large amount of data delivered, it takes a long time to complete the data download, and any other reason during this period can cause data transmission interruption, incomplete files, and errors in subsequent bioinformatics processing steps. Therefore, the current sequencing data delivery has the shortcomings of large data volume, complex manual download management, long time-consuming, etc. How to provide a dedicated software system for high-speed delivery of sequencing data for scientific research institutions, universities, hospitals, medical institutions, pharmaceutical factories, and genetic companies to solve the last mile problem of genetic data delivery is a technical problem that needs to be solved in this field. SUMMARY

[0010] In order to quickly and accurately filter the target FastQ file matching the barcode provided by the customer from the NGS generated Undetermined FastQ file, the present application discloses a method for generating a target FastQ file from an NGS generated Undetermined FastQ file, comprising the following steps:

[0011] S1, preparing original files and Index information to obtain the NGS generated Undetermined FastQ file;

[0012] S2, generating and correcting split points: cutting the NGS generated undetermined FastQ file into several FastQ fragments according to file size R; for the boundary point of each FastQ fragment obtained by cutting, the boundary search of bgzf needs to be carried out within ±M range; and using the offset obtained by searching as the actual split point, the FastQ fragment is split to obtain the split FastQ fragment; R is 0.01-20 Gb, and M is 32-128 Kb;

[0013] S3, decompression, filtering and compression: decompressing the split FastQ fragment obtained in step S2, filtering with target barcode(s), and then compressing to obtain filtered FastQ fragment;

[0014] S4, merging: re-merging the filtered FastQ fragment obtained in step S3 in order to generate the target FastQ file. Wherein, R is a real number Gb in 0.01-20 Gb, and is not limited to two decimal places. R is determined by the specific scene, and is a number obtained by the program from the actual environment such as cpu frequency, memory size and io speed. Generally, the better the computer performance, the smaller the value of R given by the program.

[0015] Further, R is 2-8 Gb, and M is 32-64 Kb. Preferably, R is 4 Gb, and M is 64 Kb.

[0016] In some embodiments, step S1 is specifically: separating data from the original file according to known Index information, and the data not separated from the original file is the NGS generated undetermined FastQ file. The obtaining of the NGS generated undetermined FastQ file generally involves the following work: extracting the first split result directory for extracting the NGS generated undetermined FastQ file and all known Indexes of the NGS generated undetermined FastQ file in the Lane during sequencing from the order task and the on-machine task respectively; selecting the Lane to be split in the work order system, and the source I7 / I5 length of the NGS generated undetermined FastQ file. Index, i.e. barcode, is a label used to identify sequence source information and the like in sequencing. I5 and I7 refer to biological labels of each DNA fragment, which are added to both ends of the DNA fragment. If it is double-end sequencing, I5 and I7 need to be added to both ends of the DNA, and if it is single-end sequencing, only I7 needs to be added. If I7 or I5 and I7 are not added to both ends of the DNA, it is impossible to distinguish which sample each DNA belongs to. I5, I7 is a biological label, and Index is its specific value.

[0017] In some embodiments, the decompression, filtering, and compression in step S3 are completed in a stream in the same task, and the filtered FastQ fragments are stored in files ending with.part-${shard number} in the Aliyun Object Storage Service (OSS).

[0018] In some embodiments, step S3 includes placing the three processes of decompression, filtering, and compression in three different threads, respectively, and connecting them through a queue.

[0019] In some embodiments, step S3 further includes running a plurality of workers on each batch computing node, each worker being composed of three different threads, respectively, responsible for decompression, filtering, and compression; and the number of workers = number of virtual machine cores / 2.

[0020] In some embodiments, step S4 further includes generating an md5 check along with the target FastQ file, for verifying the authenticity of the target FastQ file.

[0021] In some embodiments, the merging in step S4 is completed in a local computer.

[0022] In some embodiments, after completing the merging in the local computer, step S4 further includes not using multi-shard to upload to the cloud.

[0023] In some embodiments, after completing the merging in the local computer, step S4 further includes using multi-shard to upload to the Aliyun.

[0024] In some embodiments, S5, data delivery, is further included: the delivery of the target FastQ file (i.e., delivering the target FastQ file to the customer) is completed through a box delivery service system; the box delivery service system includes a box service network, a box cluster, and a zookeeper service module, the box service network has a plurality of server nodes, the box cluster includes a plurality of box bodies, each box body is independently deployed and connected to the box service network, the box service network is connected to the work order system through wireless connection to realize data interaction, and the zookeeper service module is used to manage the state of the box service network. When the box body needs to communicate with the work order system, it is not connected to a certain work order system, but is connected to the box service network, and then the work order system is connected to the box service network to perform data transmission / box management, etc.

[0025] The box delivery service system, wherein the server node is a CubeMeshNode, and the box body is a CubeNode.

[0026] The box delivery service system, wherein the server node comprises a remote procedure call service module (RPCServer), a cluster state management module (ClusterKeeper), a first metadata module (MetaData), a box runtime information module (RuntimeCubeInfo), a box state management module (CubeKeeper), a box state heartbeat detection module (CubeKeepAliveWorker), a box connection resolution module (CubeConnectionResolver), and a box message communication protocol library module (CubeTalk Message Library).

[0027] The remote procedure call service module (RPCServer) is an entrance for the work order system to remotely access the box.

[0028] The cluster state management module (ClusterKeeper) is used for maintaining state data between the server node and the zookeeper service module, and registering the server node into the box service network.

[0029] The first metadata module (MetaData) stores a meta library required by the box service network.

[0030] The box runtime information module (RuntimeCubeInfo) is used for abstracting an online box state information.

[0031] The box state management module (CubeKeeper) is used for maintaining a connection state with a box.

[0032] The box state heartbeat detection module (CubeKeepAliveWorker) is used for polling a box at a low frequency to avoid a zombie box caused by an inactive connection.

[0033] The box connection resolution module (CubeConnectionResolver) is used as a server of box communication, and is responsible for establishing a connection with a box and communication encoding and decoding.

[0034] The box message communication protocol library module (CubeTalk Message Library) is a protocol library of box communication, and stores all message type data that can be used by a box.

[0035] The box is used for receiving data push from the work order system, and managing the storage device connected to the box, and outputting data into the storage device, the box contains metadata storage module (Storage Metadata), network node connection module (MeshConnector), volume space usage check thread module (VolumeSpaceUsageCheckThread), disk operation thread module (DiskManipulatingThread), disk state monitoring module (DiskWatcher), data download and release module (DownloadExtractor), second metadata module (Metadata), large block data reading module (BLOBReader), download module (Downloader), hard disk initialization module (FSMakerchusi), box communication protocol module (Cube Talk Protocol);

[0036] The metadata storage module (Storage Metadata) is used for managing the metadata structure of the local storage device;

[0037] The network node connection module (MeshConnector) is a client for box communication, responsible for communication establishment and coding and decoding with the box service network;

[0038] The volume space usage check thread module (VolumeSpaceUsageCheckThread) is responsible for monitoring the remaining capacity change of the available partition;

[0039] The disk operation thread module (DiskManipulatingThread) is responsible for processing the management work of the local disk, including mounting, ejecting, and initializing;

[0040] The disk state monitoring module (DiskWatcher) is responsible for monitoring the change of the storage device, such as inserting a new disk, etc;

[0041] The data download and release module (DownloadExtractor) is responsible for outputting an already completed or partially completed download to the external storage device;

[0042] The second metadata module (Metadata) stores the metadata structure associated with the business in the memory;

[0043] The BLOBReader is used to provide the reading ability of the downloaded data, and the data of a single download is integrated into a blob in order to control the number of file handles of the download. The BLOBReader is needed to correctly read the downloaded content;

[0044] The Downloader is a downloader used to perform the download action for a download. The Downloader itself does not perform the connection data transmission and file writing work of the download, but is responsible for controlling the three components to complete the download.

[0045] The FragDownloader is used to split the file into data blocks after obtaining the https download address of the file without content, and the Downloader creates multiple FragDownloaders to download the data blocks in parallel through a local queue.

[0046] The FragCommitter is used to write the completed frag into the local disk according to the offset.

[0047] The ContentDownloader is used to directly obtain the content of the file from the box service network for the file with content.

[0048] The FSMakerchusi is used to provide the disk initialization ability, and multiple implementations are provided to support different types of file partition tables and partition formats.

[0049] The CubeTalk Protocol stores the network protocol used for communication between the box and the box service network.

[0050] The above box delivery service system, wherein the communication protocol between the RPCServer and the ticket system is the thrift protocol.

[0051] The first metadata module (MetaData) stores the meta library including box information (CubeInfo), hard disk information (HDDInfo), download information (Download) and release information (Extraction), wherein the box information (CubeInfo) is the registration information of the box, the hard disk information (HDDInfo) is a known hard disk, the download information (Download) is a download task, and the release information (Extraction) is the process data of outputting data to an external storage;

[0052] The metadata structure in the metadata storage module (Storage Metadata) includes block information (BlockInfo), disk information (DiskInfo), volume information (VolumeInfo), disk flag (DiskFlag) and internal storage disk (InternalStorageDisk), wherein the block information (BlockInfo) is an abstract of all Linux block device, including those which cannot be identified, the disk information (DiskInfo) is an abstract of a storage device, the volume information (VolumeInfo) is an abstract of a partition on a disk, the disk flag (DiskFlag) is an abstract of a partition flag on a disk partition table, and the internal storage disk (InternalStorageDisk) is used for abstracting and marking a disk information (DiskInfo) as internal storage, the internal storage can be used for receiving data push from a work order system, and the remaining disks can be used for outputting the data after push;

[0053] The metadata structure in the second metadata module (Metadata) includes download (Download), download record (DownloadEntry), fragment (Frag), EntryState (download state), download task state (DownloadState), extraction (Extraction), extraction state (ExtractionEntry), the download (Download) is abstract one data push, the download record (DownloadEntry) is abstract one downloaded file, the downloaded file is divided into content and no content, the fragment (Frag) is a downloaded file without content, the fragment (Frag) abstracts a piece of the downloaded record (DownloadEntry) after segmentation, the EntryState (download state) is abstract one download record (DownloadEntry) in the process of downloading, the download task state (DownloadState) is abstract one download state, the extraction (Extraction) is abstract one data output process, and the extraction state (ExtractionEntry) is abstract one file released in the data output process.

[0054] The box delivery service system, wherein the network protocol comprises a negotiation phase and a communication phase.

[0055] The negotiation phase is used to establish a link between the box and the service network, whether it is an uplink or a downlink, and its process is the same. The process of the negotiation phase is as follows:

[0056] Step A1: the box establishes a TCP (Transmission Control Protocol) connection with the box service network, and sends the protocol version of the box to the box service network;

[0057] Step A2: the box service network replies the protocol version to the box;

[0058] Step A3: the box responds to whether the protocol version is compatible;

[0059] If incompatible, the box service network will disconnect the connection;

[0060] If compatible, the box service network generates an RSA key, and sends the RSA public key length and the RSA public key in the RSA key to the box;

[0061] Step A4: the box generates an identifier, which contains the device ID of the box itself and the connection type, then uses the read RSA public key to encrypt the identifier, and sends the RSA public key length and the identifier RSA key to the box service network;

[0062] Step A5: After the box service network decrypts the identification using the RSA private key in the RSA key, it checks the registration information of the device in the identification and sends the identification result to the box;

[0063] Step A6: If the result is not identification completion, the box will disconnect the network;

[0064] Step A7: The box service network sends a negotiation completion identification;

[0065] Step A8: Enter the communication stage;

[0066] After the negotiation stage is completed, the link enters this stage. This stage is in the form of a question and answer, and is in an infinite loop. In this stage, the communication behavior of the box and the service network is mutual. The process of the communication stage is as follows:

[0067] Step B1: Read 4 bytes from the connection between the box and the box service process;

[0068] Step B2: Identify the 4 bytes read to obtain the message length M;

[0069] Step B3: Read data of length M bytes from the connection;

[0070] Step B4: Decode the M length byte data using AES. The key used by the service network for decoding is the key used by the box when registering with the service network. The key used by the box for decoding is the local configuration key. For the same box, the two keys must match;

[0071] Step B5: Process the received message (message) after decoding;

[0072] Step B6: Repeat steps B1-B5 for the next cycle.

[0073] When accepting data pushed by the work order system, the box needs to go through the following stages:

[0074] Create a download task: The work order system creates a Download for the box through the RPC service of the box service network. The box cannot see the original OSS key of each file, but can know the ID and file name of each file;

[0075] Get the download address: The box initiates a request to the service network to obtain the address of a file. The box uses the file ID to identify the file that needs to be obtained. The service network finds the file requested by the box and generates a https download address that does not contain the access secret through the object signature function of Aliyun;

[0076] Splitting data: after the box gets the download URL, the http range request splits the data into small pieces;

[0077] Download and assemble: the box downloads the split data pieces through the built-in FragDownloader (fragment download) and gives them to the Frag Committer for merging, and then writes into the final BLOB, and when all the pieces are downloaded, the download of the single file is declared complete.

[0078] In some embodiments, S5, data delivery, is also included: the delivery of the target FastQ file is completed through the box delivery service method; the box delivery service method comprises the following steps:

[0079] Step C1: connect the box to the Internet;

[0080] Step C2: configure the information related to the box and initialize the hard disk;

[0081] Step C3: after the box is connected to the network, it is automatically registered to the box service network;

[0082] Step C4: when data is generated, the box service network notifies the box to start the data download task.

[0083] The above box delivery service method, wherein, in step C1, after the box is connected to the Internet, PPOE, password, internal network card parameters, and the communication public key of the box need to be configured;

[0084] In step C2, the device two-dimensional code on the box shell is scanned by a mobile phone to enter the configuration interface, and the information related to the box is configured;

[0085] In step C3, after the box is connected to the network, it is automatically registered to the box service network according to the internal preset configuration.

[0086] The above box delivery service method, wherein, step C5: after the data is downloaded, the data can be copied to the place where it is needed to be used through an external mobile hard disk or accessed through the directory where the data of the box is located.

[0087] The information related to the box further includes operator related information, and step C2 further includes binding the operator and the box.

[0088] The application also discloses a system for generating a target FastQ file from an uncertain FastQ file generated by NGS, comprising:

[0089] A database unit is used to store the original data generated by NGS, sample information, and data generated after analysis;

[0090] A processing unit comprises:

[0091] A generating and correcting split point module is configured to split the undetermined FastQ file generated by the NGS into several FastQ segments according to a file size R; for the boundary point of each FastQ segment obtained by the splitting, a boundary search of bgzf is required within a range of ±M; and the offset obtained by the search is used as the actual split point to split the FastQ segment, thereby obtaining the split FastQ segment; R is 1-20 GB, and M is 32-128 KB;

[0092] A decompression filtering compression module is configured to decompress the split FastQ segment, filter with a target barcode (s), and then compress to obtain a filtered FastQ segment;

[0093] A merging module is configured to recombine the filtered FastQ segment in order to generate the target FastQ file;

[0094] The database unit communicates with the processing unit. Wherein, R is a real number Gb in 0.01-20 Gb, and is not limited to two decimal places. R is determined by the specific scene, and is a number obtained by the program from the actual environment such as cpu frequency, memory size, and io speed. Generally, the better the computer performance is, the smaller the value of R given by the program is.

[0095] Further, R is 2-8 Gb, and M is 32-64 Kb. Preferably, R is 4 Gb, and M is 64 Kb.

[0096] In some embodiments, the decompression filtering compression module is further configured to complete the decompression, filtering, and compression in a same task in a stream, and store the filtered FastQ segment generated in a file ending with.part-${segment number} in the Aliyun OSS.

[0097] In some embodiments, the decompression filtering compression module is further configured to put the three processes of decompression, filtering, and compression into three different threads, and connect them through a queue.

[0098] In some embodiments, the decompression filtering compression module is further configured to run a plurality of workers on each batch calculation node, each of the workers is composed of three different threads, and each of the workers is responsible for decompression, filtering, and compression; and the number of the workers = the number of virtual machine kernels / 2.

[0099] In some embodiments, the merging module is further configured to generate an md5 check along with the target FastQ file. The md5 check is used to verify the authenticity of the target FastQ file.

[0100] In some embodiments, the merging module is further configured to complete the merging in the local computer.

[0101] In some embodiments, the merging module is further configured to, after completing the merging in the local computer, upload to the cloud using multi-shard.

[0102] In some embodiments, the merging module is further configured to, after completing the merging in the local computer, upload to the Aliyun using multi-shard.

[0103] In some embodiments, a box delivery service system as described above is further included for completing delivery of the target FastQ file; the box delivery service system is in communication with the database unit; the box delivery service system is in communication with the processing unit.

[0104] The present application also provides a computer readable storage medium, wherein the storage medium stores executable instructions, and the executable instructions, when executed, implement the method for generating a target FastQ file from an Undetermined FastQ file generated by NGS.

[0105] The present application also provides a terminal, comprising:

[0106] a memory for storing executable instructions;

[0107] a processor for executing the executable instructions stored in the memory, and implementing the method for generating a target FastQ file from an Undetermined FastQ file generated by NGS.

[0108] The present application makes it possible to obtain a specific FastQ file from a large amount of Undetermined FastQ files generated by NGS, which is very practical and solves a long-standing problem of sequencing companies, and achieves unexpected technical effects. Through the method and system of the present application, customer data lost due to barcode mismatch can be quickly and accurately found.

[0109] The present application has the following beneficial effects:

[0110] 1. An ultra-large file, i.e. an Undetermined FastQ file generated by NGS, is cut into several segments with a certain size (R), and then decompressed, filtered and compressed, which greatly reduces the time for obtaining a target FastQ file through filtering.

[0111] 2. In order to reduce the number of file copying, the decompression / filtering / compression work needs to be completed in a stream in the same task. The generated fastq segment file is stored in the Aliyun OSS and ends with.part-${shard number}.

[0112] 3. In the decompression-filtering-compression process, each of the three processes consumes different resources (see Table 1). Executing all three operations directly in the same context leads to insufficient utilization of node resources; the network is idle during filtering, while the CPU is idle during decompression. Furthermore, Alibaba Cloud's batch computing nodes have relatively abundant CPU resources. Therefore, placing the three processes into three different threads, connected by a queue, is beneficial for further improving processing speed.

[0113] 4. Several workers are run on each batch computing node. Each worker consists of 3 different threads, which are responsible for decompression, filtering and compression respectively. The number of workers = number of virtual machine kernels / 2, which also helps to further improve the processing speed.

[0114] 5. MD5 checksums help ensure users receive authentic data. If the target FastQ file has been tampered with, the checksum will change. Users can easily detect data tampering by comparing the current checksum with the original checksum and finding a difference.

[0115] 6. Not using multi-segment uploads is beneficial for the application of this invention in cloud services other than Alibaba Cloud.

[0116] 7. Using Alibaba Cloud's multi-shard upload can reduce the number of copies per operation, thereby saving processing time.

[0117] Table 1 shows the different resources used by the three processes.

[0118] Process Occupied resources Decompression Network IO (down) Filtering CPU Compression CPU + network (up)

[0119] Currently, there is no product on the market that can solve the problem of high-speed delivery of massive amounts of genetic data. This invention, however, solves this problem with a box-based delivery service system. Using this box, the internal system intelligently senses the generation of new data in the cloud, automatically and synchronously downloads multi-threaded data slices in real time, and has a data retransmission mechanism. This overcomes the shortcomings of sequencing data, such as the complexity and time-consuming manual download and management due to the large data volume. Data delivery is fast, timely, complete, and secure, supporting mainstream domestic public cloud object storage services (and can be dynamically expanded to support more vendors as needed). After data is transmitted to the FlashTest Box, it can be accessed via the user's network as network storage (NAS), and also supports copying and exporting data via a high-speed USB 3.0 external hard drive.

[0120] The box delivery service method, the box contains the overall solution of software, hardware, network, the customer only needs to pay to enjoy the high-speed data download to the local service; Automatic detection cloud data generation, no manual intervention, sequencing data automatic download, can also be done to the edge of the production edge download, greatly speed up the data delivery time; Convenient and simple to use, save investment, no longer need to be equipped with computer, install special download software for download, to solve the defects caused by the prior art.

[0121] The concept, specific structure and generated technical effects of the present application will be further described below in combination with the drawings, so as to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0122] Figure 1 is a BGZF segmentation schematic diagram;

[0123] Figure 2 is a decompression-filtering-compression schematic diagram;

[0124] Figure 3 is a schematic diagram of running several workers under a node to complete the decompression-filtering-compression three threads;

[0125] Figure 4 is a schematic diagram of the merging step;

[0126] Figure 5 is a structural schematic diagram of the box delivery service system of the present application;

[0127] Figure 6 is a box service network node diagram;

[0128] Figure 7 is a box node diagram;

[0129] Figure 8 is a box communication negotiation diagram;

[0130] Figure 9 is a box communication message structure diagram;

[0131] Figure 10 is a data download phase 1 schematic diagram;

[0132] Figure 11 is a data download phase 2 schematic diagram;

[0133] Figure 12 is a data download phase 3 schematic diagram;

[0134] Figure 13 is a data download phase 4 schematic diagram;

[0135] Figure 14 is a flowchart of the box delivery service method of the present application.

[0136] Wherein, the reference signs are as follows:

[0137] Box service network 100, box cluster 200, zookeeper service module 300, work order system 400, server node 110, box body 210, remote procedure call service module 111, cluster state management module 112, first metadata module 113, box runtime information module 114, box state management module 115, box state heartbeat detection module 116, box connection analysis module 117, box message communication protocol library module 118, metadata storage module 211, network node connection module 212, volume space usage rate detection thread module 213, disk operation thread module 214, disk state monitoring module 215, data download and release module 216, second metadata module 217, large block data reading module 218, download module 219, hard disk initialization module 220, box communication protocol module 221. DETAILED DESCRIPTION

[0138] In order to make the technical means, creative features, purposes and effects of the invention easy to understand, the invention is further described below in combination with specific drawings. However, the invention is not limited to the following embodiments.

[0139] It should be understood that the structures, proportions, sizes, etc. shown in the drawings attached to the present specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and do not define the limiting conditions for the implementation of the invention, so they do not have technical substantive significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the invention, should still fall within the scope of the technical content disclosed by the invention.

[0140] Embodiment 1

[0141] The embodiment provides a method for generating a target FastQ file from an undetermined FastQ file generated by NGS, comprising the following steps:

[0142] S1, preparing original files and index information

[0143] In order to filter, the following information needs to be extracted from the order task and the machine task respectively:

[0144] 1) The first split result directory for extracting the undetermined FastQ file (which can be a normal first split result directory or a re-split result directory).

[0145] 2) All known indices on the lane where the original Undetermined FastQ file was located during sequencing. If the design is to be re-split, the newly provided indices need to be applied.

[0146] In addition, business personnel need to select the Lane to be split in the work order system, as well as the source I7 / I5 length of the original Undetermined FastQ file.

[0147] S2. Generate and correct the split points.

[0148] like Figure 1 As shown, after obtaining the original Undetermined FastQ file (in this embodiment, the file's save path and name are / path / to / fastq / Undetermined_S0_L002_R2_001.fastq.gz), the FastQ file is divided into several FastQ fragments (named as frag#number range(lower limit of number - upper limit of number)) according to a certain size (e.g., R, where R is any real number in GB between 0.01 and 20). For the boundary points of each FastQ fragment, a BGZF (a compression algorithm) boundary search needs to be performed within the range of ±M (M is 32-128Kb, and in this embodiment, M is 64Kb). The offset after the search is used as the actual segmentation point to segment the original Undetermined FastQ file encoded by BGZF.

[0149] The FastQ files generated by bcl2fastq (the file format used when the bcl sequencer is disassembled) (containing both splittable FastQ files and non-splittable raw Undetermined FastQ files) use the BGZF encoding standard, which is unidirectionally compatible with gzip encoding. The code used in the segmentation and compression / decompression processes described in this invention is referenced from... RFC 1952 (https: / / datatracker.ietf.org / doc / html / rfc1952) and SAM / BAM Format Specification It is implemented in section 4.1 (https: / / samtools.github.io / hts-specs / SAMv1.pdf).

[0150] S3, Decompression, Filtering, Compression

[0151] To reduce the number of file copies, decompression / filtering / compression operations need to be completed in the same task as a stream. The resulting FastQ fragment files are stored in Alibaba Cloud OSS and end with .part-${fragment number}.

[0152] The FastQ fragments generated in step S2 are all decompressed, filtered, and compressed. For example... Figure 2 As shown, this example only uses frag#nrange(252866201170-253401202339) for illustration. Other FastQ fragments generated in step S2 are processed in a similar manner as frag#n range(252866201170-253401202339). frag#n range(252866201170-253401202339) is read and decompressed into nucleotide sequences starting with @ using BGZF (e.g., @A00456:750:HXXXXXXX2:2:1101:6379:10471:N:0:nucleotide sequence+……). Then, the nucleotide sequences starting with @ are filtered out using known tags (e.g., @A00456:750:HXXXXXXX2:2:1101:6379:10471:N:0:nucleotide sequence+……). Next, they are written and compressed using BGZF into FastQ fragment files named .part-${fragment number} (e.g., Undetermined_xxx.fastq.gz.part-n in this example).

[0153] Since the three processes of decompression, filtering, and compression each consume different resources (Table 1), executing these three operations directly in the same context leads to insufficient utilization of node resources. Network idleness occurs during filtering, while CPU idleness occurs during decompression. However, Alibaba Cloud's batch computing nodes have relatively abundant CPU resources. Therefore, the three processes are placed in three different threads, connected by a queue.

[0154] On each batch computing node, several workers (number of virtual machine cores / 2) run. Each worker consists of three different threads, responsible for decompression / filtering / compression respectively. Figure 3 ). Figure 3 The example below will still be frag#n range(252866201170-253401202339).

[0155] S4, Merge

[0156] Since the current computing framework (Alibaba Cloud Batch Computing) does not support file merging, it is necessary to reassemble the fragments belonging to the same target FastQ in order locally in a separate task, generate MD5 checksums, and then write them back to Alibaba Cloud OSS.

[0157] As Figure 4 shown, Undetermined_xxx.fastq.gz.part-1, Undetermined_xxx.fastq.gz.part-2, Undetermined_xxx.fastq.gz.part-3, … Undetermined_xxx.fastq.gz.part-n obtained by the previous three steps are merged in order to obtain Undetermined_xxx.fastq.gz. Then Undetermined_xxx.fastq.gz is passed through md5sum to obtain the target Fastq file Undetermined_xxx.tar.fastq.md5 with a verification code. md5sum refers to Message-Digest Algorithm 5 calculation. md5sum is a command in the linux system to calculate the md5 value of a file.

[0158] In the merging process at this stage, the multi-shard upload of Aliyun is not used, although this can reduce the number of copies, but since this operation is not a standard object storage operation, and considering the possibility of deployment of other cloud services in the future, this function is not used.

[0159] S5, data delivery

[0160] The delivery of the target FastQ file (i.e. the delivery of the target FastQ file to the customer) is completed through the box delivery service system.

[0161] The box is a temporary storage for the data delivered by the work order system to the end user. The box needs to meet the following requirements: it can receive and store data generated by the work order system, including but not limited to: FastQ (a sequencing format file), BCL (a raw sequencing data format file generated by a sequencer), and UnknownFastQ (a FastQ file that cannot be identified as belonging to a certain data, such as the target FastQ file generated in this application); it can connect external storage devices to the box and copy the received data to the external storage devices; it can provide a network service access portal to allow direct access to the received data in the box on the local area network where the box is located;

[0162] It has the function of managing external storage devices, and the coverage of the function needs to be able to meet the process of: storage device insertion → data copying → storage device removal. The box itself has monitoring capabilities and can monitor the following contents:

[0163] The state of the box itself can monitor the data transmission process of receiving data from the work order system, can monitor the process of copying data to the external storage device, provides a user delivery interface, and the user can perform through the user interaction interface:

[0164] The box manages the data stored from the work order system, manages the external storage device, performs the data output process to the external storage device, manages the built-in network service of the box, and the communication between the box and the work order system is safe and controlled.

[0165] In the application process, there are both box active connection work order system and work order system active connection box scene, but because the box is deployed in an uncontrolled network environment, which means that each box does not have a fixed network entrance (IP address / domain name, etc.).

[0166] Therefore, the box needs to have passive communication function between the work order system, that is, once the box and the work order system establish connection, the work order system can send request to the box at any time.

[0167] The box delivery service system provided by the application is a whole solution of software, hardware and network, and customers can enjoy high-speed data download to local service by paying; automatic detection of cloud data generation, no manual intervention, automatic download of sequencing data, can also achieve edge generation and edge download, greatly speed up the data delivery time; easy to use, save investment, no longer need to be equipped with computer, install special download software for download.

[0168] As shown in Figure 5 The first aspect, the box delivery service system, wherein, contains box service network 100, box cluster 200, zookeeper service module 300, the zookeeper service module 300 is built-in one kind of open source distributed state management software, the box service network 100 is equipped with multiple server nodes 110, the box cluster 200 contains multiple box bodies 210, each box body 210 is individually deployed and respectively connected to the box service network 100, the box service network 100 is connected by wireless with work order system 400 and creates connection to realize data interaction, the zookeeper service module 300 is used for managing the state of the box service network 100, when the box body 210 needs to communicate with the work order system 400, it is not connected to a certain work order system 400, but is connected to the box service network 100, and then the work order system 400 is connected to the box service network 100 to carry out data transmission / box management and other work.

[0169] The above-mentioned box delivery service system, wherein the server node 110 is CubeMeshNode (box network node), and the box body 210 is CubeNode (box node).

[0170] As shown in Figure 6-3 the box delivery service system, the server node 110 comprises a remote procedure call service module 111 (RPCServer), a cluster state management module 112 (ClusterKeeper), a first metadata module 113 (MetaData), a box runtime information module 114 (RuntimeCubeInfo), a box state management module 115 (CubeKeeper), a box state heartbeat detection module 116 (CubeKeepAliveWorker), a box connection resolution module 117 (CubeConnectionResolver), and a box message communication protocol library module 118 (CubeTalk Message Library);

[0171] The remote procedure call service module 111 (RPCServer) is an entrance for the work order system 400 to remotely access the box;

[0172] The cluster state management module 112 (ClusterKeeper) is used to maintain the state data between the server node 110 and the zookeeper service module 300, and register the server node 110 in the box service network 100;

[0173] The first metadata module 113 (MetaData) stores the metadata required by the box service network 100;

[0174] The box runtime information module 114 (RuntimeCubeInfo) is used to abstract the online box state information;

[0175] The box state management module 115 (CubeKeeper) is used to maintain the connection state with a box;

[0176] The box state heartbeat detection module 116 (CubeKeepAliveWorker) is used to poll the box at a low frequency to avoid the zombie box caused by the inactive connection;

[0177] The box connection resolution module 117 (CubeConnectionResolver) is a box communication server, responsible for establishing the connection with the box and the communication encoding and decoding;

[0178] The box message communication protocol library module 118 (CubeTalk Message Library) is a box communication protocol library, which internally stores all the message type data that the box may use;

[0179] The box is used for receiving data push from the work order system 400, and managing the storage device connected to the box, and outputting data into the storage device, the box contains a metadata storage module 211 (Storage Metadata), a network node connection module 212 (MeshConnector), a volume space usage detection thread module 213 (VolumeSpaceUsageCheckThread), a disk operation thread module 214 (DiskManipulatingThread), a disk state monitoring module 215 (DiskWatcher), a data download and release module 216 (DownloadExtractor), a second metadata module 217 (Metadata), a large block data reading module 218 (BLOBReader), a download module 219 (Downloader), a hard disk initialization module 220 (FSMakerchusi), a box communication protocol module 221 (Cube TalkProtocol);

[0180] The metadata storage module 211 (Storage Metadata) is used for managing the metadata structure of the local storage device;

[0181] The network node connection module 212 (MeshConnector) is a client for box communication, responsible for communication establishment and coding and decoding with the box service network 100;

[0182] The volume space usage detection thread module 213 (VolumeSpaceUsageCheckThread) is responsible for monitoring the remaining capacity change of the available partition;

[0183] The disk operation thread module 214 (DiskManipulatingThread) is responsible for processing the management work of the local disk, including mounting, ejecting, and initializing;

[0184] The disk state monitoring module 215 (DiskWatcher) is responsible for monitoring the change of the storage device, such as inserting a new disk, etc.;

[0185] The data download and release module 216 (DownloadExtractor) is responsible for outputting a completed or partially completed download to the external storage device;

[0186] The second metadata module 217 (Metadata) stores the metadata structure associated with the business in the memory;

[0187] The blob data reading module 218 (BLOBReader) is used to provide the reading capability of the downloaded data. In order to control the number of file handles for downloading, the data of a single download is integrated into a blob. The downloaded content can be correctly read through the blob data reading module 218 (BLOBReader);

[0188] The download module 219 (Downloader) is a downloader used to perform the downloading action for a download. The download module 219 (Downloader) does not perform the connection data transmission and file writing work for downloading, but is responsible for controlling the three components to complete the downloading:

[0189] The fragment downloading (FragDownloader) is used for files without content. After obtaining the https download address, the file is divided into a plurality of data blocks. The download module 219 (Downloader) creates a plurality of fragment downloaders (FragDownloaders) and downloads the data blocks in parallel through a local queue.

[0190] The fragment confirmation writing (FragCommitter-fragcommitter) writes the completed frag into the local disk according to the offset.

[0191] The content downloading (ContentDownloader) is used for files containing content. The content of the file is directly obtained from the box service network 100.

[0192] The hard disk initialization module 220 (FSMakerchusi) is used to provide the disk initialization capability. A plurality of implementations are provided to support different types of file partition tables and partition formats.

[0193] The box communication protocol module 221 (CubeTalk Protocol) stores the network protocol used for communication between the box and the box service network 100.

[0194] The box delivery service system described above, wherein the communication protocol for connecting the remote procedure call service module 111 (RPCServer) and the work order system 400 is the thrift protocol.

[0195] The first metadata module 113 (MetaData) stores a metadata library including box information (CubeInfo), hard disk information (HDDInfo), download information (Download), and extraction information (Extraction). The box information (CubeInfo) is registration information of a box, the hard disk information (HDDInfo) is encapsulation of a known hard disk, the download information (Download) is encapsulation of a download task, and the extraction information (Extraction) is encapsulation of process data of outputting data to an external storage.

[0196] The metadata structure in the metadata storage module (Storage Metadata) includes block information (BlockInfo), disk information (DiskInfo), volume information (VolumeInfo), disk flag (DiskFlag), and internal storage disk (InternalStorageDisk). The block information (BlockInfo) is abstraction of all Linux block device, including those that cannot be identified. The disk information (DiskInfo) is abstraction of a storage device. The volume information (VolumeInfo) is abstraction of a partition on a disk. The disk flag (DiskFlag) is abstraction of a partition flag on a disk partition table. The internal storage disk (InternalStorageDisk) is used to abstract and mark a disk information (DiskInfo) as internal storage. The internal storage can be used to receive data push from the work order system 400, and the remaining disks can be used to output the pushed data.

[0197] The metadata structure in the second metadata module 217 (Metadata) includes download (Download), download record (DownloadEntry), fragment (Frag), EntryState (download state), DownloadState (download task state), extraction (Extraction), ExtractionEntry (extraction state), Download (abstract a data push), DownloadEntry (abstract a downloaded file), the downloaded file is divided into a content and a non-content, Frag (non-content downloaded file), Frag (abstract a segmented DownloadEntry), EntryState (abstract a DownloadEntry state in the downloading process), DownloadState (abstract a download state), Extraction (abstract a data output process), and ExtractionEntry (abstract a file released in the data output process).

[0198] The box delivery service system, wherein the network protocol comprises a negotiation phase and a communication phase.

[0199] As shown in Figure 8 The negotiation phase is used to establish a link between the box and the service network, whether it is an uplink or a downlink, and its process is the same. The process of the negotiation phase is as follows:

[0200] Step A1: the box establishes a TCP (Transmission Control Protocol) connection with the box service network 100, and sends the protocol version (2 bytes unsigned int) of the box to the box service network 100.

[0201] Step A2: the box service network 100 replies the protocol version (2 bytes unsigned int) to the box.

[0202] Step A3: the box responds to whether the protocol version is compatible, 1=compatible, 2=incompatible (2 bytes unsigned int).

[0203] If incompatible, the box service network 100 will disconnect the connection.

[0204] If compatible, the box service network 100 generates an RSA key, and sends the RSA public key length n (2 bytes unsigned int) and the RSA public key (n bytes) in the RSA key to the box.

[0205] Step A4: The box generates an identification containing the box's own device ID, connection type, then encrypts the identification using the read RSA public key, and sends the RSA public key length and identification RSA key to the box service network 100, which contains the box's own device ID, connection type (uplink / downlink), then encrypts the identification using the read RSA public key, and sends the length (2 bytes unsigned int) and identification;

[0206] Step A5: The box service network 100 decrypts the identification using the RSA private key in the RSA key, checks the registration information of the device in the identification, and sends the identification result to the box. The identification result: 1 = identification completed, 2 = unable to identify the box, 3 = the box is disabled;

[0207] Step A6: If the result is not identification completed, the box will disconnect the network;

[0208] Step A7: The box service network 100 sends a negotiation completion identification 1 (1 unsigned byte);

[0209] Step A8: Enter the communication phase;

[0210] As shown in Figure 9 After the negotiation phase is completed, the link enters this phase, which is a question and answer form, and is in an infinite loop. In this phase, the communication behavior of the box and the service network is mutual. The process of the communication phase is as follows:

[0211] Step B1: Read 4 bytes (4 bytes unsigned int) from the connection between the box and the box service process;

[0212] Step B2: Identify the 4 bytes read to obtain the message length M;

[0213] Step B3: Read data of length M bytes from the connection;

[0214] Step B4: Decode the M length byte data using AES. The key used by the service network for decoding is the key used by the box when registering with the service network. The key used by the box for decoding is the local configuration key. For the same box, the two keys must match;

[0215] Step B5: Process the received message (message) after decoding;

[0216] Step B6: Repeat steps B1-B5 for the next cycle.

[0217] When accepting data pushed by the work order system 400, the box needs to go through the following phases:

[0218] As shown in Figure 10 shown, a download task is created: the work order system 400 creates a Download for the box through the RPC service of the box service network 100, and the box cannot see the original OSS key of each file, but can know the ID and file name of each file;

[0219] As shown in Figure 11 shown, a download address is obtained: the box initiates a request to the service network to obtain the address of a file, and the box uses the ID of the file to identify the file that needs to be obtained, the service network finds the file requested by the box, and generates a https download address containing no access secret through the object signature function of Aliyun;

[0220] As shown in Figure 12 shown, the data is segmented: after obtaining the download URL, the box segments the data into small pieces through the http range request;

[0221] As shown in Figure 13 shown, the data is downloaded and assembled: the box downloads the segmented data pieces through the built-in FragDownloader, and hands them over to the Frag Comitter for merging and writing into the final BLOB, and when all the pieces are downloaded, the download of the single file is declared complete.

[0222] Embodiment 2

[0223] This embodiment uses multi-shard upload of Aliyun in step S4.

[0224] Step S5 data delivery: the delivery of the target FastQ file is completed through the box delivery service method. As shown in Figure 14 shown, the box delivery service method comprises the following steps:

[0225] Step C1: connect the box to the Internet;

[0226] Step C2: configure information related to the box, and initialize the hard disk;

[0227] Step C3: after the box is connected to the network, it is automatically registered to the box service network 100;

[0228] Step C4: when data is generated, the box service network 100 notifies the box to start the data download task.

[0229] The box delivery service method described above, wherein after the box is connected to the Internet in step C1, PPOE, password, intranet card parameters, and the communication public key of the box need to be configured;

[0230] In step C2, the mobile phone is used to scan the equipment two-dimensional code on the box shell to enter the configuration interface, and the information related to the box is configured.

[0231] In step C3, the box is automatically registered to the box service network 100 according to the internal preset configuration after networking.

[0232] The box delivery service method further includes step C5: after the data is completed, the data can be copied to a required use location through an external mobile hard disk or accessed through a network to the directory where the data of the box is located.

[0233] The information related to the box further includes operator related information, and step C2 further includes binding the operator and the box.

[0234] The rest is the same as example 1.

[0235] Example 3

[0236] The example provides a system for generating a target FastQ file from an NGS generated undetermined FastQ file, comprising:

[0237] The box delivery service system as described in example 1 is used to complete the delivery of the target FastQ file.

[0238] A database unit is configured to store NGS generated original data, sample information, and analyzed data.

[0239] A processing unit comprises:

[0240] A split point generation and correction module is configured to cut the NGS generated undetermined FastQ file into a plurality of FastQ segments according to a file size R; for each boundary point of the cut FastQ segment, a bgzf boundary search is performed within a ±M range; and the offset obtained after the search is used as an actual split point to split the FastQ segment, to obtain a split FastQ segment; R is 1-20 GB, and M is 32-128 KB.

[0241] A decompression and filtering compression module is configured to decompress the split FastQ segment, filter with a target barcode (s), and then compress to obtain a filtered FastQ segment.

[0242] A merging module is configured to recombine the filtered FastQ segment in order to generate the target FastQ file.

[0243] The database unit is in communication with the processing unit; the box delivery service system is in communication with the database unit; the box delivery service system and the processing unit are in communication. Wherein, R is a real number Gb in 0.01-20 Gb, and is not limited to two decimal places.

[0244] The decompression filtering compression module is further configured to complete the decompression, filtering and compression in a same task in a stream, and store the filtered FastQ fragments in a file ending with ".part-${shard number}" in the Aliyun OSS.

[0245] The decompression filtering compression module is further configured to put the three processes of decompression, filtering and compression into three different threads, and connect the threads through a queue.

[0246] The decompression filtering compression module is further configured to run a plurality of workers on each batch calculation node, each worker is composed of three different threads, and is responsible for decompression, filtering and compression respectively; and the number of the workers is equal to the number of virtual machine cores / 2.

[0247] The merging module is further configured to generate an md5 check along with the target FastQ file. The md5 check is used to verify the authenticity of the target FastQ file.

[0248] The merging module is further configured to complete the merging in the local computer.

[0249] The merging module is further configured to, after completing the merging in the local computer, upload to the cloud without using multi-sharding.

[0250] Embodiment 4

[0251] In this embodiment, the merging module is further configured to, after completing the merging in the local computer, upload to the Aliyun using multi-sharding, and the rest is the same as in Embodiment 3.

[0252] Embodiment 5

[0253] This embodiment provides a computer readable storage medium, the storage medium stores executable instructions, and the executable instructions are executed to realize the method for generating a target FastQ file from an undetermined FastQ file generated by NGS in Embodiment 1 or 2.

[0254] Embodiment 6

[0255] This embodiment provides a terminal, which comprises:

[0256] a memory configured to store executable instructions;

[0257] A processor configured to implement the method of generating a target FastQ file from an indeterminate FastQ file generated by NGS as described in embodiments 1 or 2 when executing executable instructions stored in the memory.

[0258] The preferred embodiments of the present application have been described in detail above. It should be understood that modifications and variations can be made by those skilled in the art in light of the teachings described herein without departing from the spirit and scope of the application. It is therefore contemplated that the application can encompass other variations than those specifically described herein.

Claims

1. A method for generating a target FastQ file from an undetermined FastQ file generated by NGS, characterized in that, Includes the following steps: S1. Prepare the original file and Index information to obtain the undetermined FastQ file generated by NGS; S2. Generate and correct segmentation points: Cut the undetermined FastQ file generated by NGS into several FastQ segments according to the file size R; for the boundary points of each FastQ segment obtained by cutting, a bgzf boundary search needs to be performed within the range of ±M; and use the offset after the search as the actual segmentation point to segment the FastQ segment to obtain the segmented FastQ segment; R is 1-20Gb, M is 32-128Kb; S3. Decompression, Filtering, and Compression: Decompress the FastQ fragments segmented in step S2, filter them with the target barcode(s), and then compress them to obtain the filtered FastQ fragments. S4. Merging: The filtered FastQ fragments obtained in step S3 are merged again in order to generate the target FastQ file; Step S4 also includes an md5 checksum generated along with the target FastQ file, used to verify the authenticity of the target FastQ file; The undetermined FastQ file is a sequencing result in which the barcode in the NGS primary split result does not match the barcode in the sample list; the target FastQ file is a FastQ file that matches the target barcode(s) provided by the customer.

2. The method for generating a target FastQ file from an undetermined FastQ file generated by NGS as described in claim 1, characterized in that, The decompression, filtering, and compression processes in step S3 are completed in the same task as a stream, and the resulting filtered FastQ fragments are stored in Alibaba Cloud OSS as files ending with .part-${fragment number}.

3. The method for generating a target FastQ file from an undetermined FastQ file generated by NGS as described in claim 1, characterized in that, Step S3 involves placing the three processes of decompression, filtering, and compression into three different threads, connected by a queue.

4. The method for generating a target FastQ file from an undetermined FastQ file generated by NGS as described in claim 3, characterized in that, Step S3 further includes running several workers on each batch computing node. Each worker consists of 3 different threads, which are responsible for decompression, filtering, and compression respectively. The number of workers = number of virtual machine kernels / 2.

5. The method for generating a target FastQ file from an undetermined FastQ file generated by NGS as described in claim 1, characterized in that, The merging in step S4 is completed on the local computer; step S4 also includes uploading to the cloud without using multiple fragments after the merging is completed on the local computer.

6. The method for generating a target FastQ file from an undetermined FastQ file generated by NGS as described in claim 1, characterized in that, It also includes S5, data delivery: the delivery of the target FastQ file is completed through the box delivery service system; the box delivery service system includes a box service network, a box cluster, and a zookeeper service module. The box service network has multiple server nodes, and the box cluster contains multiple box entities. Each box entity is deployed separately and connected to the box service network. The box service network establishes a connection with the work order system wirelessly to realize data interaction. The zookeeper service module is used to manage the status of the box service network.

7. A system for generating a target FastQ file from an undetermined FastQ file generated by NGS, characterized in that, include: The database unit is used to store the raw data generated by NGS, sample information, and data generated after analysis. The processing unit includes: A module for generating and correcting split points is used to cut the undetermined FastQ file generated by NGS into several FastQ segments according to the file size R. For the boundary points of each FastQ segment obtained by cutting, a bgzf boundary search needs to be performed within the range of ±M. The offset after the search is used as the actual split point to split the FastQ segment, resulting in the segmented FastQ segments. R is 1-20GB, and M is 32-128KB. The decompression and filtering module is used to decompress the segmented FastQ fragments, filter them with the target barcode(s), and then compress them to obtain the filtered FastQ fragments. The merging module is used to re-merge the filtered FastQ fragments in order to generate the target FastQ file; the merging module is also used to generate an MD5 checksum along with the target FastQ file; the MD5 checksum is used to verify the authenticity of the target FastQ file; The database unit communicates with the processing unit; the undetermined FastQ file is a sequencing result in which the barcode in the NGS first-level split result does not match the barcode in the sample list; the target FastQ file is a FastQ file that matches the target barcode(s) provided by the customer.

8. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed, implement the method for generating a target FastQ file from an undetermined FastQ file generated by NGS, as described in any one of claims 1-6.

9. A terminal, characterized in that, The terminal includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method of generating a target FastQ file from an undetermined FastQ file generated by NGS as described in any one of claims 1-6.

Images