Gene data processing method, device and related equipment under the background of high-throughput sequencing

Abstract

The application discloses a gene data processing method and device in a high-throughput sequencing background and related equipment, the method comprising: obtaining gene data blocks input from a sequencing platform in real time and inputting the gene data blocks into a first reserved area of a first memory; compressing the gene data blocks based on the data characteristics of each short sequence in the gene data blocks to obtain gene data compression blocks, retaining the gene data compression blocks in a second reserved area of the first memory, and releasing the gene data blocks from the first memory; and when the number of the gene data compression blocks in the first memory reaches J, outputting the J gene data compression blocks to a second memory and releasing the J gene data compression blocks from the first memory. The application processes each gene data block output from the sequencing platform in parallel in a streaming mode, saves waiting time, improves the utilization efficiency of computing resources, and improves the overall speed from data downloading from the sequencing platform to biological information analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sequencing technology, more particularly, to a gene data processing method and device in the context of high-throughput sequencing and related equipment. BACKGROUND

[0002] As an important means to explore the mystery of life, gene sequencing technology has become an important branch of bioinformatics research, and has a wide range of applications in species identification, gene detection, disease diagnosis, etc. The rapid development of gene sequencing technology has laid a solid foundation for precision medicine.

[0003] With the development of gene sequencing technology, the cost of sequencing is becoming lower and lower, resulting in a growing scale of sequencing business. As the scale of sequencing business continues to grow, the types of networked devices, business capacity, network structure, etc. are becoming more and more complex. How to timely process more sequencing business under existing computing resources and storage resources, and improve the delivery efficiency of bioinformatics sequencing, is a technical problem worth exploring. SUMMARY

[0004] Therefore, the present application provides a gene data processing method and device in the context of high-throughput sequencing and related equipment, which realizes high-concurrency streaming processing of gene sequencing data, thereby improving the delivery efficiency of bioinformatics sequencing.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a gene data processing method in the context of high-throughput sequencing, comprising:

[0006] When the available space in the first reserved area of the first memory reaches a preset capacity, a gene data block of a preset size is obtained, and the gene data block is input into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted in real time from a sequencing platform, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

[0007] Based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a gene data compressed block, the gene data compressed block is retained in a second reserved area of the first memory, and the gene data block is released from the first memory, and the second reserved area has the capacity to accommodate M gene data compressed blocks;

[0008] When the number of gene data compressed blocks in the first memory reaches J, J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory;

[0009] Wherein, N, M, J are all preset natural numbers, and J is not greater than M.

[0010] Preferably, the short sequences comprise metadata, base data and quality data; and the process of compressing the gene data block based on data characteristics of each short sequence in the gene data block comprises:

[0011] aligning the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the alignment result;

[0012] compressing the metadata in the gene data block using an incremental encoding technique or a run-length encoding technique;

[0013] determining the complexity of the quality data by a preset adaptive model, and determining a context statistical model of a first target order based on the complexity;

[0014] compressing the quality data using the context statistical model of the first target order to obtain a first intermediate compression result;

[0015] compressing the first intermediate compression result using a run-length encoding technique, an ANS+FSE encoding technique, an arithmetic encoding technique or a Huffman encoding technique.

[0016] Preferably, the process of aligning the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the alignment result comprises:

[0017] dividing the base data in the gene data block into a plurality of subsequences;

[0018] aligning each subsequence with a preset reference genome using a hash alignment method to obtain matching information of each subsequence, the matching information comprising a mismatch value;

[0019] for a subsequence with a mismatch value less than or equal to a preset threshold, compressing the subsequence based on the matching information of the subsequence;

[0020] for a subsequence with a mismatch value greater than the preset threshold:

[0021] determining the complexity of the subsequence by a preset adaptive model, and determining a context statistical model of a second target order based on the complexity;

[0022] compressing the subsequence using the context statistical model of the second target order to obtain a second intermediate compression result;

[0023] compressing the second intermediate compression result using a run-length encoding technique, an ANS+FSE encoding technique, an arithmetic encoding technique or a Huffman encoding technique.

[0024] Preferably, the process of aligning each sub-sequence with a preset reference genome by using the hash alignment method to obtain the matching information of each sub-sequence comprises:

[0025] Using the hash value of each sub-sequence as a query condition, querying a preset hash table to obtain the matching information of each sub-sequence;

[0026] Wherein, the preset hash table records the hash values of each reference sub-sequence in the reference genome and the position information of each reference sub-sequence in the reference genome, and each reference sub-sequence is obtained by dividing the reference genome.

[0027] Preferably, the process of dividing each reference sub-sequence from the reference genome comprises:

[0028] Divide a plurality of reference sub-sequences with a length of K and overlap from the reference genome at a preset step length, wherein K is a preset length value.

[0029] Preferably, the process of aligning the base data in the genetic data block with the preset reference genome comprises:

[0030] Aligning the base data in the genetic data block with the preset reference genome by using the BWT algorithm.

[0031] Preferably, it further comprises:

[0032] Obtaining the processing speed of the first operation, the first operation comprising: obtaining a genetic data block of a preset size and inputting it into a first reserved area of a first memory;

[0033] Obtaining the processing speed of the second operation, the second operation comprising: compressing the genetic data block based on the data characteristics of each short sequence in the genetic data block;

[0034] Based on the processing speed of the first operation and the processing speed of the second operation, determine the computing resources allocated to the first operation and the second operation.

[0035] The second aspect of the present application provides a genetic data processing device in the background of high-throughput sequencing, comprising:

[0036] a data block obtaining unit, configured to obtain a preset size of a gene data block when available space in a first reserved area of a first memory reaches a preset capacity, and input the gene data block into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted from a sequencing platform in real time, the first reserved area has a capacity of accommodating N gene data blocks, and the preset size is not greater than the preset capacity;

[0037] a data block processing unit, configured to compress the gene data block based on data characteristics of each short sequence in the gene data block, to obtain a gene data compressed block, retain the gene data compressed block in a second reserved area of the first memory, and release the gene data block from the first memory, wherein the second reserved area has a capacity of accommodating M gene data compressed blocks;

[0038] a data block exporting unit, configured to output J gene data compressed blocks to a second memory and release the J gene data compressed blocks from the first memory when the number of gene data compressed blocks in the first memory reaches J;

[0039] wherein N, M, and J are all preset natural numbers, and J is not greater than M.

[0040] The third aspect of the present application provides a gene data processing device in a high-throughput sequencing background, comprising a memory and a processor.

[0041] The memory is configured to store a program.

[0042] The processor is configured to execute the program to implement each step of the gene data processing method in the high-throughput sequencing background.

[0043] The fourth aspect of the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement each step of the gene data processing method in the high-throughput sequencing background.

[0044] As can be seen from the above technical solution, when the available space in the first reserved area of ​​the first memory reaches a preset capacity, a gene data block of a preset size is acquired and input into the first reserved area of ​​the first memory. The gene data block is a set of short sequences transmitted in real time from the sequencing platform. The first reserved area has the capacity to hold N gene data blocks, and the preset size is no greater than the preset capacity. Next, based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a compressed gene data block. The compressed gene data block is stored in the second reserved area of ​​the first memory, while the gene data block is released from the first memory. The second reserved area has the capacity to hold M compressed gene data blocks. It is understood that the process of acquiring and placing gene data blocks and the process of compressing gene data blocks are performed cyclically. That is, gene data blocks are continuously fed from the sequencing platform to a first reserved area of ​​the first memory until the first reserved area can no longer hold more gene data blocks; simultaneously, each gene data block in the first reserved area is compressed, and the resulting compressed gene data blocks are stored in a second reserved area of ​​the first memory. When the number of compressed gene data blocks in the first memory reaches J, J compressed gene data blocks are output to the second memory, and the J compressed gene data blocks are released from the first memory. Here, N, M, and J are all preset natural numbers, and J is not greater than M. This application realizes the streaming parallel processing of gene data blocks output from the sequencing platform, eliminating the need to wait for all gene data to be collected before processing. This saves waiting time, improves the utilization efficiency of computing resources, and increases the overall speed from data download from the sequencing platform to bioinformatics analysis, thus contributing to improved delivery efficiency of bioinformatics sequencing. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the gene data processing method in the context of high-throughput sequencing disclosed in the embodiments of this application;

[0047] Figure 2 A schematic diagram illustrating the streaming parallel processing of each step disclosed in the embodiments of this application is provided.

[0048] Figure 3A schematic diagram of a gene data processing method in the context of high-throughput sequencing disclosed by the embodiments of the present application;

[0049] Figure 4 A schematic diagram of a gene data processing device in the context of high-throughput sequencing disclosed by the embodiments of the present application;

[0050] Figure 5 A schematic diagram of a gene data processing device in the context of high-throughput sequencing disclosed by the embodiments of the present application;

[0051] Figure 6 A schematic diagram of a gene data processing device in the context of high-throughput sequencing disclosed by the embodiments of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. First, the related terms are explained below:

[0053] High-throughput sequencing: also known as "next-generation" sequencing technology, which is marked by the ability to sequence hundreds of thousands to millions of DNA molecules in parallel and generally shorter read length. Sequencing refers to analyzing the base sequence of a specific DNA fragment, i.e., the arrangement of adenine (A), thymine (T), cytosine (C) and guanine (G). The emergence of rapid DNA sequencing methods has greatly promoted the research and development of biology and medicine.

[0054] Base calling: identifying the base type (DNA sequence) from raw images (row images) by computer vision, writing the results to a cal file, and finally generating a sequencing report and FastQ data.

[0055] Sequence alignment: arranging two or more sequences together to indicate their similarities. Intervals can be inserted in the sequence (usually represented by a short horizontal line "-"). Corresponding identical or similar symbols (in nucleic acids, A, T (or U), C, G, in proteins, single-letter representation of amino acid residues) are arranged in the same column. It is usually used to study sequences evolved from a common ancestor, especially biological sequences such as protein sequences or DNA sequences. In alignment, mismatches correspond to mutations, and gaps correspond to insertions or deletions.

[0056] Reference genome: refers to the genome sequence of the species, which is the complete genome sequence that has been assembled, often as a standard reference for the species, such as the human genome reference sequence (fasta format).

[0057] K-mer: a subsequence of length K in a short sequence.

[0058] Networking technology: networking technology is network building technology, which is divided into Ethernet networking technology and ATM local area network networking technology. Ethernet networking is very flexible and simple, and can use various physical media to build networks with different topologies. It is the most widely used network in the world and has become the mainstream of network technology. Ethernet is divided into 10Mb / s, 100Mb / s and 1000Mb / s according to its transmission rate.

[0059] The inventors of the present application found that the existing process from sequencing to bioinformatics analysis includes:

[0060] 1) The sequencer completes the biochemical and imaging to obtain base image data, which is converted into CAL file containing DNA sequence through Base Calling, and the CAL file is converted into FASTQ text file containing sequence information through data processing, while data quality control is carried out and FASTQ is compressed and stored backup to disk using compression software.

[0061] 2) The compressed FASTQ file is read from the disk, and the sequence is compared with the sequence in the reference genome through sequence alignment software (such as BWA), the position of each sequence on the reference genome is found, and then arranged in order.

[0062] 3) Subsequent variation detection, interpretation and other related work are carried out according to research needs. The purpose of variation detection analysis is to accurately detect the variation set in each sample (such as human) genome, that is, the DNA sequence that exists between people.

[0063] From the above content, the following problems exist in the current genetic testing process:

[0064] 1) CAL file needs to be transmitted to GPU server for data conversion after Base Calling is completed, and FASTQ file needs to be transmitted to data quality control server for data quality control after data conversion, and then the data after quality control is transmitted to secondary storage server for backup storage, and at the same time, it is transmitted to the computing server for subsequent calculation and analysis. The mixed networking mode of various special equipment is complex and the cost is extremely high.

[0065] 2) CAL file in the process of data processing conversion into FASTQ, need to process a CAL file completely output a complete FASTQ file to the disk after the subsequent data transmission, quality control, backup and analysis, but the whole CAL file data volume is large, conversion time-consuming.

[0066] 3) CAL file conversion is completed, FASTQ file needs to be written to the disk, and the compression software reads from the disk for compression; When performing sequence alignment, the alignment software needs to read the FASTQ file from the disk first. Each input and output of data needs to be I / O on the disk, and the I / O is frequent and slow, which depends on the storage performance.

[0067] Because the sequencing file of gene data is a few G, dozens of G to hundreds of G, if waiting for all the data to be completed before performing subsequent data compression and bioinformatics analysis, the computing resources will be idle and not reasonably utilized, which will affect the efficiency of bioinformatics delivery. The present application takes data blocks as units, adopts a pipeline mode, and utilizes the same reference genome alignment in the process of gene data compression and sequence alignment. The data export and data compression are combined from independent steps, which can improve the overall efficiency and fully utilize the computing resources.

[0068] The gene data processing method in the high-throughput sequencing background provided by the embodiments of the present application will be introduced below. Please refer to Figure 1 The gene data processing method in the high-throughput sequencing background provided by the embodiments of the present application can include the following steps:

[0069] Step S101, acquiring a gene data block of a preset size and inputting it into a first reserved area of a first memory.

[0070] Among them, the gene data block is a short sequence set transmitted from the sequencing platform in real time, and the first reserved area has the capacity to accommodate N gene data blocks. N is a preset natural number

[0071] Exemplarily, the first memory can be a memory, in particular, the memory here can be a DDR memory. Because the gene data is large and the memory of the computing device is limited, it can be usually extended to a DDR memory. The DDR memory transmits data once on the rising edge and falling edge of the clock signal, which makes the data transmission speed of the DDR memory twice that of the traditional SDRAM. And because only the falling edge signal is added, it will not cause energy consumption to increase. As for the addressing and control signals, they are the same as the traditional SDRAM, only transmitted on the rising edge of the clock.

[0072] The short sequence set contains a plurality of short sequences (reads), and the size of the gene data block can be defined by the number of short sequences. For example, the preset size of the gene data block can be set to 40,000 to 500,000 short sequences based on the memory capacity of the computing device.

[0073] In step S102, the gene data block is compressed based on the data characteristics of each short sequence in the gene data block to obtain a gene data compressed block.

[0074] Each short sequence in the gene data block can include the following four lines:

[0075] The first line is metadata, starting with @, followed by a sequence identifier and an optional description.

[0076] The second line is base data, which is essentially an alphabetical sequence composed of A, C, G, T, and N, which is the DNA sequence we are really interested in. N represents those bases that cannot be identified during sequencing.

[0077] The third line starts with a + character and sometimes follows the same sequence identifier as the first line.

[0078] The fourth line is quality data, representing quality scores that describe the reliability of each base, which is a string of ASCII characters.

[0079] It should be noted that the lengths of the second and fourth lines must be equal, and each element in them corresponds one-to-one. From the composition of the short sequence, it can be seen that the metadata in the first line is a general description, which can be compressed using a general compression method; the base data in the second line is rich in biological characteristics, which can be compressed using a reference genome alignment method; and the quality data in the fourth line has relatively high internal correlation, which can be compressed using run-length encoding and the like.

[0080] In step S103, the gene data compressed block is retained in a second reserved area of the first memory, and the gene data block is released from the first memory.

[0081] The second reserved area has the capacity to accommodate M gene data compressed blocks, where M is a preset natural number. It should be noted that generally, the space size of the gene data compressed block will be smaller than the space size of the corresponding gene data block. However, considering that the speed of pulling the gene data block and the speed of computing the data compressed block based on the data may not be consistent, the size relationship between the capacity size N of the first reserved area and the capacity size M of the second reserved area should be further determined based on the two speeds to avoid space waste.

[0082] Step S104, when the number of the gene data compressed blocks in the first memory reaches J, output the J gene data compressed blocks to the second memory, and release the J gene data compressed blocks from the first memory.

[0083] wherein J is a preset natural number, and J is not greater than M. Illustratively, the second memory can be a hard disk in the computing device, i.e., output each J data compressed blocks as one file slice, and finally restore a complete sequencing file from each file slice.

[0084] It can be understood that, referring to Figure 2 , steps S101 to S103 are executed in a loop, i.e., step S101 continuously feeds the gene data blocks from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks; step S102 compresses each gene data block in the first reserved area, and step S103 retains the obtained gene data compressed blocks in the second reserved area of the first memory, the space of the first reserved area released by step S102 and the space of the second reserved area released by step S103 are used to receive new gene data blocks and gene data compressed blocks. In this way, for each gene data block, steps S101 to S103 can execute the compression of the sequencing data in a streaming manner; meanwhile, steps S101 to S103 of each gene data block can be highly parallel executed.

[0085] The application obtains a preset size of gene data block when the available space in the first reserved area of the first memory reaches a preset capacity, and inputs the gene data block into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted from a sequencing platform in real time, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity. Then, based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a gene data compressed block, and the gene data compressed block is retained in the second reserved area of the first memory, while the gene data block is released from the first memory. The second reserved area has the capacity to accommodate M gene data compressed blocks. It can be understood that the process of obtaining and placing the gene data block and the process of compressing the gene data block are repeated. That is, the gene data block is continuously transported from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks. At the same time, each gene data block in the first reserved area is compressed, and the obtained gene data compressed block is retained in the second reserved area of the first memory. When the number of gene data compressed blocks in the first memory reaches J, the J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory. Wherein N, M, J are all preset natural numbers, and J is not greater than M. The application realizes the parallel processing of each gene data block output by the sequencing platform in a streaming manner, without waiting for all gene data sets to be processed. In addition to saving waiting time, the utilization efficiency of computing resources is improved, the overall speed from sequencing platform data download to biological information analysis is improved, and the delivery efficiency of biological information sequencing is improved.

[0086] In some embodiments of the application, the process of compressing the gene data block based on the data characteristics of each short sequence in the gene data block in the above step S102 can include:

[0087] S1, using incremental encoding technology or run-length encoding technology to compress the metadata in the gene data block.

[0088] Wherein the metadata in the gene data block of different sequencing platforms will have different formats. Generally, the metadata starts with a symbol “@”, followed by a sequence identifier and other optional information, such as instrument name, flow cell channel, block number in the flow cell channel, “x” coordinate of the cluster in the block, “y” coordinate of the cluster in the block, sample number in the mixed sample, double-end identifier and sequence length.

[0089] For the metadata in the gene data block, they can be first parsed into different data segments using separators (punctuation marks), which generally include dots, spaces, underscores, hyphens, slashes, equal signs and colons; then different data segments are classified by recognition algorithms, and fixed information such as instrument names, sequence lengths and double-end identifiers are not coded and compressed, and other data segments are compressed by different coding methods, including incremental coding, run-length coding and context mixed compression algorithm.

[0090] For data segments containing only consecutive integers, incremental coding is used to record the difference between the two data segments; for data segments containing letters and numbers, improved run-length coding is used to record repeated characters in the data segments, and incremental coding is used to record the difference between the two data segments.

[0091] S2, the quality data in the gene data block is compressed by combining a context statistical model, run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

[0092] Specifically, it can include:

[0093] S21, determine the complexity of the quality data in the gene data block by a preset adaptive model, and determine a context statistical model of a first target order based on the complexity.

[0094] Wherein, the complexity of the quality score can be 4 quality values, 8 quality values or 40 quality values, etc., and based on different quality values, the first target order can be 1st order, 2nd order, 3rd order, 4th order, 5th order or 6th order. After the complexity of the quality data is counted by the preset adaptive model, the order of the context statistical model suitable for the quality data is determined, so that a better compression effect can be obtained.

[0095] S22, compress the quality data using the context statistical model of the first target order to obtain a first intermediate compression result.

[0096] Wherein, the first intermediate compression result is only an intermediate output, and will not form a file output to a storage such as a disk.

[0097] S23, the first intermediate compression result is compressed by using run-length coding technology, ANS+FSE coding technology, arithmetic coding technology or Huffman coding technology.

[0098] The ANS (Asymmetric Numeral System) + FSE (Finite State Entropy) is a lossless compression technology. The first intermediate compression result is compressed by any one of the lossless compression technologies, so that the compression rate can be further improved.

[0099] S3, aligning the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the alignment result.

[0100] It should be noted that the reference genome here refers to the genome sequence of the sequenced species, that is, the complete genome sequence that has been assembled. The complete genome sequence is often used as a standard reference for the species.

[0101] For example, when aligning the base data in the gene data block, the bases in the base data in the gene data block can be aligned with the reference genome, and the position of each base data on the reference genome can be found, so as to obtain the alignment result.

[0102] For the matched base data, the position in the reference genome can be used to represent it, that is, the position is recorded without retaining the original base data. The matching information is sorted and then incrementally encoded and compressed, so as to achieve the effect of data compression. For the base data that does not match, a general compression method can be used for compression.

[0103] Based on this, in some embodiments of the present application, the process of S3 of aligning the base data in the gene data block with a preset reference genome and compressing the base data in the gene data block according to the alignment result can include:

[0104] S31, dividing the base data in the gene data block into a plurality of subsequences.

[0105] For example, the base data can be divided into a plurality of subsequences with a step size of 20.

[0106] S32, aligning each subsequence with the preset reference genome by using a hash alignment method to obtain matching information of each subsequence.

[0107] The matching information includes a mismatch value.

[0108] S33, for the subsequence with a mismatch value less than or equal to a preset threshold, compressing the subsequence based on the matching information of the subsequence.

[0109] For example, the recorded sequence alignment information such as the position on the reference genome, the mismatch, etc. is sorted and then compressed by the delta encoding.

[0110] S34, for the sub-sequence with the mismatch value greater than the preset threshold, the sub-sequence is compressed by combining the context mixed compression algorithm and the delta encoding technology and the run-length encoding technology.

[0111] Specifically, the method can include:

[0112] S341, the complexity of the sub-sequence is determined by a preset adaptive model, and a context statistical model of a second target order is determined based on the complexity.

[0113] The second target order can be 0th order, 1st order or 2nd order. After the complexity of the sub-sequence is statistically determined by the preset adaptive model, the order of the context statistical model suitable for the sub-sequence is determined, so that a better compression effect can be obtained.

[0114] Specifically, the character types of the sub-sequence (base sequence) include A / T / C / G / N, if the GC content is greater than 80% or less than 20%, a 0th order context statistical model is selected for compression by prediction; if the GC content is between 60% and 80%, or the GC content is between 20% and 40%, a 1st order context statistical model is selected for compression by prediction; if the GC content is between 40% and 60%, a 2nd order context statistical model is selected for compression by prediction.

[0115] S342, the sub-sequence is compressed by using the context statistical model of the second target order to obtain a second intermediate compression result.

[0116] The second intermediate compression result is only an intermediate output and will not be output to a file in a storage such as a disk.

[0117] S343, the second intermediate compression result is compressed by using the run-length encoding technology, the ANS+FSE encoding technology, the arithmetic encoding technology or the Huffman encoding technology.

[0118] For example, the run-length encoding is used for preprocessing to eliminate redundancy, the complexity of the base data and the quality data in the short sequence is understood by using the adaptive statistical model, the data modeling is performed by using the dynamic context mixed algorithm of different orders according to different complexities, and finally the combination scheme of the arithmetic encoding and the Huffman encoding is used for compression.

[0119] In some embodiments of the present application, the process of obtaining the matching information of each sub-sequence by aligning each sub-sequence with a preset reference genome using the hash alignment method in S32 can include:

[0120] Using the hash value of each sub-sequence as a query condition, a preset hash table is queried to obtain the matching information of each sub-sequence.

[0121] The preset hash table records the hash values of each reference sub-sequence in the reference genome and the position information of each reference sub-sequence in the reference genome, and each reference sub-sequence is obtained by dividing the reference genome.

[0122] In some embodiments of the present application, the process of obtaining each reference sub-sequence by dividing the reference genome can include:

[0123] A plurality of reference sub-sequences with a length of K and overlapping are divided from the reference genome at a preset step length. K is a preset length value. For example, K can be 20.

[0124] Specifically, first, the sequence in the reference genome R is divided into overlapping k-mer long sub-sequences. Then, a hash table HR is established, and the k-mer long sub-sequences and the source position of each sub-sequence in the reference genome are mapped and stored in the hash table HR through a hash function. Next, the short sequences in the gene data block are also divided into k-mer long sub-sequences. Finally, by querying the hash table composed of the k-mer long sub-sequences of the genome, data matching is performed, and the mismatch value Mismatch is calculated.

[0125] If the Mismatch in the short sequence is less than or equal to a predefined threshold P, it indicates that all sub-sequences of the short sequence are adjacent to each other in the correct order, and completely match the reference genome sub-sequences, so the short sequence can be positioned on the reference genome, and the matching information is recorded.

[0126] The matching information can include the matching position MATCHpos, the matching length MATCHlen, the matching type MATCHtype, the mismatch position MISpos, and the mismatch base MISalt.

[0127] If the number of Mismatch in the short sequence is less than or equal to a predefined threshold P, the matching information includes the matching position, the matching length and the matching type "P (Perfect Match)"; otherwise, the matching information includes the matching position, the matching length and the matching type "I (Insertion), D (Deletion)", the mismatch position and the mismatch base. Specifically, the CIGAR information in the sequence alignment result can be referred to. The matching information is sorted according to the matching position, and the sorted matching position is conducive to further compression of the matching information, and finally the difference between the two pieces of matching information is recorded using the incremental encoding scheme.

[0128] It should be noted that when the short sequence contains SNPs / INDELs, the mismatch is located in one of the sub-sequences of the short sequence, and the other sub-sequence can be completely matched with the reference genome sub-sequence.

[0129] In this way, the completely matched seed sequence can be taken as an anchor point, and the region adjacent to the anchor point can be accurately aligned with the remaining sub-sequences (which may contain mismatches) by using a classical alignment algorithm to find the final matching position of the sequence and record the matching information.

[0130] In some embodiments of the present application, the process of S2 of aligning the base data in the genetic data block with the preset reference genome can include:

[0131] The base data in the genetic data block is aligned with the preset reference genome by using a BWT (Burrows-Wheeler Transform) algorithm.

[0132] The BWT algorithm is a data conversion algorithm that places similar characters in a string in adjacent positions to facilitate subsequent compression. Specifically, an index is constructed based on FM, and backward search and backtracking are implemented based on Burrows-Wheeler matrix (BWM) and Last-First (LF) algorithm. The BWT is a tree structure containing all suffixes and prefixes in the string, which can realize fast searching of the string. The process of backtracking the base data in the genetic data block is the process of finding the optimal alignment position of the aligned base data.

[0133] In some embodiments of the present application, referring to Figure 3 The genetic data processing method in the high-throughput sequencing background can further include:

[0134] In S105, the processing speed of the first operation is obtained.

[0135] The first operation corresponds to step S101, and includes: obtaining a preset size of a gene data block and inputting the gene data block into a first reserved area of the first memory.

[0136] In step S106, a processing speed of the second operation is obtained.

[0137] The second operation corresponds to step S102, and includes: compressing the gene data block based on data characteristics of each short sequence in the gene data block.

[0138] In step S107, based on the processing speed of the first operation and the processing speed of the second operation, a calculation resource allocated to the first operation and the second operation is determined.

[0139] Specifically, if the processing speed of the first operation is less than the processing speed of the second operation, the gene data blocks in the first reserved area are less, and the calculation resource is sufficient, three processes of base data alignment compression, metadata compression and quality data compression can be started simultaneously for one gene data block, and appropriate calculation resource is allocated to the corresponding process.

[0140] When the metadata or quality data compression of each gene data block is completed, the result is stored in the second reserved area, and the calculation resource is released; when the base data alignment task is completed, the alignment result is stored in the second reserved area, and subsequent compression processing is performed according to the alignment result of the base data, and after completion, the result is stored in the second reserved area, and the calculation resource is released.

[0141] If the processing speed of the first operation is greater than the processing speed of the second operation, multiple gene data blocks will be retained in the first reserved area, at this time, less calculation threads can be allocated to one gene data block, and data alignment of the gene data block is preferentially performed, and if there is resource for metadata and quality data compression in the alignment process, the corresponding resource is allocated, otherwise the data compression storage is performed again after the gene data block alignment is completed.

[0142] Through the above resource self-adaptive dynamic allocation algorithm, the calculation resource can be dynamically provided according to the number of gene data blocks in the first reserved area, the first-in first-out is guaranteed as much as possible, and the task management and scheduling ability of automatically adjusting the calculation resource is realized. The elastic expansion can ensure that there is sufficient resource amount after the resource demand rises in the next time period, so that the situation of resource shortage and resource supply lag does not occur, and when the resource demand increases sharply, the elastically released resource can play an effective buffering role.

[0143] The gene data processing device in the high-throughput sequencing background provided by the embodiment of the application is described below, and the gene data processing device in the high-throughput sequencing background described below can be correspondingly referred to the gene data processing method in the high-throughput sequencing background described above.

[0144] Referring to Figure 4 The gene data processing device in the high-throughput sequencing background provided by the embodiments of the present application can include:

[0145] The data block acquisition unit 21 is configured to acquire a gene data block of a preset size when available space in the first reserved area of the first memory reaches a preset capacity, and input the gene data block into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted from a sequencing platform in real time, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity.

[0146] The data block processing unit 22 is configured to compress the gene data block based on data characteristics of each short sequence in the gene data block, obtain a gene data compressed block, retain the gene data compressed block in a second reserved area of the first memory, and release the gene data block from the first memory, wherein the second reserved area has the capacity to accommodate M gene data compressed blocks.

[0147] The data block export unit 23 is configured to output J gene data compressed blocks to the second memory and release the J gene data compressed blocks from the first memory when the number of gene data compressed blocks in the first memory reaches J.

[0148] wherein N, M and J are preset natural numbers, and J is not greater than M.

[0149] In some embodiments of the present application,

[0150] The short sequences include metadata, base data and quality data; the process of compressing the gene data block based on the data characteristics of each short sequence in the gene data block includes:

[0151] aligning the base data in the gene data block with a preset reference genome, and compressing the base data in the gene data block according to the alignment result;

[0152] using an incremental encoding technology or a run-length encoding technology to compress the metadata in the gene data block;

[0153] determining the complexity of the quality data by a preset adaptive model, and determining a context statistical model of a first target order based on the complexity;

[0154] using the context statistical model of the first target order to compress the quality data to obtain a first intermediate compression result;

[0155] The first intermediate compression result is compressed by using a run-length encoding technique, an ANS+FSE encoding technique, an arithmetic encoding technique, or a Huffman encoding technique.

[0156] In some embodiments of the present application, the data block processing unit 22 aligns the base data in the gene data block with a preset reference genome, and the process of compressing the base data in the gene data block according to the alignment result can include:

[0157] The base data in the gene data block is divided into a plurality of subsequences;

[0158] Each subsequence is aligned with the preset reference genome by using a hash alignment method to obtain matching information of each subsequence, the matching information including a mismatch value;

[0159] For a subsequence with a mismatch value less than or equal to a preset threshold, the subsequence is compressed based on the matching information of the subsequence;

[0160] For a subsequence with a mismatch value greater than the preset threshold:

[0161] The complexity of the subsequence is determined by a preset adaptive model, and a context statistical model of a second target order is determined based on the complexity;

[0162] The subsequence is compressed by using the context statistical model of the second target order to obtain a second intermediate compression result;

[0163] The second intermediate compression result is compressed by using a run-length encoding technique, an ANS+FSE encoding technique, an arithmetic encoding technique, or a Huffman encoding technique.

[0164] In some embodiments of the present application, the data block processing unit 22 aligns each subsequence with the preset reference genome by using a hash alignment method to obtain matching information of each subsequence, and the process can include:

[0165] The hash value of each subsequence is used as a query condition to query a preset hash table to obtain the matching information of each subsequence;

[0166] The preset hash table records the hash values of each reference subsequence in the reference genome and the position information of each reference subsequence in the reference genome, and each reference subsequence is divided from the reference genome.

[0167] In some embodiments of the present application, the data block processing unit 22 divides each reference subsequence from the reference genome, and the process can include:

[0168] Divide the reference genome into a plurality of reference subsequences with a length of K and overlapping each other with a preset step length, where K is a preset length value.

[0169] In some embodiments of the present application, the process of aligning the base data in the gene data block with the preset reference genome by the data block processing unit 22 can include:

[0170] Aligning the base data in the gene data block with the preset reference genome by using the BWT algorithm.

[0171] In some embodiments of the present application, referring to Figure 5 The gene data processing device in the high-throughput sequencing background can further include a computing resource allocation unit 24 configured to:

[0172] Obtain a processing speed of a first operation, the first operation including: obtaining a gene data block with a preset size and inputting the gene data block into a first reserved area of a first memory;

[0173] Obtain a processing speed of a second operation, the second operation including: compressing the gene data block based on data characteristics of each short sequence in the gene data block;

[0174] Determine computing resources allocated to the first operation and the second operation based on the processing speed of the first operation and the processing speed of the second operation.

[0175] The gene data processing device in the high-throughput sequencing background provided by the embodiments of the present application can be applied to a gene data processing device in the high-throughput sequencing background, such as a computer and the like. Optionally, Figure 6 A hardware structure block diagram of the gene data processing device in the high-throughput sequencing background is shown, referring to Figure 6 The hardware structure of the gene data processing device in the high-throughput sequencing background can include: at least one processor 31, at least one communication interface 32, at least one memory 33 and at least one communication bus 34.

[0176] In the embodiments of the present application, the number of the processor 31, the communication interface 32, the memory 33 and the communication bus 34 is at least one, and the processor 31, the communication interface 32 and the memory 33 complete the communication with each other through the communication bus 34;

[0177] The processor 31 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0178] The memory 33 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory;

[0179] The memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, and the program is used for:

[0180] When the available space in the first reserved area of the first memory reaches a preset capacity, a gene data block with a preset size is obtained, and the gene data block is input into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted in real time from a sequencing platform, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

[0181] Based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a gene data compressed block, the gene data compressed block is retained in a second reserved area of the first memory, and the gene data block is released from the first memory, and the second reserved area has the capacity to accommodate M gene data compressed blocks;

[0182] When the number of gene data compressed blocks in the first memory reaches J, J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory;

[0183] Wherein, N, M, J are all preset natural numbers, and J is not greater than M.

[0184] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0185] The embodiment of the application further provides a storage medium, which can store a program suitable for the processor to execute, and the program is used for:

[0186] When the available space in the first reserved area of the first memory reaches a preset capacity, a gene data block with a preset size is obtained, and the gene data block is input into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted in real time from a sequencing platform, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity;

[0187] Based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a gene data compressed block, the gene data compressed block is retained in a second reserved area of the first memory, and the gene data block is released from the first memory, and the second reserved area has the capacity to accommodate M gene data compressed blocks;

[0188] When the number of gene data compressed blocks in the first memory reaches J, the J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory;

[0189] wherein N, M, and J are all preset natural numbers, and J is not greater than M.

[0190] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0191] In summary:

[0192] When the available space in the first reserved area of the first memory reaches a preset capacity, the application acquires a gene data block of a preset size and inputs the gene data block into the first reserved area of the first memory, wherein the gene data block is a set of short sequences transmitted from a sequencing platform in real time, the first reserved area has the capacity to accommodate N gene data blocks, and the preset size is not greater than the preset capacity. Then, the gene data block is compressed based on the data characteristics of each short sequence in the gene data block, to obtain a gene data compressed block, the gene data compressed block is retained in a second reserved area of the first memory, and the gene data block is released from the first memory. The second reserved area has the capacity to accommodate M gene data compressed blocks. It can be understood that the process of acquiring and placing the gene data block and the process of compressing the gene data block are repeated. That is, the gene data block is continuously transported from the sequencing platform to the first reserved area of the first memory until the first reserved area cannot accommodate more gene data blocks. At the same time, each gene data block in the first reserved area is compressed, and the obtained gene data compressed block is retained in the second reserved area of the first memory. When the number of gene data compressed blocks in the first memory reaches J, the J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory. Wherein N, M, and J are all preset natural numbers, and J is not greater than M. The application realizes the parallel processing of each gene data block output from the sequencing platform in a streaming manner, without waiting for all gene data sets to be processed. In addition to saving waiting time, the application improves the utilization efficiency of computing resources and the overall speed from data downloading from the sequencing platform to biological information analysis, which helps to improve the delivery efficiency of biological information sequencing.

[0193] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not intended to denote the presence of any such actual relationship or order. Moreover, the terms "include", "have", or any other variant thereof are intended to encompass non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise a list of elements are not required to comprise only those elements in the list, but can include other elements not expressly listed, or also include elements inherent in such processes, methods, articles, or apparatuses. Without additional restrictions, an element preceded by "comprises... a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the stated element.

[0194] The various embodiments in the specification are described in progressive order with each embodiment building on one or more of the previous embodiments, however the order of the embodiments presented is not intended to be construed as a requirement or limitation for these embodiments. Any one or more of the embodiments described herein are possible embodiments, which are merely examples of the herein disclosed embodiments. Unless otherwise noted, the same reference numerals are used throughout the drawings to refer to the same or like components.

[0195] The above description of disclosed embodiments provides enough information to enable those skilled in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the application is not to be restricted based on the specific embodiments illustrated herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing gene data in the context of high-throughput sequencing, characterized in that, include: When the available space in the first reserved area of ​​the first memory reaches the preset capacity, a gene data block of the preset size is obtained and the gene data block is input into the first reserved area of ​​the first memory. The gene data block is a set of short sequences transmitted in real time from the sequencing platform. The first reserved area has the capacity to accommodate N gene data blocks. The preset size is not greater than the preset capacity. Based on the data characteristics of each short sequence in the gene data block, the gene data block is compressed to obtain a compressed gene data block. The compressed gene data block is stored in the second reserved area of ​​the first memory and the gene data block is released from the first memory. The second reserved area has the capacity to accommodate M compressed gene data blocks. When the number of gene data compressed blocks in the first memory reaches J, J gene data compressed blocks are output to the second memory, and the J gene data compressed blocks are released from the first memory. Where N, M, and J are all preset natural numbers, and J is no greater than M; The short sequences include metadata, base data, and quality data; the process of compressing the gene data block based on the data characteristics of each short sequence in the gene data block includes: The base data in the gene data block is compared using a preset reference genome, and the base data in the gene data block is compressed based on the comparison results. The metadata in the gene data block is compressed using incremental encoding or run-length encoding techniques; The complexity of the quality data in the gene data block is determined by a preset adaptive model, and a contextual statistical model of the first target order is determined based on the complexity. The quality data is compressed using the contextual statistical model of the first target order to obtain a first intermediate compression result; The first intermediate compression result is compressed using run-length encoding, ANS+FSE encoding, arithmetic encoding, or Huffman encoding.

2. The method according to claim 1, characterized in that, The process of comparing the base data in the gene data block using a preset reference genome and compressing the base data in the gene data block based on the comparison results includes: The base data in the gene data block is divided into multiple sub-sequences; Each subsequence is compared with a preset reference genome using a hash alignment method to obtain matching information for each subsequence, including mismatch values. For subsequences with mismatch values ​​less than or equal to a preset threshold, the subsequences are compressed based on their matching information. For subsequences with mismatch values ​​greater than a preset threshold: The complexity of the subsequence is determined by a preset adaptive model, and a contextual statistical model for the second target order is determined based on the complexity. The subsequence is compressed using the contextual statistical model of the second target order to obtain a second intermediate compression result; The second intermediate compression result is compressed using run-length encoding, ANS+FSE encoding, arithmetic encoding, or Huffman encoding.

3. The method according to claim 2, characterized in that, The process of comparing each subsequence with a preset reference genome using a hash alignment method to obtain the matching information of each subsequence includes: Using the hash value of each subsequence as a query condition, a query is performed in a preset hash table to obtain the matching information for each subsequence. The preset hash table records the hash value of each reference subsequence in the reference genome and the position information of each reference subsequence in the reference genome, wherein each reference subsequence is obtained from the reference genome.

4. The method according to claim 3, characterized in that, The process of dividing the reference genome to obtain each reference subsequence includes: Multiple overlapping reference subsequences of length K are divided from the reference genome at a preset step size, where K is a preset length value.

5. The method according to claim 1, characterized in that, The process of comparing the base data in the gene data block using a preset reference genome includes: The BWT algorithm is used to compare the base data in the gene data block with a preset reference genome.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: The processing speed of the first operation is obtained, the first operation includes: obtaining a gene data block of a preset size and inputting the gene data block into a first reserved area of ​​the first memory; The processing speed of the second operation is obtained, the second operation including: compressing the gene data block based on the data characteristics of each short sequence in the gene data block; Based on the processing speed of the first operation and the processing speed of the second operation, the computing resources allocated to the first operation and the second operation are determined.

7. A gene data processing device in the context of high-throughput sequencing, characterized in that, include: A data block acquisition unit is used to acquire a gene data block of a preset size when the available space in the first reserved area of ​​the first memory reaches a preset capacity, and input the gene data block into the first reserved area of ​​the first memory. The gene data block is a collection of short sequences transmitted in real time from the sequencing platform. The first reserved area has the capacity to accommodate N gene data blocks. The preset size is not greater than the preset capacity. A data block processing unit is configured to compress the gene data block based on the data characteristics of each short sequence in the gene data block to obtain a compressed gene data block, store the compressed gene data block in a second reserved area of ​​a first memory, and release the gene data block from the first memory. The second reserved area has the capacity to accommodate M compressed gene data blocks. The data block export unit is used to output J gene data compressed blocks to the second memory and release the J gene data compressed blocks from the first memory after the number of gene data compressed blocks in the first memory reaches J. Where N, M, and J are all preset natural numbers, and J is no greater than M; The short sequences include metadata, base data, and quality data; the process of compressing the gene data block based on the data characteristics of each short sequence in the gene data block includes: The base data in the gene data block is compared using a preset reference genome, and the base data in the gene data block is compressed based on the comparison results. The metadata in the gene data block is compressed using incremental encoding or run-length encoding techniques; The complexity of the quality data in the gene data block is determined by a preset adaptive model, and a contextual statistical model of the first target order is determined based on the complexity. The quality data is compressed using the contextual statistical model of the first target order to obtain a first intermediate compression result; The first intermediate compression result is compressed using run-length encoding, ANS+FSE encoding, arithmetic encoding, or Huffman encoding.

8. A gene data processing device for high-throughput sequencing, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the gene data processing method in the context of high-throughput sequencing as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the gene data processing method in the context of high-throughput sequencing as described in any one of claims 1 to 6.

Images