Sequencing data compression method and apparatus, non-transitory storage medium, and electronic device

Abstract

This application discloses a sequencing data compression method, apparatus, non-volatile storage medium, and electronic device. The method includes: determining identifier format information of the sequencing data, wherein the identifier format information includes type information of the identifiers in the sequencing data; determining the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm. This application solves the technical problem of low sequencing data compression efficiency caused by the failure to consider the differences in data formats output by different sequencing platforms when compressing sequencing data in related technologies.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and more specifically, to a sequencing data compression method, apparatus, non-volatile storage medium, and electronic device. Background Technology

[0002] In related technologies, when compressing sequencing data, the same compression method is often used for sequencing data output from different sequencing platforms, without taking into account the differences in format, such as differences in sequence identifiers. However, each compression method has its applicable and unapplicable scenarios, which means that the compression method used in related technologies may be incompatible with the sequencing data, resulting in low compression efficiency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a sequencing data compression method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problem of low sequencing data compression efficiency caused by the failure to consider the differences in data formats output by different sequencing platforms when compressing sequencing data in related technologies.

[0005] According to one aspect of the embodiments of this application, a sequencing data compression method is provided, comprising: determining identifier format information of sequencing data, wherein the identifier format information includes type information of identifiers in the sequencing data; determining the data processing method of fields corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm.

[0006] Optionally, determining the identifier format information for sequencing data includes: determining the platform identifier information of the sequencing platform from which the sequencing data is output; and determining the identifier format information corresponding to the platform identifier information based on the platform identifier information.

[0007] Optionally, the data processing method includes storing the field values ​​of the fields; determining the data processing method for each field corresponding to an identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method includes: determining fixed field identifiers and fixed fields corresponding to the fixed field identifiers in the sequencing data based on the identifier type information, wherein the fixed fields include fields used to identify the ID or name of the sequencer outputting the sequencing data, and the values ​​of the fixed fields in the sequencing data output by the same sequencer are all fixed values; and storing the field values ​​of the fixed fields.

[0008] Optionally, the data processing method includes classifying the fields; determining the data processing method for the fields corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method includes: determining the field type corresponding to each identifier in the sequencing data based on the identifier type information; classifying the fields corresponding to each identifier in the sequencing data according to the field type corresponding to each identifier to obtain multiple field sets, wherein the fields in each field set have the same field type.

[0009] Optionally, compressing each field in the processed sequencing data using a preset compression algorithm includes: compressing each field in each field set using the preset compression algorithm.

[0010] Optionally, the data processing method includes converting the field attributes; determining the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and processing the field according to the data processing method includes: determining the target field in the sequencing data based on the identifier type information; converting the data type of the target field from a first type to a second type, wherein the storage space required to store the target field of the second type is less than the storage space required to store the target field of the first type.

[0011] Optionally, the first type includes strings, and the second type includes integers.

[0012] Optionally, the sequencing data compression method further includes: determining the compressed data after compression processing and the fixed fields corresponding to the compressed data, wherein the type of the fixed fields includes the sequencer identification information of the sequencer that outputs the sequencing data; after decompressing the compressed data to obtain decompressed data, splicing the decompressed data and the fixed fields to obtain sequencing data.

[0013] According to another aspect of the embodiments of this application, a sequencing data compression apparatus is also provided, comprising: a first processing module, configured to determine identifier format information of sequencing data, wherein the identifier format information includes type information of identifiers in the sequencing data; a second processing module, configured to determine the data processing method of the field corresponding to each identifier in the sequencing data based on the identifier type information, and process the field according to the data processing method; and a third processing module, configured to compress each field in the processed sequencing data using a preset compression algorithm.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to execute a sequencing data compression method when it runs.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a sequencing data compression method during runtime.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of a sequencing data compression method.

[0017] In this embodiment, the method involves determining the identifier format information of the sequencing data, wherein the identifier format information includes the type information of the identifiers in the sequencing data; determining the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm. By determining the processing method for each field in the sequencing data based on the identifier format information in the sequencing data to be compressed, the purpose of pre-compressing the sequencing data based on the identifier format information of the sequencing data before compression is achieved, thereby improving the technical effect of improving the compression efficiency of sequencing data. This solves the technical problem of low sequencing data compression efficiency caused by the failure to consider the differences in data formats output by different sequencing platforms when compressing sequencing data in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram of the structure of a computer terminal (mobile terminal) according to an embodiment of this application;

[0020] Figure 2 This is a schematic flowchart of a sequencing data compression method provided according to an embodiment of this application;

[0021] Figure 3 This is a schematic flowchart of a sequencing data compression process provided according to an embodiment of this application;

[0022] Figure 4 This is a comparative diagram of compression ratios provided according to an embodiment of this application;

[0023] Figure 5 This is a comparative schematic diagram of compression time provided according to an embodiment of this application;

[0024] Figure 6This is a comparative diagram illustrating the maximum memory usage during compression according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of a sequencing data compression device provided according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0029] Data compression technology: Data compression technology is a method to reduce data storage space and transmission time by reducing the number of bits required to represent data. Data compression technology is widely used in various fields, including file storage, data transmission, multimedia processing, and genomics. Depending on whether some data is sacrificed to improve the compression ratio, data compression technology is divided into lossy compression and lossless compression. Because gene sequences contain a large amount of biological information, genomic data compression usually employs lossless compression. Commonly used data compression algorithms include Huffman coding, Lempel Ziv (LZ) coding, dictionary algorithms (LZ77 / LZ78), differential encoding, and BSC (Block Sorting Compression).

[0030] Differential Encoding: Differential encoding is a lossless data compression technique based on data variation. It achieves compression by recording the differences between adjacent data points instead of the original data. This method is particularly suitable for data with minimal variation or high continuity.

[0031] The main steps of the difference algorithm:

[0032] Initial value record: Records the original value of the first data point.

[0033] Difference calculation: Calculate the difference between subsequent data points and the previous data point.

[0034] Encoding the difference: Encode and store the difference.

[0035] The basic idea of ​​differential encoding is to reduce data redundancy by recording the differences between data points. Since the differences between adjacent data points are usually smaller than the values ​​of the original data points, fewer bits can be used to represent these differences. Differential encoding is simple to compute and suitable for real-time data compression. A representative tool is TimescaleDB: a time-series database that uses differential encoding to compress time-series data, reducing data redundancy by recording the differences between adjacent data points.

[0036] BSC (Block Sorting Compression): The BSC algorithm is a lossless data compression algorithm based on block sorting. Its core idea is to improve compression efficiency by sorting data blocks to make repeating patterns more obvious. A typical implementation of the BSC algorithm is the Burrows-Wheeler transform (BWT), which is often combined with other compression techniques (such as move-before coding, run-length coding, and Huffman coding) to achieve higher compression ratios.

[0037] Main steps of the BSC algorithm:

[0038] Block Sorting: Sort data blocks so that similar characters are grouped together.

[0039] Transformation: Converting sorted data blocks into a more compressible form.

[0040] Encoding: Encoding the transformed data to achieve compression.

[0041] BWT is the core of the BSC algorithm. It performs specific transformations on the input data to make the clustering effect of identical characters more obvious, thereby improving the efficiency of subsequent compression algorithms.

[0042] How BWT works:

[0043] Construct a rotation matrix: Arrange all cyclic shifts of the input string into a matrix.

[0044] Sorting matrix: Sorts the rows of a matrix in lexicographical order.

[0045] Extract the last column: Extract the last column from the sorted matrix as the transformed output.

[0046] Data transformed by BWT typically has a high degree of repetition and can be further compressed using the following encoding techniques:

[0047] Move-to-Front Encoding (MTF): Converts a character to its position in the character table and moves used characters to the front of the table.

[0048] Run-Length Encoding (RLE): Encodes consecutively repeated characters into pairs of characters and their repetition counts.

[0049] Huffman coding: Based on the frequency of character occurrence, variable-length codes are used instead of fixed-length codes.

[0050] In a compression example, the compression process for the string "BANANA" using the BSC method includes BWT transformation, MTF encoding, RLE encoding, and Huffman encoding. First, BWT transformation yields "NNBAAA" and the original string position 4. Then, MTF encoding of "NNBAAA" is performed, assuming the result is [13,13,1,2,1,1]. Next, RLE encoding is performed on the MTF encoded result, assuming the result is [(13,2),(1,1),(2,1),(1,3)]. Finally, Huffman encoding is performed on the RLE encoded result to obtain the final compressed data.

[0051] A representative compression tool using the BSC method is bzip2. Bzip2 is a widely used file compression tool that uses the Burrows-Wheeler transform (BWT) as its core compression algorithm. bzip2 uses BWT to transform the input data into a more compressible form, and then uses move-before coding (MTF), run-length encoding (RLE), and Huffman coding to achieve efficient compression.

[0052] With the rapid decline in the cost of high-throughput sequencing, the output of gene sequencing data has increased dramatically, posing significant challenges and requiring further development in the storage and transmission of this data. Compressing gene sequencing data has become an important development direction in the field of genomics. Currently, the main formats for storing this data include FASTQ and BAM.

[0053] Taking FASTQ files as an example, these files mainly consist of three parts: sequence identifier (header), base sequence information (seq), and quality score (qual). The following table shows an example of a FASTQ file obtained from sequencing using platform A:

[0054] Table 1. FASTQ File Format (Platform A)

[0055]

[0056] As can be seen, each short read sequence has four lines of information: the first line is the sequence identifier, usually starting with the "@" character, followed by the unique identifier of the sequence and an optional descriptive information; the second line is the sequence line, which stores the base sequence information obtained from sequencing; the third line is the separator, usually starting with "+", which stores additional information; the fourth line is the quality value, composed of ASCII characters, with each character corresponding to a quality score for a base.

[0057] The highly random distribution of sequence information and quality values ​​in genomic data makes it difficult for general compression software such as Gzip and Bzip to achieve efficient compression. However, each sequencing platform has a different standard format for sequence identifiers, which are highly redundant within the same file. This makes efficient compression of sequence identifiers feasible. However, related technologies do not consider the impact of sequence identifiers on the compression process. Specifically, in the compression of genomic data, most compression tools use different methods to compress the sequence identifier lines, sequence lines, and quality lines of FASTQ data. Taking the highly redundant sequence identifiers and their relatively standardized format order as examples (Table 2), several sets of sequence identifiers obtained from sequencing on platform A are used as examples. The data compression algorithms used include Differential Encoding and BSC (Block Sorting Compression). However, the impact of sequence identifiers on the compression process is not considered, nor is the varying adaptability of different compression algorithms in different compression scenarios, resulting in low efficiency in the compression of sequencing data in related technologies. For example, the difference method is suitable for scenarios where data variation is small or continuity is strong. However, the sequence identifier formats of different sequencing platforms are different, and some sequencing platforms produce sequence identifiers with weak continuity, so the difference method often cannot achieve an ideal result. The BSC algorithm also has some limitations. As mentioned above, the sequence identifiers in the A platform format (Table 2) are relatively long and have many attribute fields, while the sequence identifiers in the MGI platform format (Table 3) are relatively shorter. Therefore, multiple rounds of alignment are required during sorting, wasting a lot of time and space resources. Furthermore, some fields in the identifiers are fixed and do not need to be sorted. These resource consumptions greatly limit the scalability and applicability of the compression method.

[0058] Table 2 Sequence Identifier Format (Platform A)

[0059]

[0060]

[0061] Table 3 Sequence Identifier Format (MGI)

[0062]

[0063] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0064] According to an embodiment of this application, a method embodiment for sequencing data compression is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0065] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a sequencing data compression method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0066] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0067] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the sequencing data compression method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned sequencing data compression method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0069] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0070] Under the above operating environment, embodiments of this application provide a sequencing data compression method, such as... Figure 2 As shown, the method includes the following steps:

[0071] Step S202: Determine the identifier format information of the sequencing data, wherein the identifier format information includes the type information of the identifiers in the sequencing data;

[0072] In the technical solution provided in step S202, the step of determining the identifier format information of sequencing data includes: determining the platform identifier information of the sequencing platform that outputs the sequencing data; and determining the identifier format information corresponding to the platform identifier information based on the platform identifier information.

[0073] As an optional implementation, since sequencing data output by different sequencing platforms typically has a fixed sequence identifier format, the identifier format information corresponding to the sequencing data can be determined by judging the sequencing platform that outputs the sequencing data. For example, in the example of platform A above, the number of ":" fields is fixed, and regular expressions can be used to match and select the number of input identifiers ":" and their field types. It should be noted that the sequence identifier format and sequence identifier format information mentioned in this embodiment include the specific sequencing identifiers (such as ":" in the example of platform A above) that will be included in the sequencing data output by each sequencing platform, as well as the meaning of each sequencing identifier and its corresponding field type. For example, taking the sequencing data @A00744:46:HV3C3DSXX:2:1342:20781:99561:N:0:AGC as an example, this sequencing data includes the following identifiers:

[0074] @A00744: The sequencer's ID or name;

[0075] 46: Run number;

[0076] HV3C3DSXX: Flowcell ID;

[0077] 2: Lane number.

[0078] 1342: Tile number.

[0079] 20781: X coordinate.

[0080] 9956: Y coordinate.

[0081] 1: Read number, indicating how many times this is read (e.g., 1 indicates the first read, 2 indicates the second read).

[0082] N: Whether the filter is passed (Y indicates pass, N indicates fail).

[0083] 0: Control number, usually 0, indicating that the sequence is not a control sequence.

[0084] AGC: Sample Index (Index sequence), used to distinguish different samples.

[0085] In some embodiments of this application, in the MGI sequencing platform, assuming the input sequencing tool is @V100002807L3C001R002000247 / 1, the specific meanings of each identifier are as follows:

[0086] @:read ID identifier

[0087] V100002807: Operation Number

[0088] L3: Flow Unit ID

[0089] C001: Cluster Number

[0090] R002000247:read number

[0091] 1: Pairing information, usually 1 or 2

[0092] Step S204: Determine the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and process the fields according to the data processing method;

[0093] As an optional implementation, after determining the identifier format information, such as the identifier type, the distribution of fields in the sequencing data can be further determined based on the identifier type information. Then, a pre-compression processing scheme matching the sequencing data can be determined based on the field distribution scheme. The pre-compression processing scheme includes the processing methods applied to each field in the sequencing data. By pre-processing each field in the sequencing data, compression efficiency can be further improved in subsequent compression steps.

[0094] In the technical solution provided in step S204, the data processing method includes storing the field values ​​of the fields; determining the data processing method for each field corresponding to the identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method, includes: determining fixed field identifiers and corresponding fixed fields in the sequencing data based on the identifier type information, wherein the fixed fields include a field used to identify the sequencer ID or name of the output sequencing data, and the values ​​of the fixed fields in the sequencing data output by the same sequencer are all fixed values; storing the field values ​​of the fixed fields. Furthermore, since the identifier "sequencingr ID or name" field is a fixed value in the same batch of data and this field is the same in each read, the field value only needs to be stored once.

[0095] As an optional implementation, the data processing method includes converting field attributes; determining the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and processing the field according to the data processing method, including: determining the target field in the sequencing data based on the identifier type information; converting the data type of the target field from a first type to a second type, wherein the storage space required to store the target field of the second type is less than the storage space required to store the target field of the first type.

[0096] In some embodiments of this application, the first type includes strings and the second type includes integer types.

[0097] Optionally, in sequencing data files such as FASTQ files, sequence identifiers are generally entered as strings. However, some field types, such as "x-coordinate" and "y-coordinate", are essentially integers. Since storing integers requires fewer bits than storing strings, these field types need to be converted from strings to integers for storage and encoding.

[0098] As an optional implementation method, the data processing method includes classifying fields; determining the data processing method of the fields corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method. The steps include: determining the field type corresponding to each identifier in the sequencing data based on the identifier type information; classifying the fields corresponding to each identifier in the sequencing data according to the field type corresponding to each identifier to obtain multiple field sets, wherein the fields in each field set have the same field type.

[0099] In some embodiments of this application, the step of compressing each field in the processed sequencing data using a preset compression algorithm includes: compressing each field in each field set using the preset compression algorithm.

[0100] Optionally, sequence identifiers have fixed field types, and values ​​of the same field type have high redundancy. Therefore, fields of the same type are grouped together, such as "run number," "x-coordinate," and "control bit," each belonging to a separate category. Then, when using compression algorithms such as BSC, each category of fields can be encoded separately, reducing time complexity and improving the compression ratio.

[0101] Step S206: Compress each field in the processed sequencing data using a preset compression algorithm.

[0102] In the technical solution provided in step S206, the preset compression algorithm can be any compression algorithm, and this application embodiment does not impose any limitations on it. For example, the BSC algorithm can be used to encode and store the processed fields. The decoded data can then be extracted when decompression is needed. Alternatively, existing compression tools such as zdurilcas can also be used for compression.

[0103] According to embodiments of this application, a method is also provided. Figure 3 The sequencing data compression workflow is shown below. From Figure 3 As can be seen from this, the process includes the following steps:

[0104] Step S302: Input the sequence identifier from the sequencing data and match the corresponding sequencing platform;

[0105] Step S304: Determine the corresponding compression scheme based on the matched sequencing platform;

[0106] It should be noted that the compression scheme here refers to the processing method of each field in the sequencing data, including direct storage, format conversion, and classification. Furthermore, in practical applications, regardless of the sequencing platform used, the main steps of the compression method provided in this application embodiment include direct storage, format conversion, and classification, and are not limited to the specific compression scheme provided in this application embodiment.

[0107] Step S306: Process the fields corresponding to each sequence identifier according to the compression scheme, and extract and classify them;

[0108] Step S308: Compress the extracted and categorized fields using a preset compression algorithm.

[0109] Optionally, when the sequencing platform is determined to be A, the compression process provided in this application embodiment includes the following steps:

[0110] The first step is to determine that the sequencing platform is A. After determining the sequencing platform, the input sequence identifiers can be evaluated to select the optimal compression scheme for that platform. For example, in platform A, the number of colons (":") is fixed; regular expressions can be used to match the number of colons and the type of the input identifiers.

[0111] The second step is to select a compression scheme for platform A. Different sequencing platforms have different sequence identifier formats. Based on the input sequence identifier, after the first step, select the compression scheme corresponding to platform A.

[0112] The third step is to store fixed fields. Within the same batch of data, the identifier "sequencing instrument ID or name" field has a fixed value, and this field is identical in every read. Therefore, this field value only needs to be stored once.

[0113] The fourth step is field attribute conversion. Sequence identifiers in FASTQ files (i.e., the sequencing data to be compressed) are generally input as strings. However, some field types, such as "x-coordinate" and "y-coordinate", are essentially integers. Since storing integers requires fewer bits than storing strings, these field types need to be converted from strings to integers for storage and encoding.

[0114] The fifth step is field extraction and classification. Sequence identifiers have fixed field types, and values ​​of the same field type have high redundancy. Fields of the same type are grouped together, such as "Run Number," "Flow Unit ID," "Channel Number," "Title Number," "x-coordinate," "y-coordinate," "Reading Number," "Whether it passed the filter," "Control Bit," and "Sample Index," each classified into its own category. Then, when using compression algorithms such as BSC, each category of fields is encoded separately, which reduces time complexity and improves the compression ratio.

[0115] The sixth step is to encode the algorithm. When decompression is needed, the corresponding stored value can be extracted and the fixed value stored in the third step can be added to decode it.

[0116] In some embodiments of this application, when the sequencing platform is the MGI platform, the compression process includes the following steps:

[0117] The first step is to determine if the sequencing platform is MGI. The input sequence identifier is evaluated, and the optimal compression scheme for the corresponding sequencing platform is selected. For example, in the header of the MGI platform mentioned above, the letter values ​​"V", "L", "C", "R" and the final pairing information "1" and "2" are all fixed values, so they can be matched and selected using regular expressions.

[0118] The second step is to select an MGI compression scheme. Different sequencing platforms use different sequence identifier formats. Based on the input sequence identifier, you can select the compression scheme corresponding to the MGI platform after the first step.

[0119] The third step is to store fixed fields. As shown in the sequencing data corresponding to the MGI platform above, the values ​​of the "read ID identifier" field @ and the "run number" V100002807 are fixed in the same batch of data, and this field is the same in each read. Therefore, this field value only needs to be stored once.

[0120] The fourth step is field attribute conversion. Sequence identifiers in FASTQ files are generally input as strings. However, some field types, such as "cluster number (C001)" and "read number (R002000247)", are essentially integers. Simply remove the leading uppercase letters; the remaining part is the integer value. Since storing integers requires fewer bits than storing strings, converting these field types from strings to integers for storage and encoding reduces storage resource consumption.

[0121] The fifth step is field extraction and classification. Sequence identifiers have fixed field types, and values ​​of the same field type have high redundancy. Therefore, fields of the same type are grouped together, such as "Flow Unit ID," "Cluster Number," and "Read Number" each belonging to a separate category. Then, when using compression algorithms such as BSC, each category of fields is encoded separately, which reduces time complexity and improves the compression ratio.

[0122] The sixth step is to encode the algorithm. Then, when decompression is needed, the corresponding stored value can be extracted and the fixed values ​​stored in the third and first steps added to decode the code.

[0123] As an optional implementation, the sequencing data compression method provided in this application further includes: determining the compressed data after compression processing and the fixed fields corresponding to the compressed data, wherein the type of the fixed fields includes the identifier information of the sequencer that outputs the sequencing data; after decompressing the compressed data to obtain decompressed data, splicing the decompressed data and the fixed fields to obtain sequencing data.

[0124] To further demonstrate the advantages of the sequencing data compression method provided in this application in compressing sequencing data, this application takes NGS data compression as an example and compares the compression results of the sequencing data compression method provided in this application with those of sequencing data compression methods in related technologies.

[0125] Specifically, the testing tools for the test dataset are shown in the table below:

[0126]

[0127] The compression tools tested were zdurilcas and zdurilcas-opt, where zdurilcas is a compression tool in related technologies, and zdurilcas-opt is a compression tool obtained by optimizing zdurilcas using the method provided in the embodiments of this application.

[0128] In the specific compression test, the original version of the compression tool (zdurilcas) and the version using the optimized compression scheme (zdurilcas-opt) were compared. Sufficient memory resources were allocated, and multiple CPU cores and threads were used to compress the same CPU (e.g., 16 CPUs and 16 threads). The specific operation steps are as follows:

[0129] Download the compression testing tool and set up the compilation and runtime environment;

[0130] According to the user manual, run the compression command (specific command: . / zDUR c-simtree--i1NovaSeq_HG002_15x.fq-o NovaSeq_HG002_15x.fq.zdur-t 16);

[0131] Record test results using time-v;

[0132] Test the three sets of data separately, and compile and record the test results for both versions.

[0133] It's important to note that since the compression tool zdurilcas compresses the entire FASTQ file, the time-v output here records the total compression time and maximum memory usage. However, the compression ratio of the sequence identifier will be output separately.

[0134] When comparing the compression results of zdurilcas and zdurilcas-opt, the compression ratio (size before compression to size after compression), compression time, and maximum memory usage for each version can be compared. The compression ratio comparison results are as follows: Figure 4 As shown, the comparison results of compression time are as follows: Figure 5 As shown, the comparison results of maximum memory usage are as follows: Figure 6 As shown. From Figure 4 As can be seen, after using the optimized compression scheme, the compression ratio of sequence identifiers in the three sets of NGS data by the compression tool zdurilicas was improved by 43.95%, 44.07%, and 43.91%, respectively. The average improvement was 43.98%. Figure 5 As can be seen, after using the optimized compression scheme, the compression tool zdurilicas reduced the time required to compress the three sets of NGS data by 7.84%, 7.77%, and 7.14%, respectively, with an average reduction of 7.58%. Figure 6 As can be seen, after using the optimized compression scheme, the maximum memory usage of the compression tool zdurilicas decreased by 0.8%, -1.3%, and -0.5% respectively when compressing the three sets of NGS data. The average reduction was -0.33%.

[0135] In summary, after using the optimized compression scheme, the zdurilicas compression tool significantly improved the sequence identifier compression ratio of the three NGS datasets, while slightly reducing the required compression time and maintaining almost no change in maximum memory usage. In other words, the sequencing data compression method provided in this application can significantly improve compression efficiency without requiring additional hardware resources.

[0136] By employing a method that determines the identifier format information of sequencing data, including the type information of identifiers in the sequencing data; determining the data processing method for each field corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm, the method of determining the processing method for each field in the sequencing data based on the identifier format information in the sequencing data to be compressed achieves the purpose of pre-compressing the sequencing data according to the identifier format information before compression. This achieves the technical effect of improving the compression efficiency of sequencing data, and thus solves the technical problem of low sequencing data compression efficiency caused by the failure to consider the differences in data formats output by different sequencing platforms when compressing sequencing data in related technologies.

[0137] Furthermore, the sequencing data compression method provided in this application offers an optimized compression scheme for sequence identifiers on different sequencing platforms. This reduces compression time and memory consumption while also improving the compression ratio. It alleviates the pressure on data storage and facilitates the transmission of sequencing data.

[0138] This application provides a sequencing data compression device. Figure 7 This is a schematic diagram of the device. From Figure 7 As can be seen from the diagram, the device includes: a first processing module 70, used to determine the identifier format information of the sequencing data, wherein the identifier format information includes the type information of the identifiers in the sequencing data; a second processing module 72, used to determine the data processing method of the field corresponding to each identifier in the sequencing data according to the identifier type information, and to process the field according to the data processing method; and a third processing module 74, used to compress each field in the processed sequencing data using a preset compression algorithm.

[0139] In some embodiments of this application, the step of the first processing module 70 in determining the identifier format information of the sequencing data includes: determining the platform identifier information of the sequencing platform that outputs the sequencing data; and determining the identifier format information corresponding to the platform identifier information based on the platform identifier information.

[0140] In some embodiments of this application, the data processing method includes storing the field values ​​of the fields; the second processing module 72 determines the data processing method of the fields corresponding to each identifier in the sequencing data based on the identifier type information, and processes the fields according to the data processing method, including: determining fixed field identifiers and fixed fields corresponding to the fixed field identifiers in the sequencing data based on the identifier type information, wherein the fixed fields include fields used to identify the ID or name of the sequencer outputting the sequencing data, and the values ​​of the fixed fields in the sequencing data output by the same sequencer are all fixed values; storing the field values ​​of the fixed fields.

[0141] In some embodiments of this application, the data processing method includes classifying fields; the second processing module 72 determines the data processing method of the fields corresponding to each identifier in the sequencing data based on the type information of the identifier, and processes the fields according to the data processing method, including: determining the field type corresponding to each identifier in the sequencing data based on the type information of the identifier; classifying the fields corresponding to each identifier in the sequencing data according to the field type corresponding to each identifier to obtain multiple field sets, wherein the fields in each field set have the same field type.

[0142] In some embodiments of this application, the step of the second processing module 72 compressing each field in the processed sequencing data using a preset compression algorithm includes: compressing each field in each field set using the preset compression algorithm.

[0143] In some embodiments of this application, the data processing method includes converting the field attributes; the second processing module 72 determines the data processing method of the field corresponding to each identifier in the sequencing data based on the identifier type information, and processes the field according to the data processing method, including: determining the target field in the sequencing data based on the identifier type information; converting the data type of the target field from a first type to a second type, wherein the storage space required to store the target field of the second type is less than the storage space required to store the target field of the first type.

[0144] In some embodiments of this application, the first type includes strings and the second type includes integer types.

[0145] In some embodiments of this application, the sequencing data compression device is further configured to: determine the compressed data after compression processing and the fixed fields corresponding to the compressed data, wherein the type of the fixed fields includes the identifier information of the sequencer that outputs the sequencing data; after decompressing the compressed data to obtain decompressed data, splice the decompressed data and the fixed fields to obtain sequencing data.

[0146] It should be noted that each module in the above-mentioned sequencing data compression device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0147] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, which stores a program. When the program runs, it controls the device containing the non-volatile storage medium to perform the following sequencing data compression method: determining identifier format information of the sequencing data, wherein the identifier format information includes type information of the identifiers in the sequencing data; determining the data processing method for the fields corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm.

[0148] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the following sequencing data compression method: determining identifier format information of sequencing data, wherein the identifier format information includes type information of identifiers in the sequencing data; determining the data processing method of the field corresponding to each identifier in the sequencing data according to the identifier type information, and processing the field according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm.

[0149] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following steps of a sequencing data compression method: determining identifier format information of sequencing data, wherein the identifier format information includes type information of identifiers in the sequencing data; determining the data processing method of the fields corresponding to each identifier in the sequencing data based on the identifier type information, and processing the fields according to the data processing method; and compressing each field in the processed sequencing data using a preset compression algorithm.

[0150] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0155] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of sequencing data compression, characterized by, include: Determine the identifier format information of the sequencing data, wherein the identifier format information includes the type information of the identifiers in the sequencing data; Based on the type information of the identifier, determine the data processing method for each field corresponding to the identifier in the sequencing data, and process the field according to the data processing method; A preset compression algorithm is used to compress each field in the processed sequencing data.

2. The sequencing data compression method of claim 1, wherein, The identifier format information for determining sequencing data includes: Determine the platform identifier information of the sequencing platform that outputs the sequencing data; The identifier format information corresponding to the platform identifier information is determined based on the platform identifier information.

3. The sequencing data compression method of claim 1, wherein, The data processing method includes storing the field values ​​of the fields; determining the data processing method for each field corresponding to the identifier in the sequencing data based on the type information of the identifier, and processing the fields according to the data processing method, including: Based on the type information of the identifier, a fixed field identifier and a fixed field corresponding to the fixed field identifier are determined in the sequencing data. The fixed field includes a field for identifying the ID or name of the sequencer that outputs the sequencing data, and the value of the fixed field in the sequencing data output by the same sequencer is a fixed value. Store the field values ​​of the fixed fields.

4. The method of claim 1, wherein, The data processing method includes classifying the fields; determining the data processing method for each field corresponding to the identifier in the sequencing data based on the identifier's type information, and processing the fields according to the data processing method, including: The field type corresponding to each identifier in the sequencing data is determined based on the type information of the identifier; Based on the field type corresponding to each identifier, the fields corresponding to each identifier in the sequencing data are classified to obtain multiple field sets, wherein the fields in each field set have the same field type.

5. The sequencing data compression method of claim 4, wherein, Compression of each field in the processed sequencing data using a preset compression algorithm includes: The preset compression algorithm is used to compress the fields in each of the field sets.

6. The method of claim 1, wherein, The data processing method includes performing field attribute conversion on the fields; determining the data processing method for each field corresponding to the identifier in the sequencing data based on the identifier's type information, and processing the fields according to the data processing method, including: The target field is determined in the sequencing data based on the type information of the identifier; The data type of the target field is converted from a first type to a second type, wherein the storage space required to store the target field of the second type is less than the storage space required to store the target field of the first type.

7. The method of claim 1, wherein, The sequencing data compression method further includes: The compressed data after the compression process is determined, as well as the fixed fields corresponding to the compressed data, wherein the type of the fixed fields includes the identification information of the sequencer that outputs the sequencing data; After decompressing the compressed data to obtain decompressed data, the decompressed data and the fixed fields are concatenated to obtain the sequencing data.

8. A sequencing data compression apparatus, characterized by, include: The first processing module is used to determine the identifier format information of the sequencing data, wherein the identifier format information includes the type information of the identifiers in the sequencing data; The second processing module is used to determine the data processing method of each field corresponding to the identifier in the sequencing data based on the type information of the identifier, and to process the field according to the data processing method; The third processing module is used to compress each field in the processed sequencing data using a preset compression algorithm.

9. A non-volatile storage medium, characterized by, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the sequencing data compression method according to any one of claims 1 to 7.

10. An electronic device, comprising: include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the sequencing data compression method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the sequencing data compression method according to any one of claims 1 to 7.

Images