A bitmap index compression method for genomic variant data

Abstract

The present application belongs to the field of information retrieval and big data analysis, and particularly relates to a bitmap index compression method for genomic variation data, comprising: splitting original genomic variation data sequences according to field attributes, and storing in a columnar database; establishing original bitmap indexes for genomic variation data in the columnar database; dividing the original bitmap indexes into several segments; using a run-length algorithm to compress and store the public part of the segmented bitmap data, and using an integer list or a bit sequence to compress and store the non-public part. The present application saves more storage space by merging public affixes in the bitmap sequence, improves the bit operation speed, and makes the bitmap index load into the memory faster.

Description

Technical Field

[0001] This invention belongs to the fields of information retrieval and big data analysis, and specifically relates to a bitmap index compression method for genomic variation data. Background Technology

[0002] Since the beginning of the 21st century, with the rapid development of human life science technology, the field of life science has generated immeasurable amounts of data. The widespread adoption of high-throughput sequencing technology has pushed the growth rate of data to new heights. For example, the PacBio RS II sequencer can generate 375MB of data in 3 hours; Illumina's latest HISEQ X TEN sequencer can sequence approximately 1.8TB of base data in 3 days. In addition to gene variation data, proteomics data is also growing rapidly, and the preservation of this massive amount of data has become a key issue in the biomedical field. Therefore, countries around the world have established big data centers for the biomedical and health fields. Europe established the European Institute for Bioinformatics (EBI) as early as 1987. This institution has established dozens of authoritative databases covering nucleic acid and protein sequences, genes and gene expression, protein structure, small molecule metabolism, ontology, etc. The nucleic acid sequence database currently contains approximately 3.7 billion records, containing approximately 770 billion bases, and the total data volume of all databases exceeds 20PB. Correspondingly, the United States established the National Center for Biotechnology Information (NCBI) in 1988. Currently, the center has established nearly 40 databases, including PubMed, Ref Seq, and SRA (Sequence Read Archive), containing a total of approximately 6.9 billion records. The SRA database alone contains over 850TB of data. Against this backdrop, the storage and retrieval of biomedical data is of great significance.

[0003] However, due to the sheer size of these genetic variation data, inefficient access to this data has become a bottleneck in the field of genome-wide association studies. Bitmap indexes offer a solution to this problem.

[0004] Because gene mutation data is characterized by its massive scale, rapid growth rate, and lack of deletion and modification operations in the database, bitmap indexing technology is very suitable for retrieving gene mutation data, while traditional relational databases are very inadequate in solving this problem.

[0005] Building a bitmap index for genomic variation data mainly involves creating the original bitmap and encoding / compressing it. Bitmap indexes are built on a per-database attribute column basis, with a corresponding bitmap index created for each column. To control index size and query time, index compression and categorization are necessary. Compression reduces space consumption caused by numerous 0s or 1s in the index, while categorization merges some columns of the bitmap index. For example, values ​​1.01 and 1.02 can be categorized as 1. Categorization reduces the number of columns in the bitmap index, increasing query and storage efficiency.

[0006] Currently, the main bitmap index compression methods are WAH, Concise, and COMPAX.

[0007] The WAH index compression method is the default method for Fastbit bit compression databases, which divides the original bitstream into groups of 31 bits (63 bits for WAH64). There are two types of groups: (1) Literal, which contains both 0 and 1 bits; and (2) Fill, which contains either all 0 bits or all 1 bits.

[0008] Literal type group: The type flag bit is 0, and the remaining 31 bits are the original literal group; Fill type group: It is divided into 1-Fill and 0-Fill. In this case, the type flag bit in the 32 bits is 1, the second bit is used as the Fill type flag (0-Fill is 0, 1-Fill is 1), and the remaining 30 bits are used as a counter to indicate how many consecutive 0-Fill (or 1-Fill) groups have appeared.

[0009] The Concise index compression method also divides the original bitstream into groups of 31 bits each. Groups are also divided into Literal and Fill types, just like above, but the compression method differs.

[0010] Step 1: Add flags and codes to the Fill-group and Literal-group according to the WAH method to form a set of codes in 32-bit words (a flag of 0 is called a literal word, and a flag of 1 is called a fill-word). The difference here is that for the fill-word, only the lower 25 bits are used as counters (the WAH method uses the lower 30 bits as counters, while PLWAH reserves the middle 5 bits as a position list).

[0011] Step 2: Check the word after each fill-word. If the next word is a literal word and is "early identical" (the definition of near identical is that the difference between the literal word and the previous fill-word is less than or equal to s bits, where s is temporarily 1 and will be discussed further later), then fill the position list of the next word (which is a literal word at this time) with the difference position (here it is 31 bits, so the difference bit number is 1-31, the purpose of leaving 5 bits in the first step is for this), and delete the next word (because the information has already been stored in this fill-word, there is no need to keep it). If the word after the fill-word is one of the following three cases: (1) a fill-word of a different type (2) a literal word that is not nearly identical (3) a fill-word of the same type (the reason for this case is that the continuous fill-group exceeds the counting range of the counter of one fill-word), then the position list remains unchanged.

[0012] In summary, the existing technical problems are as follows: there are many bitmap indexing technologies that can be used for correlation query analysis of large amounts of genomic variation data, but they all have their own drawbacks. For example, WAH has low compression efficiency on genomic variation datasets, resulting in large memory consumption; CONCISE and COMPAX compression algorithms are too slow when performing bitmap index AND operations on genomic variation datasets. Summary of the Invention

[0013] To address the aforementioned technical problems, this invention provides a bitmap index compression method for genomic variation data, comprising the following steps:

[0014] S1. Preprocess the genomic variation data in VCF format and save the genotype information of the preprocessed samples into the Kudu columnar database;

[0015] S2. Calculate the column attribute values ​​of the sample genotype information in each column of the Kudu columnar database, construct a bitmap vector with all zeros for each column attribute value, read each genotype sample information in the column in turn, if the genotype sample information is equal to a certain column attribute value, set the position of the record in the bitmap vector corresponding to the column attribute value to 1, and obtain the bitmap index of each column attribute.

[0016] S3. Divide the bitmap index containing each column attribute of n bits into bit segments of the same length;

[0017] S4. Align the beginning and end of the segmented bit segments, and divide the aligned bit segments into common parts and non-common parts according to whether all bit segments have the same value at a certain bit.

[0018] S5. Merge the common parts of all bit segments and store them in an array named α[i] (i = 0, 1, ...), and form a β[i] (i = 0, 1, ...) array of all non-common parts of all bit segments, where i represents the number of elements stored in the array;

[0019] S6. Use run-length encoding to compress and store the α array, and use an integer list or bit sequence array to store the β array according to the data density.

[0020] The beneficial effects of this invention are:

[0021] This invention achieves compressed encoding by constructing a bitmap index for sample genotype information and merging common parts in the bitmap index. Compared with existing Roaring bitmap, WAH, CONCISE and COMPAX methods, it saves more storage space and has a bitwise operation speed that is at least an order of magnitude faster, greatly improving efficiency. Attached Figure Description

[0022] Figure 1 This is a flowchart of a bitmap index compression method for genomic variation data according to the present invention;

[0023] Figure 2 This is a VCF file style diagram of the present invention;

[0024] Figure 3 This is a schematic diagram of bitmap index segmentation according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] A bitmap index compression method for genomic variation data, such as Figure 1 As shown, it includes:

[0027] S1. Preprocess the genomic variation data in VCF format and save the genotype information of the preprocessed samples into the Kudu columnar database;

[0028] S2. Calculate the column attribute values ​​of the sample genotype information in each column of the Kudu columnar database, construct a bitmap vector with all zeros for each column attribute value, read each genotype sample information in the column in turn, if the genotype sample information is equal to a certain column attribute value, set the position of the record in the bitmap vector corresponding to the column attribute value to 1, and obtain the bitmap index of each column attribute.

[0029] S3. Divide the bitmap index containing each column attribute of n bits into bit segments of the same length;

[0030] S4. Align the beginning and end of the segmented bit segments, and divide the aligned bit segments into common parts and non-common parts according to whether all bit segments have the same value at a certain bit.

[0031] S5. Merge the common parts of all bit segments and store them in an array named α[i] (i = 0, 1, ...), and form a β[i] (i = 0, 1, ...) array of all non-common parts of all bit segments, where i represents the number of elements stored in the array;

[0032] S6. Use run-length encoding to compress and store the α array, and use an integer list or bit sequence array to store the β array according to the data density.

[0033] like Figure 2 The image shown is a diagram of a VCF file style. Preprocessing and saving VCF-formatted genomic variation data into a columnar database includes:

[0034] The header data of the VCF format genomic variation data is removed by removing the "##" header data. The VCF format genomic variation data with the header data removed is then vertically segmented to extract metadata information and sample genotype information. The sample genotype data is saved as a separate data file according to phenotype and imported into a columnar database for storage.

[0035] The metadata information includes the first 9 columns of the VCF format file containing the cut-off header data information, including: CHROM, POS, ALT, REF, QUAL, ID, FILTER, INFO, and FORMAT.

[0036] like Figure 3 As shown, the bitmap index for each column attribute containing n bits is divided into bit segments of equal length. The length of the bit segment is determined by the data block length n, which is also the length of the bitmap index. Specifically, the segment length is...

[0037] The α array is compressed and stored using run-length encoding, including:

[0038] The bitmap index of the first bit of each common part of the bits stored in the α array is used as the run length, and the run length is the number of bits of the bit segment with the same bitmap index value as the first bit. This is denoted as "XY", where X represents the bitmap index value of the common part and Y represents the number of bits of the bit segment with the same bitmap index value.

[0039] Specifically, if the value in the α array is "00000000011", its run length is encoded as follows:

[0040] The first run is "0" and has a length of 9, so it is represented by the symbol "0" and the value "9", i.e., "09"; the second run is "1" and has a length of 2, so it is represented by the symbol "1" and the value "2", i.e., "12". Therefore, the run length of the α array "00000000011" is encoded as "0912".

[0041] The β array can be stored using either an integer list or a bit sequence array, depending on the data density, including:

[0042] Depending on whether the number of bitmap index values ​​of "1" in the β array exceeds 1 / 2 of the length of the β array, the positions of the 1s are recorded using an integer list and all bits are stored using a bit sequence array. If the number of bitmap index values ​​of "1" in the β array does not exceed 1 / 2, the integer list is used for storage; otherwise, the original bit sequence is used for storage.

[0043] The method of using an integer list for storage is as follows: if the number of bitmap index values ​​of "1" in the β array does not exceed 1 / 2, then the number of bits at the bitmap index where the bitmap index value of "1" is located is stored.

[0044] Specifically, an integer list is used for storage: if the non-common part of the β array contains the sequence "0,1,0,0,0,1,0,0,0", and 1 appears in the 2nd and 6th positions, then the integer list storage will store the two numbers 2 and 6.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A bitmap index compression method for genomic variation data, characterized in that, include: S1. Preprocess the genomic variation data in VCF format and save the genotype information of the preprocessed samples into the Kudu columnar database; S2. Calculate the column attribute values ​​of the sample genotype information in each column of the Kudu columnar database, construct a bitmap vector with all zeros for each column attribute value, read each genotype sample information in the column in turn, if the genotype sample information is equal to a certain column attribute value, set the position of the record in the bitmap vector corresponding to the column attribute value to 1, and obtain the bitmap index of each column attribute. S3. Divide the bitmap index containing each column attribute of n bits into bit segments of the same length; S4. Align the beginning and end of the segmented bit segments, and divide the aligned bit segments into common parts and non-common parts according to whether all bit segments have the same value at a certain bit. S5. Merge the common parts of all bit segments and store them in an array named α[i] (i=0,1…), and form a β[i] (i=0,1…) array of the non-common parts of all bit segments, where i represents the number of elements stored in the array; S6. Use run-length encoding to compress and store the α array, and use an integer list or bit sequence array to store the β array according to the data density. The α array is compressed and stored using run-length encoding, including: The bitmap index of the first bit of each common part of the bits stored in the α array is used as the run length, and the run length is the number of bits of the bit segment with the same bitmap index value as the first bit. This is denoted as "XY", where X represents the bitmap index value of the common part and Y represents the number of bits of the bit segment with the same bitmap index value. The β array can be stored using either an integer list or a bit sequence array, depending on the data density, including: The storage method is selected based on whether the number of bitmap index values ​​of "1" in the β array exceeds 1 / 2 of the length of the β array. If the number of bitmap index values ​​of "1" in the β array does not exceed 1 / 2, the number of bits with bitmap index values ​​of "1" is stored using an integer list; otherwise, the original bit sequence array is used for storage.

2. The bitmap index compression method for genomic variation data according to claim 1, characterized in that, Preprocessing and storing VCF-formatted genomic variation data into a columnar database includes: The header data of the VCF format genomic variation data is removed by removing the "##" header data. The VCF format genomic variation data with the header data removed is then vertically segmented to extract metadata information and sample genotype information. The sample genotype data is saved as a separate data file according to phenotype and imported into a columnar database for storage.

3. The bitmap index compression method for genomic variation data according to claim 2, characterized in that, The metadata information includes the first 9 columns of the VCF format file containing the cut-off header data information, including CHROM, POS, ID, ALT, REF, QUAL, FILTER, INFO, and FORMAT.

4. The bitmap index compression method for genomic variation data according to claim 1, characterized in that, The use of an integer list for storage includes: If the number of bitmap index values ​​of "1" in the β array does not exceed 1 / 2, then store the number of bits at the bit position where the bitmap index value of "1" is located.

Images