Customizable delimited text compression framework

Abstract

A method for compressing data includes obtaining a compression schema customized for a delimited text file format, parsing the delimited text file into multiple data blocks using the compression schema, dividing each of the data blocks into multiple data units for efficient selective access, and compressing the multiple data units in the multiple data blocks using different compression algorithms for improved compression. The delimited file is divided into the multiple data blocks based on region definitions in the schema. Each of the multiple data blocks is divided into multiple data units based on its respective data unit size specified in the schema. The multiple data units in each of the multiple data blocks are compressed using different compression algorithms indicated by compression instructions in the schema. The compressed file consists of the compressed data blocks, the compression schema, and various metadata for data decompression, file recovery, and functions such as data security and search queries.

Description

Technical Field

[0001] Cross - reference to Related Applications

[0001] This application is related to U.S. Provisional Patent Application No. 62 / 923,113, filed on October 18, 2019, and the entire contents of that U.S. Provisional Patent Application are incorporated herein by reference for all purposes.

[0002]

[0002] This application is related to U.S. Provisional Patent Application No. 62 / 923,141, filed on October 18, 2019, and the entire contents of that U.S. Provisional Patent Application are incorporated herein by reference for all purposes.

[0003]

[0003] This application is related to U.S. Provisional Patent Application No. 62 / 956,952, entitled "Systems and Methods for Effective Compression, Representation, and Decompression of Various Displayed Data" (Attorney Docket No.: 2019P00842US01), filed simultaneously with this application, and the entire contents of that U.S. Provisional Patent Application are incorporated herein by reference for all purposes.

[0004]

[0004] The various embodiments described herein relate to data compression, and more particularly, but not exclusively, to the compression of delimited text.

Background Art

[0005]

[0005] Particularly in the fields of genomics science, bioinformatics, and analytics in medicine, many large - scale data files are, by nature, delimited text, varying in their row and column definitions and other format details. Examples of genomic data in delimited text include Variant Call Format (VCF), gene expression data, Browser Extensible Data (BED), BigBed, GFF3, GTF, Wig, BedGraph, and BigWig, among others.

Summary of the Invention

Problems to be Solved by the Invention

[0006]

[0006] Various techniques have been proposed to compress data and other types of delimited files. One example of such a compression technique is gzip. However, delimited files are not suitable for compression by all types of compression techniques. Also, existing approaches to compressing delimited files use the same algorithm to compress all parts of the file. In addition, some compressors lack support for desired features (such as fast querying and random access, encryption, authentication, and access control). For at least these reasons, existing compression performance for delimited files has proven to be suboptimal.

[0007]

[0007] A brief overview of various exemplary embodiments is presented below. While some simplifications and omissions have been made in the following overview, it is intended to highlight and introduce some aspects of various exemplary embodiments, but not to limit the scope of the invention. A detailed description of exemplary embodiments suitable for enabling those skilled in the art to create and use the concepts of the invention follows in later sections. [Means for solving the problem]

[0008]

[0008] In one or more embodiments, a method for compressing data comprises the steps of: obtaining a compression schema customized to the format of a delimited text file; paring the delimited text file into a plurality of data blocks based on the compression schema; dividing each of the data blocks into a plurality of data units based on the compression schema; and compressing the plurality of data units in the plurality of data blocks using different compression algorithms, wherein the delimited text file is pared into a plurality of data blocks based on the area definitions in the schema, each of the plurality of data blocks is divided into a plurality of data units based on its respective data unit size in the schema, and the plurality of data units in each of the plurality of data blocks are compressed using different compression algorithms indicated by the compression instructions in the schema.

[0009]

[0009] The step of obtaining a compression scheme includes creating a new compression scheme or determining the best match from a set of compression schemes based on information entered by the user or the extension of a delimited text file, each of which is customized for each of several different formats of delimited text files.

[0010]

[0010] The step of obtaining a compression schema includes the steps of automatically parsing or detecting the format of the delimited text file, and automatically generating a new compression schema for optimal compression performance, or selecting the best match from multiple compression schemas stored in the schema repository, each of which is customized for each of multiple different formats of delimited text files. The files corresponding to the compression schemas stored in the schema repository have predetermined file extensions that point to multiple different formats of delimited text files.

[0011]

[0011] The method includes the step of creating a compression scheme customized to the format of a delimited text file based on a tool having a graphical user interface, the graphical user interface including a predetermined window that allows input of information to customize the compression scheme to the format of a delimited text file.

[0012]

[0012] The method includes the steps of generating a compressed file consisting of multiple compressed data units in multiple data blocks, and a compression schema that includes instructions for decompressing the multiple compressed data units and restoring the compressed file. The compressed file includes metadata information for decompression, file restoration and extended functionality. Extended functionality includes data security and search queries.

[0013]

[0013] A compressed file includes a definition of the code and usage of a special compression / decompression algorithm for the portability and accessibility of the compressed file. A compression instruction specifies different compression algorithms and their corresponding parameters that are used to compress different parts of multiple data units based on the different content of the data blocks.

[0014]

[0014] The compression instruction indicates that a first type of compression algorithm is used to compress a first data unit which includes a first group consisting of a certain type of value, a certain type of information, a certain type of data format, and a certain type of data structure, and that a second type of compression algorithm is used to compress a second data unit which includes a second group consisting of a certain type of value, a certain type of information, a certain type of data format, and a certain type of data structure, which is different from the first group.

[0015]

[0015] According to one or more embodiments, a method for selective data access includes receiving information indicating a region of interest in data (e.g., a range of rows and columns in a table), wherein the region of interest corresponds to one or more data units contained in at least one data block in a compressed file; selectively decompressing one or more data units of at least one data block associated with the region of interest in the compressed file, without decompressing at least one data block in the compressed file or other data units in other data blocks, wherein one or more data units are selectively decompressed based on one or more decompression algorithms indicated by a compression instruction in the compression schema; restoring a region of interest from the selectively decompressed one or more first data units, wherein the region of interest is restored based on a region definition in the compression schema or an output format defined by any user; and outputting information indicating the restored region of interest.

[0016]

[0016] Determining a compression schema involves selecting a compression schema from a plurality of compression schemas, each of which is customized to include decompression information for each of several different formats corresponding to the compressed file. Determining a compression schema involves selecting a compression schema from these plurality of compression schemas stored in a schema repository.

[0017]

[0017] The method includes the step of selectively accessing one or more data units based on a query of a compressed file, the query being performed based on one or more conditions or ranges of values ​​found in the one or more data units that are selectively decompressed. The delimited text file may contain genomic information, and the region of interest may correspond to a selected range of genomic coordinates or gene IDs.

[0018]

[0018] According to one or more embodiments, a system for compressing data includes: a schema manager that allows a user to create, select, or automatically generate a compression schema customized to the format of a delimited text file; a parser that parses the delimited text file into multiple data blocks based on the area definitions in the compression schema; a splitter that divides each of the data blocks into multiple data units based on their respective data unit sizes as specified in the compression schema; and a compression manager that compresses the multiple data units in the multiple data blocks using different compression algorithms indicated by compression instructions in the compression schema.

[0019]

[0019] The schema manager creates a new compression schema or determines the best match from multiple compression schemas based on user input or the extension of the delimited text file, each of which is customized for each of multiple different formats of delimited text files. The schema manager automatically parses or detects the format of the delimited text file and automatically generates a new compression schema for optimal compression performance or selects the best match from multiple compression schemas stored in the schema repository, each of which is customized for each of multiple different formats of delimited text files.

[0020]

[0020] The compression manager instantiates a compressor for each data block by extracting the code for the compression algorithm from the compressor repository or special metadata, allocating computing resources and memory, and operating and monitoring the compression of the data unit.

[0021]

[0021] In the accompanying drawings, like reference numerals refer to the same or functionally similar elements throughout different figures, as will be described in detail below. These drawings are incorporated into the specification and form a part thereof, and function to further illustrate exemplary embodiments of the concepts found in the claims and to explain various principles and the advantages of these embodiments.

[0022]

[0022] These and other more detailed and specific features are more fully disclosed in the following specification with reference to the following accompanying drawings.

Brief Description of the Drawings

[0023] [Figure 1]

[0023] It is an illustration of an embodiment of a method for generating a compression schema for a delimited text file. [Figure 2A]

[0024] It is an illustration of an example (or examples) of an instruction table for a first compression schema. [Figure 2B] It is an illustration of an example (or examples) of an instruction table for a first compression schema. [Figure 3A]

[0025] It is an illustration of an example (or examples) of an instruction table for a second compression schema. [Figure 3B] It is an illustration of an example (or examples) of an instruction table for a second compression schema. [Figure 4]

[0026] It is an illustration of an embodiment of a system for decomposing and compressing a delimited text file. [Figure 5]

[0027] It is an illustration of an embodiment of a method for decomposing and compressing a delimited text file. [Figure 6]

[0028] It is an illustration of an embodiment of a system for decompressing and constructing from a compressed delimited text file. [Figure 7]

[0029] This is a diagram illustrating one embodiment of a method for decompressing and organizing compressed delimited text files. [Figure 8]

[0030] This is a diagram illustrating one embodiment of a method for selecting and decompressing one or more blocks in a compressed delimited text file corresponding to a region of interest. [Figure 9]

[0031] This is a diagram illustrating one embodiment of a processing system used to implement the operations of the embodiments described herein. [Modes for carrying out the invention]

[0024]

[0032] The descriptions and drawings presented herein illustrate various principles. Those skilled in the art will understand that it is possible to embody these principles and devise various configurations that fall within the scope of this disclosure, even if they are not expressly described or shown herein. The term “or” as used herein means non-exclusive “or” unless otherwise indicated (for example, “exclusive OR” or “or in the alternative case”). Furthermore, the various embodiments described herein are not necessarily mutually exclusive and can be combined to produce additional embodiments incorporating the principles described herein.

[0025]

[0033] One or more embodiments described herein relate to systems and methods for providing data representation and compression frameworks for various types of information, including, but not limited to, data relating to genomes and / or bioinformatics. In one application, the system and method provides a data representation and compression framework for delimited text files. Unlike other methods previously proposed, different parts of the same delimited text file are parsed and compressed using different compression techniques. The compression technique used for each part may be optimized for compressing the data in that part, but may not be optimal for other parts. Thus, the delimited text file is compressed in a manner that is customized and optimal for specific parts of the same file or for the specific type of file being considered. Also, in at least some embodiments, the file data is represented and compressed using advanced features that facilitate screening, manipulation, and analysis of downstream data.

[0026]

[0034] Furthermore, by compressing different parts of the same delimited text file separately (using either the same or different compression algorithms), only the selected parts of the delimited text file can be retrieved, decompressed, and organized independently of other parts of the same file that are not of interest. This improves decompression efficiency and allows access to only those parts of the file independently of other parts. Thus, various embodiments present a customizable delimited text compression (CDTC) framework that allows for easy and flexible individual adaptation for lossless compression of various data formats in delimited text, with the aim of efficient storage and processing.

[0027] Compression schema

[0035] Figure 1 illustrates an embodiment of a method for generating a compression schema, which is used to decompose and compress different parts of a delimited text file using different compression algorithms, and also serves as a basis for selectively decompressing and reconstructing parts of the compressed delimited text file. Based on this compression schema, users can easily and flexibly customize how a delimited text file is divided into different components (e.g., sublines, lines, rows, columns, matrices, etc.) and how each component is compressed and stored (e.g., by different compression algorithms and their corresponding parameters). In one embodiment, the compression schema includes a list of compression instructions and global parameters configured in a predetermined format, including, but not limited to, a table format.

[0028]

[0036] Referring to Figure 1, this method obtains a delimited text file that is compressed and then decompressed in 110. This text file can be of any size and contain any type of data, but at least one embodiment is particularly suitable for storing large files. For example, in one particularly useful application, this text file contains genetic information that is broken down into data blocks and individually compressed into data units for later storage and use for research or other purposes.

[0029]

[0037] This text file is delimited in the sense that each line represents a unit or block and has a format in which fields are separated by a delimiter symbol or value. In another embodiment, a unit or block corresponds to another size or portion of a file, such as a portion of a line, a given group of lines, or one or more other types, sizes, or sections of a text file, each separated (or delimited) by a given symbol or value. The units or blocks in which the file is delimited may have the same size, or at least some of them may have different sizes, for example, according to how the schema is defined.

[0030]

[0038] In version 120, for example, given a specific type of information contained in a delimited text file, a set of global parameters is selected to define the compression scheme. These parameters, among other information, define the delimiter, the default data unit size, and the default, common compression algorithm used for different parts of the file. According to one embodiment, the following set of global parameters is selected and defined for the compression scheme:

[0031]

[0039] Input File The schema parameter includes a pointer to the delimited text file to be compressed. This pointer indicates, for example, the location / address of one or more memory or other storage devices in which the delimited text file is stored in an uncompressed format. The memory is located remotely from the processing system that implements the embodiments described herein, or it is coupled locally to the processing system. In some embodiments, the memory or other storage device is connected to the processing system through one or more networks, which include, but are not limited to, a VPN, the Internet, a cloud-based network, or other types of networks.

[0032]

[0040] Delimiters Schema parameters also include one or more symbols that function as delimiters in text files. These symbols separate data from other information in a text file into individual fields or components of the same nature that can be collectively compressed due to their common data characteristics. These fields or components correspond to any of the fields or components described herein. In some embodiments, rows in a text file are separated by one or more symbols so that each row (e.g., a unit) is divided into one or more columns in the file. An example of a delimiter symbol is the tab symbol ("\t").

[0033]

[0041] In some embodiments, as will be described in more detail later, a file contains one or more columns of data, which are referred to, for example, as data blocks. Each data block contains one or more data units; that is, in some cases the entire data block is considered a single data unit, and in other cases the data block (for example, a column of data) contains multiple data units.

[0034]

[0042] Encap_Symbol Schema parameters also include encapsulation symbols that indicate that text between symbols, if any, should not be separated into columns by delimiters. An example of an encapsulation symbol is the double quotation mark ("").

[0035]

[0043] Comment_Symbol The schema parameter also includes a comment symbol to indicate comment lines at the beginning of a portion of a text file, such as the beginning of a row. Comment lines remain intact and are stored together with the rest of the file, even after the delimited text file has been disassembled, using a default block name (e.g., "comments"). This includes comment lines in areas defined in compression instructions. An example of a comment symbol is the hash character ("#").

[0036]

[0044] Gen_Comp_Alg The schema parameter also indicates a general compression algorithm to be applied to blocks where no specific compression algorithm is specified in the schema. As described herein, in one or more embodiments, different data blocks of a delimited text file, each consisting of one or more data units, are compressed using different compression algorithms. If no compression algorithm is specified in the schema for a particular data block, that data block is compressed using the general compression algorithm identified by this parameter. Thus, this general compression algorithm can be considered the default algorithm when no other algorithm is specified.

[0037]

[0045] In one embodiment, the entire file, i.e., all data blocks and their respective data units, are compressed using the same or different compression algorithm. In another embodiment, different parts of the file are selectively compressed, for example, using different compression algorithms. For example, a file may contain one or more columns of data, which may be referred to as data blocks. Each data block contains one or more data units; that is, in some cases, the entire data block is considered a single data unit, while in other cases, a data block (e.g., a column of data) may contain multiple data units. In an embodiment of selective compression, for example, compression is applied only to selected data blocks or data units using the respective algorithms described in the schema, and the rest is stored uncompressed. This approach is useful when certain data blocks are frequently accessed or targeted by queries and should remain uncompressed to facilitate data retrieval.

[0038]

[0046] Default_Data_Unit_Size The schema parameter also indicates the default number of rows that form or define data units for compression. In some embodiments, this parameter indicates a predetermined fixed integer value. In some embodiments, this parameter indicates that the processor should execute an algorithm that implements the "Auto" function, which includes automatically selecting the size for each block based on its impact on the compression ratio and decompression speed of a single data unit for selective access. In some embodiments, the parameter indicates that the "Inf" function should be executed, which includes compressing the data block as a whole without dividing it into individual data units.

[0039]

[0047] Output_Folder The schema parameters also include an output folder for storing the compressed data portion and associated metadata. Examples of metadata will be discussed in more detail later.

[0040]

[0048] In version 130, a compression instruction table is generated / customized and included in the schema. Once the compression instruction table is included, each row defines (i) a specific region in the delimited file for data extraction, and (ii) how the extracted data should be represented and compressed. This table indicates that different compression algorithms should be used to compress different regions within a given region.

[0041]

[0049] Therefore, such a table contains information that instructs the processor to compress different areas (or parts) of a data file using different compression algorithms. This is beneficial for several reasons. For example, data or information in one area or part of a file is compressed by an algorithm that is deemed more efficient for that type of data or information. Data or information in other areas or parts of the file is compressed by a different algorithm that is more efficient for the data or information in those parts.

[0042]

[0050] In one embodiment, the compression table is configured to include fields that specify the types of information shown below.

[0043]

[0051] Region_Lines The table contains fields indicating the range of lines in a rectangular area (or other unit or block) in a delimited text file to which the current row of the compression instruction should be applied. For example: "100:500" defines the area from line 100 to line 500. "100:" indicates the region starting from line 100 and continuing until a blank line or the end of the file is reached.

[0044]

[0052] If the table does not specify a range of lines for a row, the control software instructs the processor to use the same range of lines used for the previous row. If that row is the first row, the control software instructs the system processor to start from the next uncommented / empty line and continue until it reaches a blank line or the end of the file.

[0045]

[0053] Region_Outs The table contains fields that show the set of column indices for the rectangular regions (or other units or blocks) in the delimited text file to which the current compression instruction should be applied. This is, for example: "11:15" indicates that columns 11-15 are extracted into a matrix with five columns. "11:2:15" indicates that columns 11, 13, and 15 (with a spacing of 2) are extracted into a matrix with three columns. "11-15" indicates that columns 11-15 are broken down into a single column while retaining their delimiters.

[0046]

[0054] If not specified, the remainder of the line (after the rightmost column previously defined for the same range of the line) is extracted as a single column and is not further divided by the delimiter.

[0047]

[0055] DatabaseType Tables also indicate fields that represent certain types of data elements. Examples of these types include string, fstring (formatted string), char, int, uint (unsigned integer), and float. The number of characters or bits is specified, for example, within brackets; for example, char(8) means 8 characters, and uint(8) means an 8-bit unsigned integer. For the data type fstring, the string format is specified as a string enclosed in brackets; for example, ("rs %uint(24)") represents a string element that begins with the prefix "rs" followed by an unsigned integer. If not specified, the data type is automatically selected by the system processor to correspond to the default type or to optimize performance. Furthermore, if values ​​in a data block are used for query access, a "key" qualifier may be included in the definition of the data type. In such cases, a search index is generated for the data block and stored separately as a metadata component.

[0048]

[0056] Comp_Alg The table also includes fields indicating the names of the compression algorithms and their parameters, if any, for each region / block in the delimited text file. In some embodiments, the type of compression algorithm used is determined based on the content of the region / block being compressed. For example, a region / block containing numbers is compressed using a different algorithm than the one used for formatted strings. In some embodiments, if a formatted string contains multiple data elements, comma-separated compression algorithms are identified for each data element in the same order. The following is a non-exhaustive list of examples of compression algorithms: • Run-Length Coding (RLE): This type of compression algorithm is used to compress long, continuous data elements with the same value, such as genotype values ​​in single nucleotide polymorphism (SNP) array data. • "Delta" (delta coding): This type of compression algorithm is applicable to numbers that code only the difference between the current element and the previous element, rather than storing the entire value. This algorithm is used, for example, for genome coordinates. • Enumeration: This type of compression algorithm is used when the data to be compressed contains repetitive items selected from a small set of possible values. In this case, compression is achieved by coding each unique value with a fixed minimum number of bits that is long enough to cover all possible values. Enumeration compression is used, for example, for feature prediction of variants (missense, nonsense, silent, frameshift, splice-site, etc.). • "Index": This type of compression algorithm is used when the data to be compressed contains a set of values ​​with a fixed format and numerical components that increase or decrease at regular intervals. In this case, compression is performed by deriving and storing (i) the data format, (ii) the initial values ​​of the numerical components, (iii) the interval, and (iv) the number of elements. • "Sparse": This type of compression algorithm is used when the data to be compressed contains a sparse matrix where most elements are default values. In this case, compression is performed by transforming the matrix into a matrix market-like coordinate format that contains only the low index, column index, and non-default entries. Furthermore, any symmetry of the matrix is ​​utilized by storing only entries from the lower triangular portion. This approach is used, for example, for genotype values ​​in NGS data. • General-purpose compressor: Users can specify one of the general-purpose compression algorithms (e.g., gzip, bzip2, 7-zip, or arithmetic coding) to compress general data types that do not fit into any of the above categories. • Compound Algorithms: Users can specify a set of coding algorithms to be performed sequentially on a data unit. For example, "Enum+RLE" means first converting the original data to an enumeration code, and then applying RLE to the converted value. • "Auto": This value indicates that the software controlling the system processor will select a compression algorithm based on data analysis. The selected compression algorithm must be recorded for proper decompression. • "Default" or "" (blank): This value indicates that the general compression algorithm defined in the global parameter Gen_Comp_Alg should be applied. • "Original": This value indicates that the original data in the region / block of the delimited text file should be stored without compression. This allows for faster selective access queries to the data field.

[0049]

[0057] Data_Unit_Size The table also includes a field indicating whether the size of the data unit deviates from the default value Default_Data_Unit_Size in the global parameters. Similarly, its value can be an integer, either "Auto" or "Inf".

[0050]

[0058] Column_Name The table also includes a field indicating the names of (one or more) columns covered by the defined area. In one embodiment, the user specifies a comma-separated string of column names or uses a reserved expression, "First_Row," to indicate that the first row contains the column names and should not be compressed with the remaining rows. If not specified, a name is automatically generated for each column.

[0051]

[0059] Block_Name The table also includes a field that indicates a name that uniquely identifies the data compression block. If not specified, Column_Name is used.

[0052]

[0060] In one embodiment, the user creates compression and associated decompression algorithms to process a specific data type. For protection against malicious software, each compressor / decompressor is accompanied by a digital signature as proof of origin and authenticity. In some embodiments, such a digital signature is required for the algorithm created by the user. The executable file, along with its digital signature, is imported into the compressor / decompressor repository along with the associated ID and method signature (list of input parameters) used in the schema definition or stored as part of the compressed data file for portability and accessibility. In some scenarios, a certain algorithm requires data from another column or block as input. This is supported, for example, by the user identifying a column / block that is pre-fixed by a special character such as "$" as part of the method signature in Comp_Alg.

[0053]

[0061] Rows in the instruction table are ordered based on the location of the defined regions. In some embodiments, regions with fewer starting lines must come first. If multiple regions have the same number of starting lines, the region with the smaller starting column index comes first. Blocks of entire lines not covered by the instruction table are collected along with other comments / blank lines for compression. The number of those lines in the original text is stored as metadata for future file recovery. Any other regions missing from the instruction table are identified by the software as separate blocks to be compressed using the algorithm defined by the global parameter Gen_Comp_Alg. In some embodiments, a Region_Error is returned if there is any ambiguity or overlap in the region definition. In some embodiments, the definitions of global parameters and the instruction table are scattered throughout the schema to allow for changes to the global parameter between compression instructions.

[0054]

[0062] The command for a group of blocks is: <blocks>< / blocks> Each block is marked by labels such as, <block>< / block> These are marked by labels such as . The fields described earlier are then identified as attributes to these labels. In at least one implementation example, each block is divided into subblocks, for example, through a nested block structure.

[0055]

[0063] In some embodiments, the start and end points of each data table are: It is surrounded by labels such as: • ID: This is the name of the table. • Start_Line: The line number of the first row in the table, including the header if it exists. If not specified, it starts from the current position of the file parser. • Num_Rows: The number of rows in the table. If not specified, the table terminates when it reaches a blank line or the end of the file. • First_Row_Col_Names: If true, the first row contains the names of columns that are processed separately from the data entry and stored in metadata. The default value is false. • First_Col_Row_Names: If true, the first column contains the row names, which are processed separately from the data entries and stored in metadata. The default value is false. • Col_Names: A list of column names in the same order as the columns in the table. • Row_Names: A list of row names in the same order as the rows in the table. • Col_Span: A list of integer values, each corresponding to a column name, indicating the column name and the number of associated data columns. This is useful for grouping multiple data columns under the same header. Unless otherwise specified, a one-to-one mapping is assumed between the column name and the data column.

[0056]

[0064] In table definitions, the same data elements (e.g., column names) are defined at the table or block level. In such cases, later values ​​take precedence over earlier values. Data elements in a table are named according to a hierarchical naming approach. For example, suppose a table has an ID "Tab1" with four columns, where the first two columns are named "Col_1" and "Col_2", and columns 3 and 4 are grouped under the name "Cols_3_4". Then all columns are named Tab1.cols, the first column is named Tab1.col[1] or Tab1.col["Col_1"], the fourth column is named Tab1.col[4] or Tab1.col["Col_3_4"][2] (for example, the second column is grouped under "Col_3_4").

[0057]

[0065] Figures 2A and 2B illustrate example instruction tables for a first type of compression schema, illustrating how blocks for compression are defined. Figure 2A contains information for splitting the original delimited text into blocks that will be individually compressed. Figure 2B illustrates a table of relevant instructions for performing the split in equivalent extended and compact formats. The compact table indicates the region starting from the fifth line, so the first four rows in the file should be compressed as general text. Rows 2-4 in the extended format collapse into a single row in the compact format because the same compression instruction is applied to three columns. The "First_Row" entry indicates that the column names should be extracted from the first row of each column.

[0058]

[0066] Figures 3A and 3B illustrate examples of instruction tables for a second type of compression schema, illustrating how blocks for compression are defined. Figure 3A contains information for dividing the original delimited text into blocks that will be individually compressed. Figure 3B illustrates a table of the relevant instructions for performing the division. The table shows a region starting from the fifth line, so the first four rows should be compressed as general text. For lines 5-8, the column "2:3" indicates that columns 2 and 3 should be compressed and stored separately. For lines 9-10, the hyphen "2-3" indicates that columns 2 and 3 should be merged into a single column for compression.

[0059]

[0067] The use of compression schemas is particularly useful for at least some applications because it allows users to design compression schemas according to specific applications. The schema and its associated compression and decompression features, therefore, allow for the customization of one or more embodiments, while simultaneously enabling selective access to only the parts to be decompressed (e.g., data blocks, data units within data blocks) without requiring the decompression of other parts of the compressed file. This not only allows for targeting only specific parts of a compressed file for access, but also speeds up the process of accessing the target portion of genomic data when the file relates to such applications, by excluding other parts (e.g., those not of immediate interest) from decompression.

[0060]

[0068] In 140, the compression schema is stored in storage, such as a schema repository, but is not limited to this. The compression schema is then retrieved to guide the processor to perform operations including decomposing a delimited text file, compressing different parts of the decomposed file using different compression algorithms, decompressing the compressed parts of the file, and restoring the file from the decompressed parts. The compression schema is stored with or in association with metadata as described herein.

[0061] File decomposition and compression

[0069] Figure 4 illustrates an embodiment of a system for decomposing and compressing a delimited text file, for example, containing genomic information. Figure 5 illustrates an embodiment of a method for decomposing and compressing a delimited text file, for example, performed by the system in Figure 4.

[0062]

[0070] Referring to Figures 4 and 5, this method includes uploading a delimited text file 405 from a data source to the system's file manager in 510. The data source is, for example, a computer or other type of processing system that captured and / or stored the data as originally acquired. For example, if the data corresponds to genomic information, the data is originally acquired from an experimental instrument. The data is uploaded directly from the experimental instrument or stored in its raw or unprocessed form. In some embodiments, the data is in an unprocessed state to match a data representation formatted and configured according to the embodiments described herein. When represented, formatted, or otherwise organized in this manner, compression of different blocks of the delimited text file (and / or different data units in one or more data blocks) can be performed efficiently.

[0063]

[0071] In version 520, the data format of a delimited text file is detected. This is achieved, for example, by detecting the file extension of the delimited text file. The file extension or other information indicating the file format is detected, for example, by a compression schema generator or selector or by other management logic.

[0064]

[0072] In 530, a compression schema corresponding to the format of the detected delimited text file is determined or selected. This operation is performed by the compression schema generator / selector 410, for example, either alone or in combination with one or more other features. For example, if there is a predefined schema associated with the file extension of a delimited text file, the compression schema generator retrieves the schema from the schema repository 430, which is preloaded and stored with schemas for use with delimited text files having a corresponding compatible format.

[0065]

[0073] If the format of a delimited text file is a new file format, the user defines and imports a compression schema for that new file format. For example, this is accomplished by a compression schema editor 420, which receives and generates a customized compression schema 425 for the new file format based on user input 415. In one embodiment, the compression schema editor 420 is a compression schema creation tool that assists the user in defining a new schema using support functions, and includes, for example, (i) automatic generation of a compression schema through parsing of delimited text, and (ii) a user interface for customizing the schema with automatic suggestions for compression methods and parameters. The customized compression schema is then stored in a schema repository, associated with one or more file extensions for future use.

[0066]

[0074] In one embodiment, the compressed format generated by the delimited text file format and / or compression scheme includes embedded code with appropriate security protection (e.g., a compressor executable within the file format itself). This code is used by the same or different entities to decompress at least a selected portion of the compressed file corresponding to the embedded code. The embedded code is included independently of the compressor or the content of the compressed data, but may be particularly useful for content compressed using a customized compression algorithm. This code is also used to compress the data as needed.

[0067]

[0075] In 540, the schema interpreter 440 interprets the compression schema determined to correspond to the detected format of the delimited text file. The schema is interpreted in various ways. For example, the interpretation of the compression schema includes updating global parameters in runtime memory with values ​​defined in the schema. These new values ​​are used only in subsequent instructions. In some embodiments, a compression instruction is active only when parsing of the delimited text (e.g., line by line from top to bottom, and for each line, column by column from left to right) enters the rectangular region associated with the instruction. For each active instruction, a buffer is created to hold a vector or matrix of values ​​extracted from the associated region, and the compressor is configured according to a defined algorithm and parameters.

[0068]

[0076] In version 550, the delimited text file is divided into multiple blocks 4551-455 to match the schema interpreted by the schema interpreter. N The data is parsed to extract the necessary data. The blocks are divided into data units of the same size, or at least some of which are of different sizes. The different sizes are determined randomly or according to a corresponding schema. The parsing operation is performed in various ways by the parser and the data extraction logic. For example, a delimited text file is parsed line by line to generate multiple corresponding blocks. This is done, for example, by dividing each line of the delimited text file into tokens using a delimiter, and then allocating each token to a block buffer according to its line number and column index. The tokens in each buffer are then collected into data units of a predefined size for compression. In another embodiment, a delimited text file is parsed into two-dimensional blocks. Once the blocks are generated, they are fed into a compression manager.

[0069]

[0077] In 560, the compression manager 460 compresses the block using one or more compression techniques. For example, the compression manager uses multiple compressors 4651-465 N This includes, however, N≧1 here. Compressor 4651~465 N Each of these implements a different compression algorithm to compress one or more of the blocks generated by the block extraction logic. The compressor / algorithm used to compress each block is determined based on information corresponding to the interpretation of the applicable schema output from the schema interpreter. In some embodiments, the compression of blocks by different compressors is performed in parallel to achieve improved efficiency and performance. Figure 4 illustrates a one-to-one correspondence between blocks to be purged and compressors, but in some embodiments, any one or more of the compressors may compress multiple blocks.

[0070]

[0078] In 570, the compressed blocks are 4681, 4682, ..., 468 N However, these are stored in the respective storage areas of the archive. In one embodiment, the compressed blocks are stored as individual file portions, along with a master index table that identifies one of each compressed block to support random data access. One or more storage devices constitute the storage. For example, the storage devices are one or more buffers, database locations, memory, caches, or other types of data storage.

[0071]

[0079] Various types of information are stored with or in relation to the compressed blocks. This information includes, for example, a compression schema 470 used for parsing delimited text files, and / or metadata 475 that describes each of the compressed blocks or is otherwise linked. Examples of metadata include table row and column names, a specific compression algorithm automatically selected for the data block (not specified in the schema), and delimiter symbols for each block (when multiple delimiter symbols are used). To facilitate fast, random access to specific lines and columns and queries based on specific conditions, the metadata also includes index information. Executable files for any specific compression and decompression algorithms required for any data block are also stored, along with their IDs and method signatures, to improve the portability and accessibility of the compressed files.

[0072]

[0080] Furthermore, or alternatively, information identifying the specific type of compression algorithm used by the compressor to compress each block is stored in a table that links the corresponding block or the type of compression algorithm used for each compressed block.

[0073]

[0081] In 570, all generated file components, including compressed blocks, metadata, and any special compressors and decompressors, are organized and packaged into archive 490 via file manager 485.

[0074]

[0082] In another embodiment, instead of storing compressed data units, schemas, and metadata as parts of a file in the archive, these various components are further organized and stored in a compact file format, as described in the relevant U.S. Patent Application (PHI3170).

[0075]

[0083] The systems and methods described above include several additional features. For example, the system includes a compressor / decompressor repository 492 that stores the actual algorithms for each of the compression and decompression techniques used with the definitions used in the schema instructions. In one embodiment, all or some of these algorithms are stored in the repository 492 in an encrypted form. In addition, the encrypted algorithms are stored in 494 in association with a digital signature that verifies the encryption. The digital signature may or may not be stored with a digital certificate that authorizes the use of the schema in the system.

[0076]

[0084] In some cases, one or more blocks of comments / blocklines, or rows not covered by the area presented by the compression scheme, are extracted and collected into blocks, along with the line count in the original text being recorded. In these cases, a predetermined type of text compression is then applied to the compressed blocks, which are stored as separate file portions.

[0077] Data decompression and file recovery

[0085] Figure 6 illustrates one embodiment of a system for decompressing a compressed portion of a delimited text file and then restoring the decompressed portion to a delimited text file. Figure 7 illustrates one embodiment of a method for performing the decompression and file restoration operations, which is carried out, for example, using the system in Figure 6.

[0078]

[0086] Referring to Figures 6 and 7, this method involves retrieving the compressed file 605 (for example, in DTC format) in 710 and loading it into the system's file manager 610. The file manager 610 is either the same file manager used during compression or a different file manager. The compressed file is retrieved from storage, which is either an archive or another type of storage, as previously shown. The compressed file is retrieved in response to a request from an application or system that uses compressed data (for example, genetic data) for research or other purposes. This request is received from a local processor contained in or connected to the processor, or from a network. In the latter case, the archive or storage is, for example, a server, cloud storage, or other repository connected to the file manager via a network.

[0079]

[0087] In 720, information 620 corresponding to the compression schema and metadata is extracted by the file manager from the compressed file (or retrieved from a table stored for the compressed file). This information itself is compressed using a predetermined compression algorithm known to the file manager. When the information corresponding to the compression schema and metadata are stored in an encrypted and compressed format, the file manager decrypts and decompresses the compression schema information and metadata using a decompressor that reverses the compression performed by the known compression algorithm. As previously shown, in some embodiments, the compression schema information and metadata not only instruct, for example, a compression command for compressing blocks of a delimited text file, but also, in some cases, one or more delimiter symbols used for block and / or index information.

[0080]

[0088] In 730, information about the decompression algorithm applied to different data blocks is extracted from the file's compression schema. Based on this information, the decompression algorithm code is then retrieved (e.g., authenticated, decrypted, and / or decompressed) in 665 from the compressor / decompressor and / or in 610 from a special compressor / decompressor embedded module.

[0081]

[0089] In the 740, the decompression manager loads the code for each of those algorithms, sets one of the decompression parameters, and allocates computing resources and runtime storage to recover the portion of the original delimited data file, thereby processing multiple decompressors 6551-655. N This creates (instantizes) an instance of the decompressor. The diagram illustrates that the number of decompressors and the number of compressed blocks are the same, but this may not be the case in some embodiments. For example, when two or more blocks are compressed by the same algorithm, each decompressor may decompress two or more compressed blocks.

[0082]

[0090] The decompression manager 650, based on the information received by the schema interpreter 660, adjusts the decompressor instances to decompress the blocks using different corresponding algorithms, but the schema interpreter 660 may be the same schema interpreter used during the decompression phase of this method, or it may not be the same schema interpreter. The schema interpreter reads and executes decompression instructions based on schema information and metadata, and retrieves the code for the decompression algorithm to be applied to the compressed data blocks. The schema interpreter then sends the corresponding information to the decompression manager, which decompresses the compressed blocks according to the instructions from the schema interpreter. For example, the decompression of each file portion is performed by one of the decompressors (compatible with the compression algorithm used) that are instantiated based on the algorithm and parameters specified in the compression schema. To speed up the decompression process, the decompression of individual file portions or even individual data units may be performed in parallel.

[0083]

[0091] In one embodiment, once a specific decompression algorithm and its corresponding parameters are determined from a compression schema obtained through a file manager, a schema interpreter retrieves the code corresponding to the appropriate decompression algorithm from repository 665 or an embedded module 630 and sends the code and its associated parameters to a decompressor manager to instantiate a decompressor.

[0084]

[0092] In 750, the compressed blocks 6401-640 N However, it is extracted from the bundled file by the file manager. As mentioned above, N is 1 or greater, and the blocks are compressed based on different compression algorithms.

[0085]

[0093] In 760, the compressed block is input to the decompression manager 650. Once the decompressor is instantiated and configured using code from the compressor / decompressor repository and / or embedded module, decompressors 6551-655 N However, in order to recover blocks of an uncompressed delimited text file, the compressed blocks are decompressed. The blocks are stored in their respective buffers for use, for example, by file recovery logic.

[0086]

[0094] In 770, the file recovery manager 680 now recovers the uncompressed blocks 6701-670 to form the original delimited text file 690 that has now been recovered. N The file recovery manager determines how to reassemble the uncompressed blocks in order to recover the delimited text file based on the compressed schema and metadata and other information determined by the schema interpreter. This involves reassembling lines, columns, blocks, or other parts of blocks in order to restore the delimited text file in the original format as it existed before decompression and compression. In one embodiment, the restoration of the original file is performed line by line by extracting data elements from a buffer and assembling them by inserting the correct delimiter symbols according to the compressed schema and metadata.

[0087]

[0095] The selective compression and decompression performed by the embodiments described herein allows one or more blocks in a portion of a compressed delimited text file to be retrieved, decompressed, and restored without retrieving, decompressing, and restoring blocks in other portions of the file. For example, a specific region containing information of interest to the user (e.g., one or more rows and / or one or more columns) can be retrieved from the compressed data without retrieving and / or decompressing other portions of the compressed delimited text file. Thus, only the data of the delimited file having the portions of interest can be retrieved and used independently of the rest of the file. This allows only the portions of the delimited text file that are of interest to be selectively decompressed and accessed, which is beneficial for supporting fast queries and random access.

[0088]

[0096] Figure 8 illustrates an embodiment of a method for selectively accessing one or more blocks in a compressed delimited text file, independently of accessing other parts of the file (e.g., decompressing, breaking down, etc.).

[0089]

[0097] Referring to Figure 8, this method includes receiving information in 810 that points to one or more regions of a compressed delimited text file of interest. These one or more regions of interest correspond, for example, to a portion of a gene data file. This information is received, for example, by extracting instructions from the compression schema associated with the regions of interest. In one embodiment, this region information includes a table / block identifier (ID) defined in the compression schema, which identifies one or more portions of the compressed delimited text file of interest.

[0090]

[0098] In 820, compressed data blocks (e.g., file portions) associated with (one or more) regions of interest are identified based on instructions extracted from the compressed schema. This operation is performed, for example, by a schema interpreter.

[0091]

[0099] In 830, for each data block identified in operation 820, one or more data units associated with one or more regions of interest are identified.

[0092]

[0100] For operations 820 or 830, or both, the location of one or more portions (e.g., data blocks, data units) of a compressed delimited text file is determined according to location information stored in a table accessed by the file manager. This is achieved, for example, in the following manner: First, the start and end line numbers of the one or more file portions of interest are mapped to the corresponding block index and offset line numbers in the block. This is achieved, for example, based on equations (1) and (2). Data_Unit_Index=Floor((Line_Number-Data_Block_Loc) / Data_Unit_Size)+1 (1) Data_Unit_Offset=Line_Number-(Data_Unit_Index-1)*Data_Unit_Size (2)

[0093]

[0101] In these equations, Data_Block_Loc is the block location, which is, for example, the starting line number of the block in the original text, and Data_Unit_Size is the number of lines by data unit. Both elements are indicated by the information contained in the compressed schema. If the Row_Index of the table is used instead of Line_Number, then Data_Block_Loc is instead the index of the first row of the block in the table.

[0094]

[0102] To execute a query based on column values, the columns included in the query conditions are unpacked. Alternatively, the query may be executed against a search tree generated based on the column values, and stored as metadata components associated with those columns. The line numbers of matching rows are then calculated, and equations (1) and (2) are used to determine the (one or more) blocks and (one or more) offsets of the corresponding data units.

[0095]

[0103] For all relevant blocks, the block indicated in operation 840 is identified, and the relevant rows within that (one or more) block are extracted using the calculated line offset.

[0096]

[0104] In 840, the data decompression manager instantiates and configures decompressors using one or more algorithms and parameters identified for the region of interest and associated data blocks. This includes configuring one of the decompressors, or, if not, selecting a decompressor that is already configured using the corresponding decompression algorithm.

[0097]

[0105] In 850, data units in one or more data blocks associated with one or more regions of interest are decompressed by their corresponding components in the decompressor.

[0098]

[0106] In 860, once decompression occurs, the decompressed (one or more) blocks are assembled in a selected region according to the format defined in the compression schema. In one embodiment, the user specifies the output format of the extracted data units (by information in user input) by identifying a restoration schema that describes how the blocks should be organized using semantics similar to those of the compression schema. The decompressed (one or more) blocks of interest are then output in an assembled format, for example, on a display, without decompressing any blocks that are not of interest in the compressed delimited text file. In one embodiment, the region of interest where the decompressed (one or more) blocks of interest are displayed corresponds, for example, to a specific portion of the data in the whole genetic information corresponding to a particular subject or sample of interest.

[0099]

[0107] According to one embodiment, the compression scheme is customized for processing virtual contact files (VCFs) and BED files using the proposed CDTC framework. The following example illustrates how the compression scheme can be defined for each VCF and BED file format.

[0100] [Table 1]

[0101]

[0108] Referring to the example VCF file in Table 1, the following compression scheme is applied to the following code as a possible (but not necessarily optimal) approach. Delimiters = '\t' Comment_Symbol = '##' <blocks data_unit_size="5"> <block region_cols="1" data_type="‘uint’" comp_alg="‘RLE’" block_name="‘Chromosome’"> < / block> <block region_cols="2" data_type="‘uint’" comp_alg="‘Delta’" block_name="‘Position’"> < / block> <block region_cols="3" data_type="‘string’" comp_alg="‘Auto’" block_name="‘ID’"> < / block> <block region_cols="4" data_type="‘char’" comp_alg="‘Enum’" block_name="‘Ref’"> < / block> <block region_cols="5" data_type="‘char’" comp_alg="‘Enum’" block_name="‘Alt’"> < / block> <block region_cols="6" data_type="‘uint(8)’" comp_alg="‘Auto’" block_name="‘Quality’"> < / block> <block region_cols="7" data_type="‘string’" comp_alg="‘Enum’" block_name="‘Filter’"> < / block> <block region_cols="8" data_type="‘string’" comp_alg="‘VCF_Info($Comments)’" block_name="‘Info’"> <block region_cols="9" data_type="‘string’" comp_alg="‘Enum’" block_name="‘Format’"> <block region_cols="10:12" data_type="‘string’" comp_alg="‘VCF_Sample($Format)’"> < / block> < / block> < / block> < / blocks>

[0102]

[0109] In this example, it should be noted that two special compression algorithms, "VCF_Info" and "VCF_Sample," are designed to process the Info and Sample data (NA00001, NA00002, NA00003). For the VCF_Info method, the argument $Comments indicates that the information in the comment block should be used to identify all variant attributes. The corresponding attribute values ​​in the Info column are then stored as a matrix according to the attributes that are extracted and compressed separately. For the VCF_Sample method, the input argument $Format indicates that the attributes in the Format column (GT, GQ, DP, HQ) should be used to split the data elements and organize them into separate matrices for effective compression of individual attributes.

[0103] [Table 2]

[0104]

[0110] Referring to the example BED file in Table 2, the following compression scheme is applied using the following code as a possible (but not necessarily optimal) approach. Delimiters = '\t' <blocks data_unit_size="5"> <block region_cols="1" data_type="‘string’" comp_alg="‘Enum" + rle’ block_name="‘Chromosome’"> < / block> <block region_cols="2" data_type="‘uint’" comp_alg="‘Delta’" block_name="‘Chr_Start’"> < / block> <block region_cols="3" data_type="‘uint’" comp_alg="‘Delta’" block_name="‘Chr_End’"> < / block> <block region_cols="4" data_type="‘string’" comp_alg="‘Auto’" block_name="‘Name’"> < / block> <block region_cols="5" data_type="‘uint(10)’" comp_alg="‘Delta’" block_name="‘Score’"> < / block> <block region_cols="6" data_type="‘char(1)’" comp_alg="‘Enum" + rle’ block_name="‘Strand’"> < / block> <block region_cols="7" data_type="‘uint’" comp_alg="‘Delta’" block_name="‘Thick_Start’"> < / block> <block region_cols="8" data_type="‘uint’" comp_alg="‘Delta’" block_name="‘Thick_End’"> <block region_cols="9" data_type="‘fstring(‘%uint(8)" , %uint(8), %uint(8)’)’ comp_alg="‘RLE’" block_name="‘Item_RGB’"> < / block> < / block> < / blocks>

[0105]

[0111] Figure 9 illustrates an example of a processing system used to perform the operation of embodiments of the systems and methods described herein. This processing system includes at least one processor 910, memory 920, storage 930, a communication interface 940, and an output device 950.

[0106]

[0112] At least one processor 910 performs the manager, selector, interpreter, and parser operations described herein, as well as other information generation and processing operations. In some embodiments, the processor 910 may have multiple cores, each dedicated to performing a different compression and / or decompression algorithm. In other embodiments, multiple processors may be included to perform different predetermined operations, including different compression / decompression algorithms, and / or various other operations, including parsing, schema generation, schema interpretation, and other operations associated with the embodiment. In some embodiments, the same processor may perform all compression and decompression. In doing so, at least one processor 910 performs file construction and restoration, generates tables, data structures, and schemas, and also performs generation and editing operations, such as interpreting schemas and enabling users to generate customized schemas.

[0107]

[0113] Memory 920 stores instructions for causing at least one processor 910 to perform the operations of the system and method embodiments. This memory is one or a combination of non-temporary (one or more) computer-readable media locally connected to at least one processor. In some embodiments, the processor and memory are located in a workstation used in a research facility, laboratory, or other location where information from a delimited text file is used in relation to one or more intended applications. This is the case for the contents of a delimited text file storing genetic data.

[0108]

[0114] Storage 930 is a database, repository, archive, or other storage for storing delimited text files in their original form, compressed form, or both. Similar to memory, storage is one or more non-temporary computer-readable media (one or more) locally connected to at least one processor. In some embodiments, storage may also be remotely connected to one or more processors via a network connection. This may be the case, for example, when storage 930 is part of a storage network, cloud computing network, or other processing and / or data storage architecture.

[0109]

[0115] The communication interface (I / F) 940 receives raw data, which is then processed by at least one processor 910 to form a delimited text file. This processing includes converting the data to a text file format using delimiters and other symbols and information described in relation to the compression scheme discussed herein. The interface 940 may also receive requests arising in relation to embodiments, and similarly, requests from other entities that may be related to viewing or using the delimited text file.

[0110]

[0116] The output device 950 may be a display that produces all or selected portions of the delimited text file stored and / or processed as described herein. This is particularly useful when only the region of interest is output for analysis, such that only the (one or more) blocks of interest in the compressed and delimited text file stored in storage 930 are decompressed for output, while other blocks unrelated to the region of interest are not decompressed.

[0111]

[0117] The methods, processes, and / or operations described herein are performed by code or instructions executed by a computer, processor, controller, or other signal processing device. This code or instruction is stored in a non-temporary computer-readable medium, according to one or more embodiments. Since the algorithms forming the basis of the methods (or the operations of the computer, processor, controller, or other signal processing device) are described in detail, the code or instructions for implementing the operations of embodiments of the methods cause the computer, processor, controller, or other signal processing device to be converted into a dedicated processor for performing the methods described herein.

[0112]

[0118] The processors, interpreters, generators, parsers, extractors, editors, compressors, decompressors, managers, restorers, disassemblers, selectors, and other information generation, processing, and computation features of the embodiments disclosed herein are implemented, for example, in logic including hardware, software, or both. When implemented at least partially in hardware, these processors, interpreters, generators, parsers, extractors, editors, compressors, decompressors, managers, restorers, disassemblers, selectors, and other information generation, processing, and computation features may be, for example, one of a variety of integrated circuits including, but not limited to, application-specific integrated circuits, field-programmable gate arrays, combinations of logic gates, systems on a chip, microprocessors, or other types of processing or control circuits.

[0113]

[0119] When implemented at least partially as software, the processor, interpreter, generator, parser, extractor, editor, compressor, decompressor, manager, restorer, disassembler, selector, and other information generation, processing, and computation features include, for example, memory or other storage devices for storing code or instructions executed by a computer, processor, microprocessor, controller, or other signal processing device. Since the algorithms forming the basis of the method (or the operation of a computer, processor, microprocessor, controller, or other signal processing device) are described in detail, the code or instructions for implementing the operation of embodiments of the method cause the computer, processor, controller, or other signal processing device to be converted into a dedicated processor for performing the method herein.

[0114]

[0120] It should be apparent from the above description that various exemplary embodiments of the present invention can be implemented as hardware or firmware. Furthermore, various exemplary embodiments can be implemented as instructions stored on a machine-readable storage medium, which are read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium includes any mechanism for storing information in a format readable by a machine, such as a personal or computer computer, server, or other computing device. Thus, machine-readable storage media include read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and similar storage media.

[0115]

[0121] Those skilled in the art will understand that every block diagram in this specification represents a conceptual diagram of an exemplary circuit configuration embodying the principles of the present invention. Similarly, it will be understood that every flowchart, flow diagram, state transition diagram, pseudocode, etc., substantially represents various processes performed by a computer or processor, regardless of whether such computer or processor is explicitly indicated, in a machine-readable medium.

[0116]

[0122] While various exemplary embodiments have been described in detail above, with particular reference to certain exemplary aspects, it should be understood that other embodiments are also possible, and that the details are modifiable in various obvious respects. As will be readily apparent to those skilled in the art, changes and modifications are possible while remaining within the spirit and scope of the invention. Therefore, the above disclosures, descriptions, and drawings are for illustrative purposes only and do not limit the invention in any sense, and the invention is defined solely by the claims.

Claims

1. A step of obtaining a compressed schema which includes a table defining multiple ranges of line numbers and multiple ranges of column numbers for the components in the delimited text file, enabling the user to customize how the data in the input delimited text file is divided into different components consisting of lines and columns. The steps of parsing the data in the delimited text file into multiple data blocks corresponding to the different components based on the table of the compression schema, The steps include compressing the data units in the aforementioned plurality of data blocks using different compression algorithms, It has, A method for compressing genome data, wherein the different compression algorithms are instructed by compression commands in the compression schema.

2. The step of obtaining the aforementioned compression scheme is, The method according to claim 1, comprising the steps of creating a new compression scheme or determining the best match from a plurality of compression schemes based on information entered by a user, wherein each of the plurality of compression schemes is customized for each of a plurality of different delimited text files.

3. The method according to claim 1, further comprising the step of creating the compression schema customized to the format of the delimited text file based on a tool having a graphical user interface, wherein the graphical user interface includes a predetermined window that allows input of information to customize the compression schema to the format of the delimited text file.

4. The method according to claim 1, further comprising the steps of generating a compressed file comprising compressed data units in a plurality of data blocks, and a compression schema including instructions for decompressing the compressed data units and restoring the compressed file.

5. The method according to claim 4, wherein the compressed file includes metadata information for decompression, file recovery, and extended functionality.

6. The method according to claim 5, wherein the extended functionality includes data security and search queries.

7. The method according to claim 4, wherein the compressed file includes a definition of the code and usage of a special compression / decompression algorithm.

8. The method according to claim 1, wherein the compression instruction specifies different compression algorithms and their corresponding parameters to be used to compress different parts of the data unit based on different contents of the data block.

9. The method according to claim 2, wherein the step of determining the compression schema includes the step of selecting the compression schema from a plurality of compression schemas stored in a schema repository.

10. A step of receiving information indicating a region of interest in the data, wherein the region of interest corresponds to one or more data units contained in at least one data block in a compressed delimited text file, A step of selectively decompressing one or more data units of at least one data block associated with the region of interest in the compressed delimited text file, without decompressing other data units in the at least one data block or other data blocks in the compressed delimited text file, wherein the one or more data units are selectively decompressed based on one or more decompression algorithms indicated by a compression instruction in the compression schema, the compression schema includes a table defining multiple ranges of line numbers and multiple ranges of column numbers of the components in the delimited text file, which allows a user to customize how the data in the delimited text file is divided into different components consisting of lines and columns, and the data units to be decompressed are selected based on the table; The steps include restoring the region of interest from one or more selectively decompressed first data units, The steps include outputting information indicating the restored region of interest, A method for selective data access having [a certain characteristic].

11. The method according to claim 10, further comprising the step of selectively accessing one or more data units based on a query of the compressed delimited text file, wherein the query is performed based on one or more conditions or ranges of values ​​found in the selectively decompressed data units.

12. The method according to claim 10, wherein the region of interest may correspond to a selected range of genomic coordinates or gene IDs.

13. A schema manager that enables a user to create, select, or automatically generate a compressed schema customized to the format of an input delimited text file, the schema manager including a table that defines multiple ranges of line numbers and multiple ranges of column numbers for the components in the delimited text file, enabling the user to customize how the compressed schema is divided into different components consisting of lines and columns, A parser that parses the data of the input delimited text file into multiple data blocks corresponding to different components based on the table of the compression schema, A compression manager that compresses multiple data units in the multiple data blocks using different compression algorithms instructed by compression commands in the compression schema, A system for compressing data, equipped with the necessary features.

14. The system according to claim 13, wherein the schema manager creates a new compression schema or determines the best match from a plurality of compression schemas based on information entered by the user, and each of the plurality of compression schemas is customized for each of a plurality of delimited text files of different formats.

15. The system according to claim 13, wherein the compression manager instantiates a compressor for each data block by obtaining the code for the compression algorithm from a user or a compressor repository, allocating computing resources and memory, and operating and monitoring the compression of the data unit.

Images