System and method for effective compression, representation and decompression of diverse tabulated data

Abstract

To provide a system and a method for effective compression, representation and decompression of diverse tabulated data.SOLUTION: A method for controlling compression of data includes the steps of: accessing genomic annotation data of one first file format out of a plurality of first file formats; extracting a plurality of attributes from the genomic annotation data; dividing the genomic annotation data into a plurality of chunks; processing the extracted attributes and chunks into correlated information; selecting different compressors for the attributes and chunks to be identified in the information; and generating a file of a second file format, that includes the correlated information, and information indicating the different compressors for the attributes and chunks indicated in the correlated information, and allowing selective development of the attributes and chunks indicated in the correlated information.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001]

[0001] This application is related to U.S. Provisional Patent Application No. 62 / 923,141, filed October 18, 2019, the entire contents of which are incorporated by reference into this specification for all purposes.

[0002]

[0002] This application is related to U.S. Provisional Patent Application No. 62 / 923,113, filed October 18, 2019, the entire contents of which are incorporated by reference into this specification for all purposes.

[0003]

[0003] This application is related to U.S. Provisional Patent Application No. 62 / 956,941 (Attorney Docket No. 2019P00831US01), "Customizable Delimited Text Compression Framework," filed concurrently herewith, the entire contents of which are incorporated by reference into this specification for all purposes. [Technical field]

[0004]

[0004] One or more embodiments herein relate to managing the compression and decompression of information such as, but not limited to, tabular data and delimited text files. [Background technology]

[0005]

[0005] Genomic data is known to be important for many research applications. Genomic data in raw format may contain a large number of sequenced genomes. In some cases, the number of sequences is unmanageably large. Therefore, various attempts have been made to extract relevant information from the raw data or to better interpret it.

[0006]

[0006] One processing technique involves the use of annotation tools. Such tools can be used to define coding regions and their corresponding locations within the genomic data. Annotation can also provide an indication of the number and spatial distribution of repetitive regions that may convey information about disease or genetic abnormalities. Examples of genomic annotation data include mapping statistics, quantitative browser tracks, variants, genomic functional annotations, gene expression data, and Hi-C contact matrices.

[0007]

[0007] Annotation data (and other genome-related information) are currently represented in a variety of file formats, such as the variant call format (VCF), browser extensible data (BED) format, and wiggle (WIG) format. These formats are not compatible with each other. As a result, the use of these formats involves problems related to interoperability and requires frequent format conversion to be able to visualize the data. The lack of a single unified and standardized format has also hindered efforts at compression algorithms, leading to the widespread use of suboptimal compression algorithms (e.g., gzip).

[0008]

[0008] Shortcomings associated with existing algorithms for compressing genomic information relate, among other things, to a lack of selectivity. For example, genomic annotation data typically includes multiple fields (attributes) with different statistical properties. Existing algorithms compress all fields together, and therefore cannot selectively recognize, use, and extract information within individual data fields. Existing algorithms also do not allow for the extraction of specific fields without expanding all attributes.

[0009]

[0009] Other shortcomings relate to the lack of general applicability of existing compression methods. For example, these methods rely only on one standard compressor, or a small set of standard compressors, and therefore can only be applied to one type of annotation data. These methods also cannot perform selective encryption, nor can they link multiple annotation data sets with each other and with sequencing data. Several specialized methods have been proposed. However, many of these methods are based on disk-based array management tools (e.g., TileDB and HDF5), which do not have high-level features, such as, but not limited to, metadata, linkage, and attribute-specific indexing. Summary of the Invention

[0010]

[0010] A brief summary of various exemplary embodiments is presented below. In the following summary, some simplifications and omissions may be made, but this is intended to highlight and introduce some aspects of various exemplary embodiments, and is not intended to limit the scope of the present invention. A detailed description of the exemplary embodiments sufficient to enable a person skilled in the art to make and use the embodiments is provided below.

[0011]

[0011] According to one or more embodiments, a method for controlling compression of data includes the steps of accessing genome annotation data in one first file format of a plurality of first file formats, extracting a plurality of attributes from the genome annotation data, splitting the genome annotation data into a plurality of chunks, processing the extracted attributes and chunks into correlated information, selecting different compressors for the attributes and chunks identified in the correlated information, and generating a file in a second file format containing the correlated information and information indicating the different compressors for the chunks and attributes indicated in the correlated information, wherein the plurality of first file formats are incompatible with each other and differ from the second file format, and the information indicating the different compressors is processed into the second file format to enable selective expansion of the attributes and chunks indicated in the correlated information.

[0012]

[0012] The correlated information may include at least one table including first information indicating one or more of the attributes and second information corresponding to chunks associated with one or more attributes indicated in the first information. The first information may include a two-dimensional cell array, each cell identifying one or more corresponding attributes contained within a chunk corresponding to the two-dimensional cell array. The first information may include at least one one-dimensional table of dimension-specific attributes associated with the chunks. The genome annotation data may be divided into chunks of equal size. The genome annotation data may be divided into chunks of different sizes.

[0013]

[0013] The method includes generating one or more first indexes linking the correlated information and generating one or more second indexes linking the correlated information to information indicative of different compressors, and the one or more first indexes and the one or more second indexes are processed into a file in a second file format. The method may include integrating the file in the second file format into an MPEG-G file.

[0014]

[0014] The method includes generating access control policy information for the correlated information and integrating the access control policy information into a file of a second file format, the access control policy information including first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level. The extracted attributes may include at least one of chromosomal attributes and genomic location attributes.

[0015]

[0015] According to one or more embodiments, an apparatus for controlling compression of data comprises a memory configured to store instructions and at least one processor configured to execute the instructions to perform operations, the operations including: accessing genome annotation data in one of a plurality of first file formats, extracting attributes from the genome annotation data, splitting the genome annotation data into chunks, processing the extracted attributes and chunks into correlated information, selecting different compressors for attributes and chunks identified in the correlated information, and generating a file in a second file format including the correlated information and information indicating different compressors for chunks and attributes indicated in the correlated information. The plurality of first file formats are incompatible with each other and are different from the second file format. The information indicating the different compressors is processed into the second file format to enable selective expansion of attributes and chunks indicated in the correlated information.

[0016]

[0016] The correlated information may include at least one table including first information indicating one or more of the attributes and second information corresponding to chunks associated with one or more attributes indicated in the first information. The first information includes a two-dimensional cell array, each cell identifying one or more corresponding attributes included in a chunk corresponding to the two-dimensional cell array. The first information may include at least one one-dimensional table of dimension-specific attributes associated with the chunks. The genome annotation data may be divided into chunks of equal size. The genome annotation data may be divided into chunks of different sizes.

[0017]

[0017] At least one processor is configured to execute instructions to generate one or more first indexes linking the correlated information and generate one or more second indexes linking the correlated information to information indicative of a different compressor, and the one or more first indexes and the one or more second indexes are processed into a file of a second file format. At least one processor is configured to execute instructions to aggregate the file of the second file format into an MPEG-G file.

[0018]

[0018] At least one processor is configured to execute instructions to generate access control policy information for the correlated information and integrate the access control policy information into a file in a second file format, the access control policy information including first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level. The extracted attributes may include at least one of chromosomal attributes and genomic location attributes. [Brief description of the drawings]

[0019]

[0019] In the accompanying drawings, like reference numbers refer to identical or functionally similar elements throughout the individual views. The accompanying drawings, together with the following detailed description, are incorporated into and form a part of the specification. The accompanying drawings serve to further explain exemplary embodiments of the concepts recited in the claims and to explain various principles and advantages of those embodiments.

[0020]

[0020] The above and other more detailed and specific features are more fully disclosed in the following specification, taken in conjunction with the accompanying drawings. [Figure 1]

[0021] FIG. 1 illustrates an embodiment of a system for controlling compression and decompression of genome annotation data. [Diagram 2]

[0022] FIG. 2 shows a method for controlling the compression and decompression of genome annotation data. [Diagram 3]

[0023] FIG. 3 shows an embodiment of a unified and standardized file format for genome annotation data. [Figure 4]

[0024] FIG. 4 illustrates an embodiment for processing information into annotation files for genome annotation data. [Diagram 5]

[0025] FIG. 5 illustrates an embodiment for processing information in a file header of a unified and standardized file. [Figure 6]

[0026] FIG. 6 illustrates an embodiment for processing information for a unified and standardized file. [Figure 7]

[0027] FIG. 7 illustrates an embodiment for processing compressed information for a unified and standardized file. [Figure 8]

[0028] FIG. 8 shows an embodiment for processing information for a one-dimensional array. [Figure 9]

[0029] FIG. 9 illustrates an embodiment for processing information into a two-dimensional array, along with arrays for dimension-specific attributes. [Figure 10]

[0030] FIG. 10 illustrates an embodiment for processing attribute information indexed for genome annotation data. [Figure 11]

[0031] FIG. 11 illustrates an embodiment for processing additional attribute-related information. [Figure 12]

[0032] FIG. 12 illustrates an embodiment for processing attribute information. [Figure 13]

[0033] FIG. 13 illustrates an embodiment for selectively compressing chunks of data. [Figure 14]

[0034] FIG. 14 illustrates an embodiment for processing compressed file index information. [Figure 15]

[0035] FIG. 15 illustrates an embodiment for processing index information. [Figure 16]

[0036] FIG. 16 illustrates an embodiment for processing chunks. [Figure 17]

[0037] FIG. 17 illustrates an embodiment for generating fixed size chunks. [Figure 18]

[0038] FIG. 18 illustrates an embodiment for generating chunks of variable size. [Figure 19]

[0039] FIG. 19 illustrates an embodiment for processing chunks and attribute information. [Figure 20]

[0040] FIG. 20 illustrates an embodiment for processing chunks and attribute information. [Figure 21]

[0041] FIG. 21 illustrates an embodiment for processing a data payload. [Figure 22]

[0042] FIG. 22 illustrates an embodiment for selectively deploying genome annotation data in a unified and standardized format. [Figure 23A-23B]

[0043] 23A and 23B illustrate a method for converting a file format into a folder hierarchy according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021]

[0044] It should be understood that the drawings are merely schematic and are not drawn to scale, and that the same reference numerals are used throughout the different drawings to denote the same or similar parts.

[0022]

[0045] The description and drawings illustrate the principles of various exemplary embodiments. Thus, it will be understood that those skilled in the art can conceive of various configurations that embody the principles of the present invention and fall within the scope of the present invention, even though not explicitly described or illustrated herein. Moreover, all examples described herein are expressly intended to be primarily for instructional purposes to aid the reader in understanding the principles of the present invention and the concepts the inventors provide for the advancement of the art, and should be construed as not being limited to such specifically described examples and conditions. Also, the terms "or" and "or" as used herein refer to non-exclusive disjunctions (i.e., and / or) unless otherwise indicated (e.g., "otherwise" or "or instead"). Also, various exemplary embodiments are not necessarily mutually exclusive, and some exemplary embodiments may be combined with one or more other exemplary embodiments to form new exemplary embodiments. Descriptive terms such as "first", "second", "third", and the like are not intended to limit the order of the elements discussed, but are used to distinguish one element from the next, and are generally interchangeable. Values ​​such as maximum or minimum values ​​may be predetermined and set to different values ​​based on the application.

[0023]

[0046] Exemplary embodiments describe systems and methods for managing compression of information, such as, but not limited to, tabular data and delimited text files. The information to be compressed may include genomic information (e.g., annotation, sequencing, or other forms of genome-related data). In other embodiments, other types of information unrelated to genomic information may be managed for compression.

[0024]

[0047] In genomic applications, the systems and methods may manage the standardization of genomic data compression and decompression. This includes processing the data to conform to formats that reduce or eliminate issues related to interoperability and the need for frequent format conversion to make the data visible. In at least some applications, these features may result in improved and optimized compression algorithms for specific types of data.

[0025]

[0048] According to one or more embodiments, the systems and methods may process data to enable selective compression of different fields, attributes, and / or sections of the data. For example, in some implementations, all of the fields, attributes, or sections of the data (including, e.g., those having different statistical characteristics) may not all be compressed using the same compression algorithm. Instead, the systems and methods may be implemented to selectively compress one or more sections of the data and not compress other sections, or to selectively compress different sections using different compression techniques. The compression techniques selected may be beneficial or optimal for different types of data within corresponding respective ones of the multiple sections, for example.

[0026]

[0049] This selective compression technique may, in turn, enable selective deployment. For example, by selectively compressing data, the entire data file need not be deployed. Instead, in at least one implementation, one or more sections of data of interest may be selectively deployed without deploying other sections of compressed data that are not of interest. This improves efficiency and the ability to avoid burdens and other delays during a study.

[0027]

[0050] The processing performed by the systems and methods may also enable at least one embodiment to be generally applicable to all types of data, e.g., using multiple standard or customized compression algorithms selectively applied to different data sections. The systems and methods may also perform selective encryption of data sections that have been compressed (e.g., in different ways). The systems and methods may also be implemented to link multiple annotation data sets with each other and with sequencing data. The systems and methods may also process data to include metadata, linkages, and / or attribute-specific indexing.

[0028]

[0051] 1 illustrates an embodiment of a system for managing compression of information. The system may be implemented, at least in part, in, for example, a laboratory workstation, a research facility, a data center, a university or corporate environment, or another location where data is processed in connection with an intended application. In another embodiment, the system may be implemented on one computer and / or other device, or multiple connected or networked computers and / or other devices that communicate with each other to perform the processing operations described herein.

[0029]

[0052] With reference to FIG. 1, the system includes a processor 110, a memory 120, and a storage area 130. The processor 110 executes instructions stored in the memory 120 to perform operations including managing compression of information. The memory 120 may be any form of non-transitory computer-readable medium that stores instructions for controlling the processor to perform operations described herein. The information may be received from a data source 140. The data source may vary, for example, by the type of information to be processed. For example, in one implementation, the data source 140 may be a software-based tool for generating genome annotation data. In another implementation, the data source may provide another type of genome-related data, or in some cases data that is not related to genome data.

[0030]

[0053] The processor 110 may be connected to the data sources locally or over a network. In the latter case, the information that the system processes may be obtained from anywhere in the world, provided that a network connection can be established between the processor and the data source. The network connection may be a local connection, an Internet connection, a virtual private network connection, a connection to a cloud computing or cloud-based storage device, and / or another type of connection.

[0031]

[0054] The storage area 130 may store the processed results of the processor 110. As described in more detail below, the processed results may include tabular data or delimited text files conforming to a unified and standardized file format that includes multiple fields, attributes, sections, or other portions that may be selectively compressed and / or otherwise manipulated. The storage area may be or be included in a database, an archive, a storage area network, a cloud-based storage system, memory included or coupled to the processor's computer, system, or device, or another type of storage area.

[0032]

[0055] 1 may include or be coupled to one or more additional features for compressing the processed information into a unified and standardized file format. These features may include a parser 150, a compression manager 160, and an aggregator 170.

[0033]

[0056] Parser 150 may parse files that have been stored and later retrieved from storage area 130. For example, processor 110 may receive a request for data based on user input or from another device. The processor may, for example, determine one or more files stored in storage area 130 that correspond to the requested data and retrieve the determined one or more files. Processor 110 may determine which files correspond to the requested data using, for example, an index that links the contents of the files to one or more identifiers specified in the request. Once the files are found, they are retrieved from storage area 130 and sent to the parser.

[0034]

[0057] The parser may parse the file in various ways. For example, the file may contain delimiters that separate different sections, parts, attributes, fields, etc. that contain different pieces of data (or other types of content) in the file. Using these delimiters as a guide, the parser 150 may parse the file into individual sections (or chunks) that correspond to requests. In one embodiment, the parser may perform these operations according to a pre-defined compression scheme that indicates how the file is to be parsed for compression.

[0035]

[0058] The compression manager 160 receives files in a unified standard format when the files are retrieved from the storage area 130. The compression manager comprises multiple compressors 1611, 1612, ..., 161 Nwhere N≧2. Each compressor may perform a different compression algorithm to compress different sections of the parsed file. The compression algorithm applied by each compressor may be predetermined based on the parsed data or content of the file. For example, one compression algorithm may be more suitable for compressing certain types of information, from an efficiency standpoint or for other reasons. Other compression algorithms may be more suitable for compressing other types of information. The compression scheme may control which compression algorithm is used to compress certain data or content in the file. (The parser may also route different parsed sections of the file to specific compressors based on instructions derived from the compression scheme.) Thus, in this manner, the compression manager 160 can selectively compress different parsed portions of the file.

[0036]

[0059] Aggregator 170 aggregates or combines the different compressed sections into a compressed file according to a unified and standardized file format for the compressed file or according to different formats. After compression, the compressed file (e.g., a compressed tabular data file or a compressed delimited text file) may optionally be stored in storage area 130, stored in a different storage area, sent to a predetermined location or device, or routed according to another scheme, such as a scheme determined by predetermined control instructions for the system.

[0037] Uniform / standardized data processing

[0060] According to one or more embodiments, the processor 110 processes information from the data sources to conform to a unified and / or standardized file format that allows for selective compression (and therefore subsequent selective decompression) of different portions of the file. The processing may be performed to allow for storage of the genome annotation data in a file format that allows for support of one or more predetermined features. Examples of these features include fast queries, random access, multiple resolutions (e.g., zoom), selective encryption, authentication, access control, and traceability, etc.

[0038]

[0061] In one embodiment, the genomic application data is processed into a format that allows for significant compression gains. This may be accomplished, for example, by separating different attributes of the data into multiple sections (e.g., multiple tables). Multiple compressors may then be used to compress each of the separated attributes in the genomic annotation data. In one embodiment, all or some of the compressors may be specialized or customized compressors designed to compress specific attributes of the files assigned to the compressor. The processing performed by the processor 110 may also generate metadata and linkage for the sequencing data associated with the annotations, as well as linkage to other annotation data that may or may not be associated with the sequencing data. The metadata and / or linkage may be stored in the same unified and standardized format as the genomic application data, allowing seamless integration with existing file formats for sequencing data (e.g., MPEG-G).

[0039]

[0062] 2 illustrates a method for managing compression of information according to a unified and standardized file format. The method may be performed, for example, by the system of FIG. 1 or another system according to instructions stored in a memory for controlling one or more processors to perform the operations described below. For purposes of explanation, it is assumed that the method is performed in part or in whole by the system of FIG. 1 and that the data is genome annotation data.

[0040]

[0063] 2, the method includes receiving data (e.g., genome annotation data) from a data source 140 at 210. The genome annotation data may be received in an existing format, for example, generated by an annotation tool used to generate the data. One example of an existing format is MPEG-G format, although in another embodiment, the annotation data may be received in a different existing format. The data may be received by the processor 110 from the data source via a local or network connection.

[0041]

[0064] At 220, portions of the genome annotation data are retrieved for processing into a unified and standardized file format. Because the genome annotation data is organized according to an existing known format, the content retrieved in each portion of the known format has a particular type of data known to the processor 110. Thus, instructions controlling the processor 110 can retrieve the particular type of data in each portion of the known format to generate a new file in a unified and standardized format according to one or more embodiments.

[0042]

[0065] At 230, the processor 110 executes instructions in the memory 120 to process the genome annotation data to conform to a unified standard file format 300, for example as shown in Figure 3. For example, the processor may process the data to generate a file header section 310, a file protection section 320, a file metadata section 330, a file traceability section 340, a table information and index section 350, a section 360 indicating one or more compression parameters, and one or more tables 370. The chart in Figure 4 illustrates these sections and corresponding descriptions, along with other information that will be described in more detail below.

[0043]

[0066] The file header section 310 includes various types of identifying information. For example, the file header section may include a file name field 311, a file type field 312, and a file version field 313 (see, e.g., FIG. 5). The file name field may include a string that gives the file a name generated, for example, by an annotation tool or based on information from a user. The file type field may include a string that indicates the type of file (e.g., variant, gene expression, etc.). The file version field may include a string that identifies the purpose and / or version of the file (e.g., this file is for recording updates).

[0044]

[0067] The file protection section 320 includes information related to the access control policy of the file, which may be determined, for example, by a user, an administrator, a data owner, or other entity that manages access to and / or distribution of the data. The access control policy may indicate the same level of access for the entire file, or may indicate different levels of access for different parts of the file. In one embodiment, the file protection section may include a payload size field 321 and a payload field 322 (see, for example, FIG. 6). The payload size field may include one or more integer values ​​that indicate, for example, a size of a protection information payload that allows this field or information related thereto to be skipped or taken into account. The payload field includes information regarding the protection and access control policy of the file.

[0045]

[0068] The file metadata section 330 and the file traceability section 340 may include information as described in more detail below. In one embodiment, the file protection information, metadata, versioning, and traceability information may be compressed, for example, using a common compression algorithm, such as 7zip. In some implementations, a JSON / XML / XACML-based schema may be used to compress this information along with a uniform resource identifier (URI) representation (e.g., as done in MPEG-G part 3).

[0046]

[0069] The table information and index section 350 may include an nTables field 351 indicating the number of tables (n) in the file. In one embodiment, the unified and standardized format may store annotation data in multiple tables (e.g., n>2). In this case, different tables of the multiple tables may store various types of annotated data. In some cases, different tables in the file may be used to store the same genome annotation data at different resolutions. This is provided only as an example of table content. In other embodiments, the tables may store other types of annotation data.

[0047]

[0070] In addition to the nTables field 351, the table information and index section may store a Table ID[i] section 352, a TableInfo[i] section 353, and a ByteOffset[i] section 354. The Table ID[i] section 352 may store one or more integer values ​​indicating a unique identifier for the corresponding table i of the n tables in the file. The TableInfo[i] section 353 may store information indicating the resolution of the corresponding one of the tables i of the n tables. The ByteOffset[i] section 354 stores one or more integer values ​​indicating a byte offset of each of the tables i of the n tables in the file. Using this format, basic information (e.g., resolution level) about one or more of the tables in the file may be extracted without reading the entire file, for example using the TableInfo field in a format such as JSON / XML. Similarly, the byte offset of a table in the compressed file may be available to jump directly to a particular table.

[0048]

[0071] At 240, the genomic data in the received (source) file is divided into chunks or chunk groups. The chunks may have the same fixed size or may have variable sizes depending, for example, on whether the chunks are attribute-dependent and / or based on other parameters associated with the data. The chunks may be generated, for example, by parser 150 as described above.

[0049]

[0072] Different compressors (or compression algorithms) are specified for compressing the split chunks or chunk groups at 250. The compressors may be external to the unified and standardized file and / or may be initiated internally, for example by embedded code or by a selectable one stored in a compression parameter table.

[0050]

[0073] Thus, the compression parameters section 360 may include a field 361 that indicates the number of different compressors (or compression algorithms) used to compress the different information (e.g., attributes) contained within the table of the file. This information may be represented, for example, as one or more integer values. Section 360 may also include a field 362 that contains information listing the different compressors indexed by unique identifiers. These can be referenced within the table, thus avoiding repeated descriptions of compressors used in multiple tables or for multiple attributes.

[0051]

[0074] In one embodiment, as shown in FIG. 7, field 362 (or fields separate from or hierarchically related to field 362 in compression parameters section 360) may include a CompressorID field 363, an nDependencies field 364, a CompressorNameList field 365, and a CompressorParametersList field 366.

[0052]

[0075] The CompressorID field 363 may contain information (e.g., a string) that indicates a unique identifier for a corresponding compressor among multiple compressors. Thus, the CompressorID field may be used to identify the compressor used to compress each attribute or other portion of a file. In one embodiment, the unique identifier may be used in a table to point to the corresponding compressor.

[0053]

[0076] The nDependencies field 364 may include information (e.g., one or more integer values) indicating that compression of an attribute is performed based on other attributes. For example, a system processor may process information for compressing one attribute using one or more other attributes as side information (provided there are no circular dependencies). If so, this information may be included in the nDependencies field 364 of the unified and standardized file format. In one embodiment, an index in the variable nDependencies may indicate the number of dependency attributes used for decompression (e.g., the attribute information structure in FIG. 12 shows an example of the corresponding attributeID). Compressors such as context-based arithmetic coding may easily support the incorporation of side information. Another mechanism for incorporating side information from other attributes is to sort or split the values ​​of the current attribute based on the values ​​of other attributes. This allows similar values ​​to be grouped together to provide better compression. The parameters describe the dependencies used by each compressor in the list.

[0054]

[0077] The CompressorNameList field 365 may contain information (e.g., a string) indicating a list of compressor names. The compressor names may refer to one or more compressors external to the file (e.g., implemented by the compression module or the system processor 110), one or more compressors embedded in the file (e.g., via embedded or executable code), or both. One or more of the compressor names and parameters may correspond to standard compressors or describe a decompression mechanism. In one embodiment, one or more of the compressor executable code and parameters may be indicated in CompressorParameters, for example, by setting CompressorName to "EMBEDDED". In some embodiments, multiple compressors may be used to compress the same attribute or file portion. In this case, multiple compressors may be applied in order and displayed in a list included in a unified and standardized file format.

[0055]

[0078] The CompressorParametersList field 366 may contain information (eg, in a list or other form) that indicates one or more parameters that should be used to perform decompression of information compressed by all or some of the specified compressors.

[0056]

[0079] At 260, the method includes processing the received (source) file and / or policy information generated by the data owner or custodian to generate additional information including metadata, traceability information, linkage information, indexes, and protection information that is stored in corresponding fields as described herein.

[0057]

[0080] At 270, the unified and standardized genome annotation data file is stored for later retrieval. Because the different chunks of data have been selectively compressed, only designated portions of the compressed file may be extracted to access specific information within the file without the need to extract other information within the file. This improves processing speed and user convenience by targeting only those portions of the genome data that may be of particular interest at any given time.

[0058]

[0081] All or a portion of the information specified in the unified and standardized file may correspond to correlated information set forth in tables, structures, indexes, or other types of correlated information that allow selective compression and expansion of, and access to, genome annotation data. Examples of tables in correlated information are described in more detail below.

[0059] table

[0082] The tables 370 in each unified and standardized file may be related to each other or independent with respect to the attributes and / or other information stored. Each table includes multiple cells, and each cell may include one or more attributes. Each attribute has a specific data type and may be compressed using one of the compressors. If a cell in a table includes multiple attributes, the data corresponding to the multiple attributes in each cell of the table may be compressed separately to improve compression and to allow selective access to the attributes when later decompressing. In a genomic functional annotation file, the attributes may include, for example, a chromosome, a start position, an end position, a feature ID, and a feature name.

[0060]

[0083] Compression may be performed in different ways in different embodiments, e.g., attribute-by-attribute, cell-by-cell, table-by-table, etc. For example, for these purposes, groups of cells in the same table may be compressed separately from other groups of cells. Cells grouped together may have attributes corresponding to the same data, and other groups of cells may have different attributes corresponding to the same or different data. In one implementation, attributes in each cell may be compressed using different compression algorithms, or attributes in one or more cells may be compressed by a first compression algorithm, attributes in one or more other cells may be compressed using a second compression algorithm, attributes of one or more other cells may be compressed using a third compression algorithm, and so on.

[0061]

[0084] In an additional implementation, all or some of the compressors 1611, 1612,..., 161N may execute different compression algorithms specialized for different data / attribute types within each of the multiple cells. Although an example of a unified and standardized file format is discussed below as including multiple tables, in an embodiment, the file format may include only one table.

[0062]

[0085] In some examples, the content in each cell is described as an attribute, but the content may include other types of data (e.g., types of data, attributes, characteristics, etc.) or combinations of multiple types of data in other embodiments. For example, tables 370 in one file may store content or data that all represent the same genomic sequence, but each table in the array may represent that content or data at a different resolution.

[0063]

[0086] The system processor 110 may process and store the information in the file tables 370 in a variety of ways. For example, processing may be performed to generate one-dimensional tables and / or multidimensional tables. All tables in a given standardized file may be of the same type or may include a combination of types, e.g., both one-dimensional and multidimensional tables. In a genome annotation application, a one-dimensional table may include, e.g., genome annotation data or quantitative browser tracks, and a multidimensional table may include, e.g., variant call data and gene expression data for one or more samples. In one embodiment, a one-dimensional table may include multiple attributes.

[0064]

[0087] 8 illustrates an example table that stores data in a one-dimensional array 810 of cells. Each cell 820 in the array contains multiple attributes, labeled Attribute 1, Attribute 2, and Attribute 3. Each cell in the table has the same number of attributes, although in some embodiments the number of attributes in the cells of the table may vary. Example table structures and examples of how a system processor may process content for inclusion in a table structure are described in more detail below.

[0065]

[0088] 9 illustrates an example of a multi-dimensional table that stores data in a two-dimensional array of cells 910. Each cell 920 in the table contains one or more attributes that correspond to the data. The number of attributes in the cells may be the same or different. Example table structures and examples of how a system processor may process content for inclusion in the table structures are described in more detail below.

[0066]

[0089] In one embodiment, one or more of the multi-dimensional tables 370 in each file may store dimension-specific attributes in addition to the two-dimensional array 910. An exemplary configuration for storing dimension-specific attributes is similarly shown in FIG. 9. Here, along with the attributes stored in the array 910, the table 370 stores dimension-specific attributes in two additional tables. A first array 940 of dimension-specific attributes contains cells that store row attributes. A second array 950 of dimension-specific attributes contains cells that store column attributes. Thus, each array 940 and 950 can be considered to be part of an additional one-dimensional table in the table 370. One example of a table 370 including these features is intended to represent variant data of multiple samples in a two-dimensional array table 910, with the sample genotype and sample-level likelihood being attributes of each cell in the 2D array. Additionally, the array table 940 may store a dimension-specific attribute corresponding to a variant position, and the array table 950 may store a dimension-specific attribute corresponding to a sample name. An example of this table structure, and an example of how a system processor may process content for inclusion in the table structure, is described in more detail below.

[0067]

[0090] Figure 10 shows an embodiment of how the system processor 110 may process genome annotation data into a particular table format. Figure 11 shows an embodiment of a table format for the information processed to generate the table.

[0068]

[0091] As shown in Figures 10 and 11, the tabular format includes a number of sections similar to sections of unified and standardized file formats. For example, the tabular format may include a table header section 1010, a table protection section 1020, and a table metadata section 1030. The table header section 1010 may include a TableID field 1105 that indicates the name of the table, and a TableInfo field 1110 that indicates the payload (data) type and / or other information. The table protection section 1020 may include information that indicates the access control policy for the table. The table metadata section 1030 may include linkage, traceability, and / or other information.

[0069]

[0092] In addition to these features, the table may include a summary statistics section 1040, an attributes section 1050, an index section 1060, and a data section 1070. The summary statistics section 1040 may include information indicating averages, counts, distributions, and / or other information related to the data / attributes stored in the table, which may correspond to one or more key values. The statistics may be used, for example, to enable fast access to relevant statistics. This information corresponds to reference numeral 1140 in FIG. 11.

[0070]

[0093] Attribute section 1050 stores various attributes for one-dimensional and / or multi-dimensional array tables as described herein, and may or may not have intrinsic dimension attributes. For example, if dimension-specific attributes are included in the table, section 1050 may include fields 1150 indicating the size, name, and metadata of each dimension i (i=1,...,N,N>=2), as well as the dimension-specific attributes corresponding to each dimension as shown in the table. For two-dimensional data, the SymmetryFlag field may also store information indicating whether the dimensional array is symmetric, e.g., whether it is symmetric Hi-C data.

[0071]

[0094] The table includes an nAttributesMain field 1160 indicating the number of attributes in the main array (or two-dimensional array), followed by attribute information for each attribute. The attribute information may include content and structure as described in more detail herein. Field 1160 may also include byte offset information of the attribute / data in the 2D array table. Selective compression of attributes based on different algorithms and byte offsets corresponding to the attributes may allow selective access to the attributes (particular portions of the genome annotation data) during decompression without the need to decompress other attributes, table sections, or other data contained in the unified and standardized file. Index section 1060 may store indexes as described in more detail herein, and data section 1070 may store data including, for example, chunks or chunk groups as described herein.

[0072]

[0095] FIG. 12 illustrates an embodiment of how the system processor processes genome annotation data to generate attribute information and associated structures that may be included within tables or other sections of the unified and standardized file.

[0073]

[0096] 12, attribute information may be processed into a structure including an AttributeInfoSize field 1205, an Attribute ID field 1210, an AttributeName field 1215, an AttributeMetadata 1220 field, and an Attribute Type field 1225. The AttributeInfoSize field includes information indicating the size of this information, which allows a processor to skip this structure. The Attribute ID field may include a unique identifier for the attribute. The AttributeName field includes information indicating the name of the attribute, and the Attribute Metadata field may include information for linking or grouping the attribute with one or more other attributes.

[0074]

[0097] The Attribute Type field 1225 may contain information that identifies the attribute as one of two types: the first type is a base type that indicates whether the attribute corresponds to a character, a string (null terminated), a float, a double, a Boolean, or signed and unsigned integers with different bit widths, and the second type is a derived type that indicates whether the attribute corresponds to a variable length array or a fixed length array.

[0075]

[0098] In addition to these features, the attribute information structure may include a DefaultValue field 1230, a Summary Statistics field 1235, and a CompressorID field 1240. The DefaultValue field indicates whether most values ​​of the attribute are equal to a predefined default value. If so, at least the corresponding attribute in the file may be encoded using a predefined type of encoding (e.g., sparse encoding). The Summary Statistics field may include information indicating averages, counts, distributions, and / or other statistical data that may enable fast analysis of the attribute.

[0076]

[0099] The Compression ID field 1240 includes information indicating the type of compression algorithm to be used for the attribute. This information may, for example, identify one of the compressors 1611, 1612, ..., 161N assigned to compress the attribute. This information controls which decompression algorithm should be used to restore the attribute when retrieved in compressed form by the system processor. If the compressor uses side information / context during the decompression process, the corresponding dependency attributes may also be specified in the attribute information. In the case of multi-dimensional arrays, the side information may be obtained from the multi-dimensional array attributes or from dimension-specific attributes. For example, in a VCF file, variant-specific fields (e.g., dimension-specific attributes) may be used as side information for compression of genotype data (which is an attribute of a two-dimensional array table). In one embodiment, the system processor may process the attribute information to include additional data required for decompression in the variable CompressorCommonData. This additional data may, for example, be common to all chunks. This additional data may be useful for storing codebooks, dictionaries, or statistical models computed from all or part of the genome annotation data file. Chunks and chunk-related processing are described in more detail below.

[0077] Compressor

[0100] Information (e.g., attributes, data, and other information) received by compression manager 160 may be selectively compressed using, for example, different compressors 1611, 1612,..., 161N of the system shown in Figure 1. A non-limiting list of these compressors includes: Run-length compression: This compression algorithm is used, for example, for long runs of the same value and may involve performing a substitution operation based on the value and length of the run. Differential encoding: this compression algorithm is used, for example, to increase a sequence of numerical values ​​and may involve substitution operations based on the difference between successive values. Dictionary-based / enumeration: This compression algorithm may be used, for example, for attributes whose values ​​correspond to a small set of options. In this case, the algorithm may perform a substitution operation based on an index in the set and store the resulting dictionary in the CompressorCommonData field. Sparse: This compression algorithm may be used to compress attributes that differ slightly (e.g., less than a predetermined value) from the default value, for example, as shown in the attribute information of FIG. 12. This type of compression may involve representing the attribute as a coordinate location and a value for a non-default value. To improve compression, the coordinate location may further be differentially encoded within each chunk, for example, in a two-dimensional sparse array, row indices may be differentially encoded and column indices may be differentially encoded within each row. Variable length arrays: This compression algorithm may be used, for example, to separate a variable length array into a stream of values ​​and a stream of lengths. Tokenization: This compression algorithm is used, for example, for structured string attributes and may involve splitting the attribute into different types of tokens and encoding each token based on its previous value (e.g., match with previous token, delta, new value, etc.). · General purpose compression / entropy coding methods: These types of compression methods include, for example, gzip, bzip2, 7-zip, adaptive arithmetic coding, and other lossless data compression (for example, but not limited to, BSC's lossless block sorting compression).

[0078]

[0101] The above list of compression algorithms is not intended to be a limiting list. In an embodiment, one or more specialized or customized compression algorithms may be included in addition to or instead of one or more of the above compressors. One example of such a specialized compression algorithm is GTC (genotype compressor), which may be used to compress variant call data. Such compression may support fast random access to information contained in the rows / columns of the compressed table. In some cases, parts of the file may not be compressed, for example to allow faster selective access.

[0079]

[0102] One or more of the compressors 1611, 1612, ..., 161N may generate multiple streams. One example is a sparse compressor that generates a stream of coordinates and a stream of values. In one embodiment, these streams may be compressed using, for example, different entropy coders implemented with appropriately specified parameters. For example, consider the following: CompressorNameList=['sparse','gzip','7-zip'] CompressorParameterList=[ {“outStreams”:[“coordinate”,“value”]}, {“inStreams” :[“coordinate”]}, {“inStreams” :[“value”]} ]

[0080]

[0103] In this case, gzip compression may be applied to the coordinate stream and 7-zip to the value stream (e.g., JSON may be used to represent the parameters). This allows the application of a compressor per data stream, which may produce predetermined or optimal results. If no stream is specified, the same compression method may be applied to all received streams.

[0081]

[0104] As described above, in addition to or instead of one or more external compressors, the compressed file (processed into a unified and standardized file format) may include one or more embedded compressors. That is, the system processor may process the genome annotation data to include an embedded compressor executable within the unified and standardized file format, preferably with appropriate security protection. In the case of an embedded compressor, a corresponding unpacking executable code may be included in the compression parameters, along with one or more digital signatures as proof of origin and authenticity to protect against malicious software. For interoperability between different platforms, a standardized virtual machine bytecode may be used for the unpacking executable code.

[0082] Chunking and Indexing

[0105] 13 illustrates an embodiment of a method for processing information organized into a unified and standardized file format, in which compressed files are processed to allow efficient random access to the compressed information during decompression and information recovery.

[0083]

[0106] 13, the method includes, at 1310, dividing the attributes, data, (1D and / or 2D) tables, and / or other information in each table 370 (or the entire file) into chunks. The chunks may all have the same fixed size or may have variable sizes. The same chunks may or may not be used for all attributes. In one embodiment, the attributes in each table 370 are divided into rectangular chunks to allow efficient random access.

[0084]

[0107] At 1320, after division, the chunks are selectively compressed by different ones of a number of compressors (eg, as dictated by instructions executed by the system processor 110).

[0085]

[0108] The processor then generates at least one index at 1330 for each compressed chunk or chunk group to be included in the unified and standardized format compressed file. The index may include information for efficiently determining the location of each particular chunk within the compressed file. Processing the data into an index may enable fast access to any location within the data by only selectively decompressing the corresponding chunk without decompressing other chunks that are unrelated or unrelated to the chunk or chunks of interest. To support fast random access based on the value of a particular attribute, the system processor may also process the data to generate attribute-specific indexes. In one embodiment, each compressed file may also include information that allows sharing of a codebook or statistical model for performing decompression across multiple chunks.

[0086]

[0109] FIG. 14 illustrates an example of an index structure generated by the processor. This example corresponds to an index structure for a one-dimensional table when the flag AttributeDependentChunks field is false, as described below. The index includes a first section 1410 and a second section 1420. The first section 1410 includes information corresponding to a number of chunks, each chunk including start index information 1411, end index information 1412, and byte offset information 1413. The second section 1420 includes a number of additional indexes for attributes included in the chunk or other parts of the associated compressed file. Each of the additional indexes may include indexed attribute information 1421, index type information 1422, and index data information 1423. The index file structure may be included, for example, in the index section of each table (e.g., as shown in FIG. 10) or in selected tables of the compressed consolidated and normalized file.

[0087]

[0110] FIG. 15 illustrates an example index structure according to one or more embodiments, and FIG. 16 illustrates an example chunk structure that may correspond to and provide a better explanation of the index structure of FIG. 14 according to one or more embodiments.

[0088]

[0111] Referring first to FIG. 16, the chunk structure may include an nChunks field 1605 and a VariableSizeChunks field 1610. The nChunks field 1605 includes information (e.g., an integer) indicating the number of chunks corresponding to a particular attribute. The VariableSizeChunks field 1610 includes a flag indicating whether the nChunks have the same fixed size or different sizes. While fixed-size chunks are simpler to process (especially when included in a multi-dimensional sequence table), variable-size chunks may be useful when the choice of a fixed chunk size is not optimal, for example, because the sparsity of the data varies widely. In some cases, variable-size chunks may be better suited to certain attributes of genome annotation data (e.g., chromosomes, genome locations, etc.), and therefore may enable faster random access for these attributes.

[0089]

[0112] If the chunk structure indicates that variable sized chunks are being used, then the start index 1615 and end index 1620 of each chunk along each dimension may be included within the chunk structure. This information is also shown in the compressed file index structure of Figure 14. In the case of fixed sized chunks, the chunk size along each dimension may be indicated. In either case (fixed or variable size), byte offset information 1625 may be included within the file for each chunk to allow random access.

[0090]

[0113] 17 shows an example of a portion of a genome annotation data file associated with a particular attribute. In this example, the data is arranged in a two-dimensional array and divided into chunks 1710 of the same fixed size (e.g., ChunkSize(1) is 5 and ChunkSize(2) is 11) with a particular ByteOffset[j]. The portion of data shown contains a total of 15 chunks (nChunks=15), with the VariableSizeChunks field indicating a false value to specify the fixed size.

[0091]

[0114] FIG. 18 shows an example of a portion of a genome annotation data file related to a particular attribute. In this example, the data is arranged in a two-dimensional array and divided into chunks 1810-1840 with variable sizes. Since the chunk sizes are variable, a start index and an end index are provided for each block along with an associated ByteOffset. For example, Startindex[1][1]=1 and Endindex[1][1]=17 are shown for the first direction of the first rectangular chunk (chunk 1), and Startindex[1][2]=1 and Endindex[1][2]=17 are shown for the second direction of the first rectangular chunk. Corresponding ByteOffset information is also provided for chunk 1. Similarly, Startindex[2][1]=1 and Endindex[2][1]=11 are shown for the first direction of the second rectangular chunk (chunk 2), and Startindex[2][2]=18 and Endindex[2][2]=31 are shown for the second direction of the second rectangular chunk. The data portion shown includes a total of four chunks (nChunks=4), with the VariableSizeChunks field indicating a true value to specify the variable size. The remaining chunks 3 and 4 may be specified similarly. Information corresponding to all four chunks for the attribute may be incorporated into the chunk structure, index structure, corresponding tables, and / or other information associated with the unified and standardized file of genome annotation data.

[0092]

[0115] Referring again to Figure 15, the index structure includes information including an AttributeDependentChunks field 1510. This field may include a flag indicating whether the chunk size is dependent on one or more attributes or whether the same chunking (e.g., the same chunk size) is used for all attributes associated with the file. If the flag has a first value, then the same chunking is used for all attributes at 1515. Using the same chunking for all attributes significantly reduces the size of the index structure and may be useful, for example, when all attributes in a chunk are queried most of the time.

[0093]

[0116] 19 shows an example of information included in a data payload structure for the one-dimensional case when the AttributeDependentChunks flag has a first value (e.g., false). In this example, data 1910 is split into multiple chunks 1920 (e.g., nChunks=5), with each chunk corresponding to one or more attributes 1930. In this case, chunk 1 corresponds to three attributes. Each attribute also includes information indicating the payload size and the payload.

[0094]

[0117] Referring again to FIG. 15, if the flag has a second value, at 1520, attribute-dependent chunking is used as shown in the chart of FIG. 16. Attribute-dependent chunking may be useful, for example, when the optimal chunk sizes of different attributes for compression and random access are significantly different (e.g., above a predetermined value). For example, if some attributes are sparse and other attributes are dense, using the same chunk size may not optimize compression. Attribute-dependent chunking may also be useful when all chunks of a single attribute are queried frequently, for example, above a predetermined frequency value.

[0095]

[0118] FIG. 20 illustrates an example of a data payload structure for the one-dimensional case when the AttributeDependentChunks flag has a second value (e.g., true). In this example, data 2010 corresponds to one or more attributes 2020. Each attribute includes or corresponds to one or more chunks 2030 into which the data is divided. Each chunk also includes a payload size and information indicative of the payload. As is evident from a comparison of FIG. 15 and FIG. 20, according to one embodiment, the organization of chunks and attributes depends on the mode of operation, e.g., whether attribute-dependent chunking is performed or not.

[0096]

[0119] In one embodiment, random access during the selective unfolding process may be performed without using row and / or column numbers. This embodiment may be performed, for example, for applications where random access is performed with respect to a particular attribute, for example, for applications where random access is performed with respect to a genomic location.

[0097]

[0120] In these cases, the index structure of FIG. 15 may include one or more attribute-specific indexes 1525. For example, if attribute-dependent chunking is performed, nAdditionalIndexes may be used (n≧1). To perform faster queries by attribute rather than by row / column, the index structure may include one or more attribute indexes. For example, for genome annotation data, indexes for chromosome and location attributes may be generated by the system processor 110 and included in the index structure of FIG. 15. When a query is performed by the system processor, chunks or chunk numbers that match the chromosome and location attributes of the query may be returned for selective expansion. Different attributes and / or different numbers of attributes may be used in other embodiments.

[0098]

[0121] The index structure may also include other information related to the nAdditional index. For example, for each attribute-specific index, the index structure may include an AttributeIDsIndexed field 1530, an IndexType field 1535, an IndexSize field 1540, and an IndexData field 1545. The AttributeIDsIndexed field 1530 may include a list of one or more attributes (e.g., chromosomes, genomic locations, etc.) that are indexed.

[0099]

[0122] The IndexType field 1535 may include information indicating the type of indexing performed for each attribute listed in the AttributeIDsIndexed field. In one embodiment, the type of indexing for one or more attributes may differ from the standard set. For example, an R-tree or CSI index may be used for each of the chromosome and genome location attributes, as well as for range queries. A B-tree index may be used for database type queries. A genome range indexing may store the left and right coordinates of each chunk, which allows for fast identification of chunks that overlap with the queried range. Similarly, a B-tree index may store a map from attribute values ​​to chunks that contain the value and the location of the value within the chunk.

[0100]

[0123] The IndexSize field 1540 may contain information indicating the size of the index generated by the system processor for each listed attribute.

[0101]

[0124] The IndexData field 1545 may contain the actual indexing data in a predefined (e.g., binary) format. The type of format may depend, for example, on the type of index created for the attribute.

[0102]

[0125] FIG. 21 illustrates exemplary logic that may be executed by the system processor 110 to generate payload information (e.g., as shown in the data of FIG. 19 and / or FIG. 20). This logic is implemented based on the value in the AttributeDependentChunks field. More specifically, as is evident from a comparison of FIG. 19 and FIG. 20, the structure of the payload differs depending on the value of the AttributeDependentChunks field. If the chunks of data in the payload are attribute-independent, the system processor implements logic to generate the structure shown in FIG. 19 as described above. If the chunks of data in the payload are attribute-dependent, the system processor implements logic to generate the structure shown in FIG. 20 as described above.

[0103]

[0126] FIG. 22 illustrates one embodiment of a query method that may be performed by the system processor 110 to selectively identify and expand annotated genomic data based on an index structure (such as, but not limited to, the index structure of FIG. 15). The query may be performed based on a lookup operation implemented as follows. First, at 2210, the system processor 110 receives a user query including information indicating one or more attributes of interest. At 2220, the system processor parses the query and identifies one or more attributes in the query. Within the query, the one or more attributes may be specified in various ways, such as a string such as "abcd" or a particular attribute number in a series of numbers that identify the respective attribute. At 2230, the system processor searches the attribute-specific index to determine one or more chunks or one or more chunk numbers that correspond to the attributes identified in the query. At 2240, the system processor searches the genomic annotation data to retrieve one or more chunks or one or more chunk numbers that match the query attributes (or otherwise obtain access to such chunks or chunk numbers). By restoring the chunks using the chunk index, values ​​that match the condition can be filtered (as the chunk may contain non-matching values).

[0104]

[0127] At 2250, the recovered chunks may be selectively expanded (e.g., expanded without expanding other chunks unrelated to the specified attribute). Selective expansion is possible because multiple data chunks were previously selectively compressed independently of one another. The expansion may be performed based on the algorithms shown in the attribute information structure of FIG. 12 and / or the index, table, or other structures described herein. If the chunks were globally compressed, the global compressed data may be shared between the chunks, for example, using the CompressorCommonData mechanism shown in the attribute information structure. In one embodiment, in the case of a symmetric two-dimensional array table (e.g., when the SymmetryFlag of the table in FIG. 11 is true), the chunks may only cover a portion, for example, the lower triangular portion and the diagonal. In this case, the expansion process may address the upper diagonal values ​​by filling them with the corresponding lower triangular values. In other cases, the chunks may cover the full range of the index without overlapping.

[0105] interoperability

[0128] The unified and standardized file format of genome annotation data may have various additional features. One additional feature relates to the linkage and interoperability associated with files generated in this format. In one embodiment, the system processor can process the file information into the unified and standardized format as described herein, so as to be independent of any encoding or formatting of the source file (e.g., data provided by data source 140 of FIG. 1).

[0106]

[0129] In another embodiment, the system processor can process the file information into a unified and standardized format so as to be linked with the encoding and / or formatting of the source file and to be interoperable. The data source file may be in MPEG-G format / encoding or another type of format or encoding.

[0107]

[0130] Once processed into a linked, interpretable state, the files can be treated as part of the source file (e.g., an MPEG-G file) by storing the associated information in a dataset. The MPEG-G file stores data for an entire study, with each dataset group corresponding to a different individual. Each MPEG-G dataset group is further split into multiple datasets corresponding to multiple different sequencing runs.

[0108]

[0131] To store data corresponding to a single individual, multiple different annotation files are assembled into multiple separate datasets, each dataset containing a single annotation file or sequencing data. Examples of separate datasets are shown below. Dataset Group (Single Individual) → Dataset 1 (Sequencing data) Dataset 2 (Sequencing data) Dataset 3 (variant call data) Dataset 4 (Gene expression data) ...

[0109]

[0132] When collecting annotation data from a large-scale survey, the dataset may be organized as follows: Dataset Group (large-scale survey) → Dataset 1 (variant call data) → Annotation file (sample 1) Annotation file (sample 2) ... Dataset 2 (gene expression data) → Annotation file (sample 1) Annotation file (sample 2) ... ...

[0110]

[0133] In one embodiment, multiple different annotation files may be merged to improve compression and analysis performance, for example, as shown below. Dataset Group (large-scale survey) → Dataset 1 (variant call data) → Annotation files (all samples) Dataset 2 (gene expression data) → Annotation files (all samples) ...

[0111]

[0134] To perform this implementation, the system processor may extend the existing dataset header structure with additional fields to support the data type (sequencing / variant / gene expression / ...), the number of annotation files in the dataset, and the byte offset of each of these files. If a compressor is shared between annotation files or datasets, the compressor's parameters may be stored at the dataset level or dataset group level, respectively. In one embodiment, one or more of the annotation files includes a compressor structure with compressor name "POINTER" and compression parameters that store the position, e.g., {"DatasetGroupId":1, "DatasetId":2, "CompressorId":5} indicates that the compressor is as specified in dataset group 1, dataset 2, 5th compressor.

[0112] linkage

[0135] In addition to the above features, the system processor may process the data to generate a unified and standardized file in a format that includes linkage between different types of annotation data and corresponding sequencing data. In one embodiment, this linkage may be provided based on metadata stored in or associated with the file.

[0113]

[0136] This may be accomplished by the system processor specifying the dataset group or datasets that store the sequencing data or associated annotation data in the FileMetadata or TableMetadata fields of Figure 4, for example, using a uniform resource identifier (URI) notation as described in MPEG-G part 3, or by using JSON. For example, the following JSON may be used in the FileMetadata to provide linkage to the sequencing datasets: "Linkages":[{ “DataType” :“Sequencing”, “DatasetGroup”:5, “Dataset”:2 }] Although the above examples show only a single linkage, in alternative embodiments multiple linkages may be provided.

[0114]

[0137] Additionally or alternatively, the system processor may generate table-level linkage. In one embodiment, the system processor may generate table-level linkage by index. In this case, for example, the nth row (column) of one table may correspond to the mth row (column) of another table. This type of linkage can prevent repetition when multiple annotation files / tables share the same row / column (e.g., multiple VCFs that are not yet merged and consist of the same variant). Similarly, this type of linkage can be useful when sample-related information is stored in a single table, to which both the VCF table and the gene expression table link.

[0115]

[0138] In another embodiment, the system processor can generate table-level linkages by index. In this case, a particular attribute can be linked by matching the value with an attribute in another table. For example, gene expression data can include a gene name without detailed information about the gene, which is available in another file. An example of the use of such linkages could be a query requesting gene expression data for all genes of the MHC (major histocompatibility complex) that correspond to an autoimmune disease and specify a coordinate range in human chromosome 6. To address this query, multiple gene names for a coordinate range can be obtained from the gene information file based on the genome coordinate index, and the required data can be obtained by querying these names in the gene expression file. The following example relates to these features, specifically, a row (dimension 1) can be linked to a row in another table (e.g., table No. 3 in the same annotation file). "Linkages":[{ “Type” : “byIndex”, “DimensionInCurrentTable”:1, “Table”: 3, “DimensionInLinkedTable” :1 }]

[0116]

[0139] An example of linking a column (dimension 2) to a row in another table by an attribute value (attribute 2 in dimension 2 of the current table is linked to attribute 5 in dimension 1 of table 3 in dataset 4 of file 2) may be performed by the system processor as follows: "Linkages":[{ “Type”:“byValue”, “DimensionInCurrentTable”:2, “AttributeInCurrentTable”:4, “Dataset”: 4, “AnnotationFile”:2, “Table”: 3, “DimensionInLinkedTable” :1, “AttributeInLinkedTable” :5, }]

[0117]

[0140] Because the metadata structure supports arbitrary information storage, the system processor may further extend the linkage to link three or more tables, for example using a standardized format (e.g., table 3 may convert a gene ID used in table 1 to a gene name in table 2). Although a particular JSON-based format is used for the linkage in the above example, another format may be used, for example, but not limited to, XML.

[0118]

[0141] In one embodiment, attribute-based linkage may be implemented. While metadata-based linkage may be useful for high-level linkage, in some cases, row / column-wise linkage may be beneficial. For example, in a VCF file with multiple samples, sequencing data corresponding to a particular sample may be linked by adding attributes SequencingDatasetGroup and SequencingDataset to column attributes. Such linkage attributes may have "LinkageAttributeFlag" set to true in the metadata so that the decompressor can distinguish linkage attributes from normal attributes.

[0119]

[0142] In some cases, the system processor may perform a mapping operation between annotation datasets according to genomic regions. This may be achieved, for example, by indexing each dataset separately. For example, to find sequencing data corresponding to a region in the VCF file, one or more suitable access units may be determined by accessing a master index table of sequencing data. By using separate indexing for different datasets, a predetermined (e.g., optimal) chunk size and other parameters can be selected for each dataset. Furthermore, in some cases, variants may not be directly linked to AUs because the AU classes are different. Similarly, in a VCF file with multiple samples, variants are mapped to access units across multiple datasets, and storing this information may occupy large storage. In one embodiment, the AUId or byte offset of the sequencing data is stored as a row attribute in the VCF file, which allows for fast lookup of the access unit corresponding to the current variant. Also, genes may be mapped to a list of variants by using a list type attribute for genes.

[0120]

[0143] In one embodiment, the access control policies use a predefined format such as XACML and may be specified at the file level (e.g., in the annotation file of FIG. 4), the table level (e.g., in the information of FIG. 10 and FIG. 11), or both. Certain users may have access to all data, while other users may only have access to coarse resolution data (recall that different resolutions are stored in different tables). This type of policy may be specified at the file level. Policies specific to attributes in a table may be specified, for example, at the table level. This may include access to only a subset of attributes, or to only certain chunks based on the value of the attribute. Another type of policy may allow access to metadata and information, but not to the actual data.

[0121] Expand

[0144] The process for extracting selected portions of a file, or the entire file, in the unified and standardized format described herein may first include performing one or more types of queries. The queries and / or extraction algorithms may not be mutually exclusive and may be combined in some embodiments. For example, both metadata and specific attributes may be extracted, or selected attributes from selected chunks may be extracted. The extraction may be performed by a system processor or by another processing entity coupled to the system processor. In some cases, some of these queries may be restricted by access control policies. Application programming interfaces (APIs) may be used to support these. Such APIs may be similar to MPEG-G part 3, or may be different types of APIs.

[0122]

[0145] Metadata / Information Queries: These types of queries may query only metadata and information corresponding to the requested table (e.g., resolution level), compressor, attribute, and / or chunk. When performing a query, the top-level information may first be directly accessed as described at the beginning of the annotation file in FIG. 4. Table-specific metadata / attribute details may be selectively accessed using the ByteOffset of the table in FIG. 4.

[0123]

[0146] Full Data Expansion: This type of expansion may involve the expansion of the entire data, including all tables and attributes. This may be performed by first reading the top-level metadata and table information of the file, as described above. Next, the compression parameters are loaded into the expansion manager. Then, for each table, the table information, dimensions, and attributes are read, and then the index is read to determine the chunk's location along each dimension. Then, the data payload is selectively expanded (serial or parallel) for each chunk and each attribute. If an attribute is compressed using another attribute as a dependency / context, the expansion may be performed by expanding that other attribute first. If the attribute uses CompressorCommonData, this information may be loaded before expanding the chunk. In the case of two-dimensional symmetric arrays (e.g., see SymmetryFlag above), only the diagonal and lower triangular matrices may be expanded, and the upper triangular portion may be filled in using symmetry.

[0124]

[0147] Extraction of only one table: This type of extraction is similar to the full data extraction, except that the ByteOffset field in the annotation file (e.g., in Figure 4) is used to jump to the requested table, and only that table is extracted.

[0125]

[0148] Querying selected attributes of a table: This method is similar to extracting only one table, except that only the information corresponding to the requested attributes is read. Other attributes are skipped based on the AttributeInfoSize variable in the attribute information structure of FIG. 12. In one embodiment, only the requested attributes are extracted by skipping other attributes based on the Payload Size[i][j] information of FIG. 22. If attribute-dependent chunks are used, all chunks of a given attribute may be stored together, for example, as shown in the data payload structure of FIG. 20.

[0126]

[0149] Querying only selected index ranges in an array: This method is similar to unpacking only one table, except that the indexes are loaded and depending on the type of chunking (fixed size / variable size), the chunks that overlap with the requested range are determined. Also, the ByteOffset information in the chunk structure table in FIG. 16 may be used to jump to the payload of the chunks determined above. In some cases, the method may be implemented more efficiently if attribute-dependent chunking is not used and all attributes of a given chunk are stored together. Also, the requested chunk is unpacked and only the overlapping indexes are returned. If the compressor of an attribute allows efficient random access within a chunk, this random access may be utilized as a basis to further increase the unpacking speed. Examples of scenarios where this may happen include specialized compressors for sparse arrays or genotypes such as GTC.

[0127]

[0150] Querying based on values / ranges of specific attributes: This approach is similar to querying only selected index ranges in an array, but if an additional attribute-specific index (e.g., as shown in the index structure of FIG. 15) is available for the chunk of interest, this index may be used to determine the chunk or chunks. If such an index is not available, the attribute of interest may be expanded for all chunks, and then the associated chunks may be determined. Even if no additional index is used, the expansion speed may be improved since only some attributes are expanded for all chunks. In one embodiment, the remaining attributes may be expanded only for the associated chunks.

[0128] Folder Structure and Editing

[0151] The processing of genome annotation data into a unified and standardized file format by the system processor as described above provides many advantages in terms of efficiency, convenience, reduced system requirements, fast querying, and data accessibility. Additional variations and adjustments may be made to the embodiments to better suit specific applications.

[0129]

[0152] For example, if the data is stored on a single machine and is frequently edited, the data may be stored in a directory / folder hierarchy using a file manager. This hierarchy may make it easy to manipulate portions of the data by only modifying files that correspond to single chunks and attributes, rather than overwriting the entire file. When editing is complete and the data is sent, the file may be converted back to a single file format. This can be done by recalculating the index based on the data payload size and folding the folder hierarchy back into a single file.

[0130]

[0153] An example of how to convert a file format (e.g., FIG. 23A) to a folder hierarchy (e.g., FIG. 23B) and back to a single table is shown. In this example, the conversion is performed by treating each table in the table section 2310 of the file format as a folder in the folder hierarchy, and treating each chunk of data in the data section 2320 of the file format as a folder in the folder hierarchy (e.g., assuming a false value is stored in the AttributeDependentChunks field). In the folder hierarchy, blocks 2331-2343 correspond to files, and blocks 2321-2326 correspond to folders. In the folder hierarchy, the chunks are already stored in separate folders, so the index only needs to store attribute-specific indexes. A simple example is shown in FIG. 9.

[0131]

[0154] Considered as a folder hierarchy, the proposed scheme can be loosely related to that proposed in the above-referenced related patent application by changing the terminology as follows: "attribute": "region with one or more rows / columns" and "chunk": "block".

[0132]

[0155] To illustrate how one or more of the above embodiments may be implemented, the following two examples for storing various annotation data while providing related functionality are discussed: (1) variant call data, and (2) genomic functional annotation data. [Table 1]

[0133]

[0156] Table 1 above shows a section of a VCF file containing five variants and three samples. A VCF file can be processed by the system processor 110 into a unified and standardized file format, both to hold data and to provide additional associated functionality. Such a file can have the following features:

[0134]

[0157] Metadata: Comment lines (starting with ##) may be kept as part of the FileMetadata. If this is saved as part of an MPEG-G file along with the sequencing data, the metadata may also include the corresponding dataset group that contains the sequencing data that corresponds to this variant call data.

[0135]

[0158] Traceability: When traceability information is stored as part of an MPEG-G file along with the sequencing data, it may include one or more commands for the generation of one or more variant calls, starting with the URI of the raw sequencing data and the tools used and their versions. The traceability information may be used to validate the file in a reproducible manner.

[0136]

[0159] Tables: If the variant data is stored at a single resolution, the variant data may be stored in a single table in the unified file format (nDimensions=2).

[0137]

[0160] Dimension Attributes: For the first dimension (variant), there may be multiple dimension attributes such as CHROM, POS, ID, REF, ALT, QUAL, FILTER, and INFO fields. As described in the comments, the INFO field may be split into multiple attributes such as NS, DP, AF. The types of these attributes may also be mentioned in the comments field. Attribute metadata may be used to group them (e.g., NS, DP, AF may belong to the group INFO). The default value depends on the attribute, e.g., for the FILTER attribute, the default value may be set to "PASS".

[0138]

[0161] For the second dimension (sample), the sample name (e.g., NA00001) may be the only attribute present in the original VCF file. Additional attributes may be added to support linkage to sequencing data, e.g., the dataset group and dataset that contain the sequencing data corresponding to this sample. Additional dimension attributes may be added to support fast access to specific quantities, e.g., counts or average quantities corresponding to specific variants. A description of the INFO attribute in a comment may be saved as part of the AttributeMetadata.

[0139]

[0162] 2D Table Attributes: These attributes are described in FORMAT fields such as GT, GQ, DP, etc., each of which is a two-dimensional array. The type of these attributes may also be described in comments. If most variants are not represented, the default value of GT attributes may be set to e.g. 0 / 0. The description of these attributes in comments may be stored as part of the AttributeMetadata.

[0140]

[0163] Compressors: Compressors for attributes may be selected based on the type and characteristics of the attribute. For example, CHROM may be compressed using gzip after an enumeration-based scheme, and POS may be compressed using gzip after differential encoding. Sample names (such as NA00001) may be efficiently compressed using, for example, a tokenization-based string compressor. Some INFO fields may only be present for a small number of variants and therefore may be encoded with a sparse representation. Similarly, genotypes (GT) may be encoded using a sparse representation, or a compressor dedicated to genotypes (e.g., GTC).

[0141]

[0164] The length of a particular variable length attribute may depend on one or more other attributes. For example, the length of the AF (allele frequency) attribute may be equal to the number of variant alleles. In such a case, the compressor's nDependencies may be set to 1, and this dependency may be exploited to enhance compression. Similarly, the value of the GT field may be used as side information for the compression of other FORMAT fields, exploiting the dependency between the two.

[0142]

[0165] Chunking and Indexing: Chunking of the main 2D array may be performed depending on the access pattern. If most accesses are to variants in a particular region, then each chunk may contain all samples and a few variants (e.g., horizontal chunks). If most accesses are to all variants of a particular sample, then a chunk may contain all variants and a few samples (e.g., vertical chunks). If both types of queries occur frequently, then in some cases it may be better to use rectangular chunks that contain a few variants and samples. By increasing the size of the chunks, random access performance may be traded off for compression ratio.

[0143]

[0166] For random access based on genomic regions, additional indexes may be used (e.g., based on CSI indexing) as shown in Table 2. [Table 2]

[0144]

[0167] Instead of specifying an actual file location as is done with CSI, a list of chunk IDs that overlap with the genomic region of interest may be returned (or indicated). The locations of these chunks in the file may then be determined from a default index structure. If indel or structural variants are common, CSI indexing may be done based on both the START and END locations of the variants. More attributes may be indexed to allow for fast random access queries. For example, indexing the FILTER attribute may enable faster filtering of variants based on whether FILTER=PASS or not.

[0145]

[0168] Protection: Access control policies can take various forms, depending on, for example, the use case. For example, certain users may have access to all data, while others may only have access to variants in specific genomic regions (e.g., specified by CHROM and POS). Similarly, access may be restricted to only certain samples. In some cases, this may require chunks to be selected accordingly. Access control may also be imposed at the attribute level, for example, access to INFO fields may be allowed, but not to individual sample data. [Table 3]

[0146]

[0169] Table 3 above shows a section of a BED file with annotation data. A system processor may process the information in this file to conform to a unified and standardized file while retaining the data and providing additional functionality, as described below.

[0147]

[0170] Metadata: The comment lines (e.g., the first three lines) may be retained as part of the FileMetadata. If this is saved as part of an MPEG-G file along with the sequencing data, the metadata may also include the corresponding dataset group that contains the sequencing data that corresponds to this annotation data.

[0148]

[0171] Tables: To allow data to be displayed at different scales and resolutions, multiple tables may be generated by the system processor with pre-calculated values ​​for different resolutions. For example, the TableInfo field may store parameters indicating the resolution and other information in a predefined format. This may allow a user to query a list of available resolutions without reading the entire file. Also, the desired resolution may be directly accessible by using a ByteOffset variable for each table. One or more of the multiple tables may have a single dimension, for example.

[0149]

[0172] Attributes: In one embodiment, each column can act as an attribute as follows: chrom(string), chromStart(integer), chromEnd(integer), name(string), score(integer), strand(character), thickStart(integer), thickEnd(integer), itemRGB(8-bit integer array of length 3).

[0150]

[0173] Compressor: A compressor for an attribute may be selected based on the type and characteristics of the attribute. For example, chrom may be compressed using an enumeration-based scheme followed by gzip or run-length compression, and chromStart and chromEnd may be compressed using gzip followed by differential encoding. If the values ​​of thickStart and thickEnd are close to chromStart and chromEnd, compression may be improved by using these values ​​as side information, for example.

[0151]

[0174] In this example, the value of chromStart matches the value of chromEnd from the previous line. One way to take advantage of this is to consider chromStart and chromEnd as a single attribute of type "integer array of length 2". This can be done, for example, if the visualization tool understands this alternative representation.

[0152]

[0175] Chunking and Indexing: For random access based on genomic regions, additional indexes may be used (based on CSI indexing) as shown in Table 4. Instead of specifying the actual file location as is done in CSI, a list of chunk IDs that overlap with the genomic region of interest may be indicated. Then, from the default index structure, the location of these chunks in the file can be determined. [Table 4]

[0153]

[0176] Instead of specifying actual file locations as is done in CSI, a list of chunk IDs that overlap with the genomic region of interest may be given, and the locations of these chunks within the file can be determined from the default index structure.

[0154]

[0177] Protection: Access control policies can also take different forms depending on the use case. For example, certain users may have access to all data, while others may only have access to coarse resolution data (recall, for example, that different resolutions may be stored in different tables). Similarly, access may be restricted to only certain genomic regions. In this case, chunks may be selected accordingly.

[0155] Single-cell RNAseq expression data

[0178] Single-cell RNAseq expression data may contain a sparse two-dimensional matrix of integer / float expression values. Each row corresponds to a gene and each column corresponds to a barcode representing a cell. In this case, the expression values ​​are stored as a two-dimensional sparse attribute array, while the information related to genes becomes dimension-specific row attributes and the information related to barcodes becomes dimension-specific column attributes. The sparse array may be split into a coordinate stream and a value stream. These are compressed separately, and the row coordinates and column coordinates of each row are differentially encoded before compression. To allow fast random access, the data may be chunked into a fixed number of genes (rows) per chunk. Finally, an additional B-tree index may be used to perform selective queries based on gene id. This index maps gene ids to chunks containing gene ids and the position of the gene id within the chunk.

[0156]

[0179] We applied the method to a dataset consisting of 10,000 brain cells from E18 mice. The sparse array contained approximately 31,000 genes, 6.8 million barcodes, and 40 million integer entries. The proposed method with BSC as the final compression layer reduces the size from 750MB (uncompressed) to 67MB, which is less than half the compressed size when gzip or BSC is applied directly to the original file without splitting columns into attributes or differential encoding of coordinates.

[0157]

[0180] The methods, processes, and / or operations described herein may be performed by code or instructions executed by a computer, processor, controller, or other signal processing device. The code or instructions may be stored on a non-transitory computer-readable medium according to one or more embodiments. Because the algorithms underlying the methods (or the operations of a computer, processor, controller, or other signal processing device) are described in detail, the code or instructions for implementing the operations of the method embodiments may transform a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods described herein.

[0158]

[0181] The processors, compressors, decompressors, managers, selectors, parsers, and other information generating, processing, and computation features of the embodiments disclosed herein may be implemented, for example, in logic that may include hardware, software, or both. When implemented at least partially in hardware, the processors, compressors, decompressors, managers, selectors, parsers, and other information generating, processing, and computation features may be, for example, any of a variety of integrated circuits, such as, but not limited to, an application specific integrated circuit, a field programmable gate array, a combination of logic gates, a system on a chip, a microprocessor, or another type of processing or control circuitry.

[0159]

[0182] When implemented at least partially in software, the processor, compressor, decompressor, manager, selector, parser, and other information generating, processing, and computational features may include memory or other storage devices for storing code or instructions executed by, for example, a computer, processor, microprocessor, controller, or other signal processing device. Because the algorithms underlying the methods (or the operation of a computer, processor, microprocessor, controller, or other signal processing device) have been described in detail, the code or instructions for implementing the operations of embodiments of the methods may be transformed into a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods described herein.

[0160]

[0183] From the above description, it will be apparent that various exemplary embodiments of the present invention may be implemented in hardware or firmware. Also, various exemplary embodiments may be implemented as instructions stored on a machine-readable storage medium. These instructions may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a machine-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, and similar storage media.

[0161]

[0184] Those skilled in the art will appreciate that the block diagrams presented herein are conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like, are tangibly represented in a machine-readable medium and thus represent various processes that may be performed by a computer or processor (whether or not a computer or processor is explicitly depicted).

[0162]

[0185] Although various exemplary embodiments have been described in detail with particular reference to certain exemplary aspects of those exemplary embodiments, it should be understood that the invention is capable of other embodiments and its details can be modified in various obvious respects. One or more embodiments may be combined with one or more other embodiments to form new embodiments. As will be readily apparent to those skilled in the art, variations and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the foregoing disclosure, description, and drawings are for illustrative purposes only and do not in any way limit the invention, which is defined solely by the claims.

[0163]

[0186] According to one or more embodiments, a system and method for controlling selective compression and decompression of genome annotation data includes processing information from one of multiple incompatible file formats into a unified file that controls selective compression and decompression. The annotation data is processed and information is extracted and further processed to support features such as fast queries, random access, multiple resolutions (zooms), selective encryption, authentication, access control, and traceability. Processing may also be performed to allow significant compression gains by isolating different attributes of the data and allowing specialized compressors to be used for these. Additional processing may be performed to support metadata and linkage to sequencing data related to annotations, as well as linkage to other annotation data from the same study. This allows seamless integration with existing MPEG-G file formats for sequencing data.

[0164]

[0187] In one or more embodiments, additional processing may be performed to generate protection (access control) information at multiple levels in the hierarchy, allowing fine-grained security settings. Similarly, metadata and attributes allow for effective linking of multiple different types of annotation data and sequencing datasets. The files generated by the processing may be used as standalone files or as part of MPEG-G files. Also, the generation of such files, especially for genomic annotation data, provides sufficient flexibility to achieve state-of-the-art compression performance for various data types by incorporating compression techniques appropriate to the attributes of interest.

Claims

1. 1. A method for controlling compression of data, the method comprising: accessing tabular data in a first file format of a plurality of first file formats; extracting a plurality of attributes from the tabular data; splitting the tabular data into a number of chunks; processing the extracted attributes and the chunks into correlated information; selecting different compressors and dependency attributes for the attributes and chunks identified in the correlated information; storing global data for the compressor that is shared between the chunks; generating a file in a second file format including the correlated information and information indicating the different compressors for the chunks and the attributes indicated in the correlated information; wherein the first file formats are incompatible with one another and differ from the second file format, and the information indicative of the different compressors is processed into the second file format to enable selective expansion of the attributes and chunks indicated in the correlated information.

2. The correlated information includes: First information indicative of one or more of the attributes; and second information corresponding to chunks associated with the one or more attributes indicated in the first information; The method of claim 1 including at least one table.

3. The method of claim 2 , wherein the first information comprises a two-dimensional cell array, each cell identifying one or more corresponding attributes contained within the chunk that corresponds to the two-dimensional cell array.

4. The method of claim 3 , wherein the first information comprises at least one one-dimensional table of dimension-specific attributes associated with the chunk.

5. The method of claim 1 , wherein at least one of the different compressors is embedded in the file.

6. The method of claim 1 , wherein at least one of the different compressors encodes one or more attributes of the tabular data as a sparse array.

7. The method of claim 1 , wherein the tabular data is divided into chunks of different sizes.

8. The method further comprises: generating one or more attribute-specific indexes; and incorporating the one or more attribute-specific indexes into the file of the second format, the attribute-specific indexes enabling identification of chunks of the tabular data that contain values ​​or ranges of particular attributes.

9. The method further comprises:

2. The method of claim 1, further comprising the step of: merging said files of said second file format into an MPEG-G file.

10. The method further comprises: generating access control policy information for the correlated information; and integrating the access control policy information into the file of the second file format; 2. The method of claim 1, wherein the access control policy information includes first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level.

11. 1. An apparatus for controlling compression of data, the apparatus comprising: a memory configured to store instructions; and at least one processor configured to execute the instructions to perform operations, the operations including: accessing tabular data in a first file format of a plurality of first file formats; extracting a plurality of attributes from the tabular data; splitting the tabular data into a number of chunks; processing the extracted attributes and the chunks into correlated information; selecting different compressors and dependency attributes for the attributes and chunks identified in the correlated information; storing global data for the compressor that is shared between the chunks; generating a file in a second file format including the correlated information and information indicating the different compressors for the chunks and the attributes indicated in the correlated information; the plurality of first file formats are incompatible with one another and differ from the second file format, and the information indicative of the different compressors is processed into the second file format to enable selective expansion of the attributes and the chunks indicated in the correlated information.

12. The correlated information includes: First information indicative of one or more of the attributes; and The apparatus of claim 11 , further comprising at least one table including second information corresponding to chunks associated with the one or more attributes indicated in the first information.

13. The apparatus of claim 12 , wherein the first information comprises a two-dimensional cell array, each cell identifying one or more corresponding attributes contained within the chunk that corresponds to the two-dimensional cell array.

14. The apparatus of claim 13 , wherein the first information comprises at least one one-dimensional table of dimension-specific attributes associated with the chunks.

15. The method of claim 1 , wherein at least one of the different compressors is embedded in the file.

16. The method of claim 1 , wherein at least one of the different compressors encodes one or more attributes of the tabular data as a sparse array.

17. The apparatus of claim 11 , wherein the tabular data is divided into chunks of different sizes.

18. The at least one processor executes the instructions to: generating one or more attribute-specific indexes; 12. The apparatus of claim 11, configured to incorporate the one or more attribute-specific indexes into the file of the second format, the attribute-specific indexes enabling identification of chunks of the tabular data that contain values ​​or ranges of particular attributes.

19. 12. The apparatus of claim 11, wherein the at least one processor is configured to execute the instructions to merge the files of the second file format into an MPEG-G file.

20. The at least one processor executes the instructions to: generating access control policy information for the correlated information; configured to integrate the access control policy information into the file in the second file format; 12. The apparatus of claim 11, wherein the access control policy information includes first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level.

Images