Systems and methods for efficient compression, representation, and decompression of diverse tabular data - Patents.com

Abstract

A method for controlling data compression includes accessing genome annotation data in one first file format of a plurality of first file formats, extracting a plurality of attributes from the genome annotation data, dividing the genome annotation data into a plurality of chunks, and processing the extracted attributes and chunks into correlated information. The method also includes selecting different compressors for attributes and chunks identified in the correlated information, and generating a file in a second file format that includes the correlated information and information indicating the different compressors for the chunks and attributes indicated in the correlated information. The information indicating the different compressors is processed into the second file format to enable selective expansion of the attributes and chunks indicated in the correlated information.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001]

[0001] This application is related to U.S. Provisional Patent Application No. 62 / 923,141, filed October 18, 2019, the entire contents of which are incorporated herein by reference for all purposes.

[0002]

[0002] This application is related to U.S. Provisional Patent Application No. 62 / 923,113, filed October 18, 2019, the entire contents of which are incorporated herein by reference for all purposes.

[0003]

[0003] This application is related to U.S. Provisional Patent Application No. 62 / 956,941 (Attorney Docket No. 2019P00831US01), "Customizable Delimited Text Compression Framework," filed concurrently herewith, the entire contents of which are incorporated herein by reference for all purposes. [Technical Field]

[0004]

[0004] One or more embodiments herein relate to managing the compression and decompression of information such as, but not limited to, tabular data and delimited text files. [Background technology]

[0005]

[0005] Genomic data is known to be important for many research applications. Genomic data in raw format can contain a large number of sequenced genomes. In some cases, the number of sequences can be unmanageably large. Therefore, various attempts have been made to extract relevant information from or better interpret the raw data.

[0006]

[0006] One processing technique involves the use of annotation tools. Such tools can be used to define coding regions and their corresponding locations within genomic data. Annotation can also provide an indication of the number and spatial distribution of repetitive regions, which may convey information about disease or genetic abnormalities. Examples of genomic annotation data include mapping statistics, quantitative browser tracks, variants, genomic functional annotations, gene expression data, and Hi-C contact matrices.

[0007]

[0007] Annotation data (and other genome-related information) are currently represented in a variety of file formats, such as the Variant Call Format (VCF), Browser Extensible Data (BED), and Wiggle (WIG) formats. These formats are not compatible with each other. As a result, the use of these formats is fraught with interoperability issues and requires frequent format conversions to enable data visualization. Furthermore, the lack of a single, unified, and standardized format has hindered efforts at compression algorithms, leading to the widespread use of suboptimal compression algorithms (e.g., gzip).

[0008]

[0008] Shortcomings associated with existing algorithms for compressing genomic information relate, among other things, to a lack of selectivity. For example, genome annotation data typically contains multiple fields (attributes) with different statistical properties. Existing algorithms compress all fields together, preventing them from selectively recognizing, using, and extracting information within individual data fields. Additionally, existing algorithms do not allow for the extraction of specific fields without also expanding all attributes.

[0009]

[0009] Other drawbacks relate to the lack of general applicability of existing compression methods. For example, these methods rely on only one standard compressor or a small set of standard compressors, and are therefore only applicable to one type of annotation data. These methods also cannot perform selective encryption, nor can they link multiple annotation datasets to each other and to sequencing data. Several specialized methods have been proposed. However, many of these methods are based on disk-based array management tools (e.g., TileDB and HDF5), which lack high-level features such as, but not limited to, metadata, linkage, and attribute-specific indexing. Summary of the Invention

[0010]

[0010] A brief summary of various exemplary embodiments is presented below. In the following summary, some simplifications and omissions may be made, but this is intended to highlight and introduce some aspects of various exemplary embodiments and is not intended to limit the scope of the present invention. A detailed description of the exemplary embodiments sufficient to enable those skilled in the art to make and use the embodiments is provided below.

[0011]

[0011] According to one or more embodiments, a method for controlling compression of data includes the steps of accessing genome annotation data in one first file format of a plurality of first file formats, extracting a plurality of attributes from the genome annotation data, dividing the genome annotation data into a plurality of chunks, processing the extracted attributes and chunks into correlated information, selecting different compressors for the attributes and chunks identified in the correlated information, and generating a file in a second file format containing the correlated information and information indicating the different compressors for the chunks and attributes indicated in the correlated information, wherein the plurality of first file formats are incompatible with each other and differ from the second file format, and the information indicating the different compressors is processed into the second file format to enable selective expansion of the attributes and chunks indicated in the correlated information.

[0012]

[0012] The correlated information may include at least one table including first information indicating one or more of the attributes and second information corresponding to chunks associated with one or more attributes indicated in the first information. The first information may include a two-dimensional cell array, each cell identifying one or more corresponding attributes included in a chunk corresponding to the two-dimensional cell array. The first information may include at least one one-dimensional table of dimension-specific attributes associated with the chunks. The genome annotation data may be divided into chunks of equal size. The genome annotation data may be divided into chunks of different sizes.

[0013] The method includes generating one or more first indexes linking the correlated information and generating one or more second indexes linking the correlated information to information indicating different compressors, wherein the one or more first indexes and the one or more second indexes are processed into a file in a second file format. The method may include assembling the file in the second file format into an MPEG-G file.

[0014] The method includes generating access control policy information for the correlated information and integrating the access control policy information into a file in a second file format, the access control policy information including first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level. The extracted attributes may include at least one of chromosome attributes and genome location attributes.

[0015] According to one or more embodiments, an apparatus for controlling compression of data comprises a memory configured to store instructions and at least one processor configured to execute the instructions to perform operations, the operations including: accessing genome annotation data in one of a plurality of first file formats, extracting attributes from the genome annotation data, dividing the genome annotation data into chunks, processing the extracted attributes and chunks into correlated information, selecting different compressors for the attributes and chunks identified in the correlated information, and generating a file in a second file format that includes the correlated information and information indicating different compressors for the chunks and attributes indicated in the correlated information. The plurality of first file formats are incompatible with each other and are different from the second file format. The information indicating the different compressors is processed into the second file format to enable selective decompression of the attributes and chunks indicated in the correlated information.

[0016]

[0016] The correlated information may include at least one table including first information indicating one or more of the attributes and second information corresponding to chunks associated with one or more attributes indicated in the first information. The first information includes a two-dimensional cell array, each cell identifying one or more corresponding attributes included in a chunk corresponding to the two-dimensional cell array. The first information may include at least one one-dimensional table of dimension-specific attributes associated with the chunks. The genome annotation data may be divided into chunks of equal size. The genome annotation data may be divided into chunks of different sizes.

[0017]

[0017] At least one processor is configured to execute instructions to generate one or more first indexes linking the correlated information and generate one or more second indexes linking the correlated information to information indicative of different compressors, and the one or more first indexes and the one or more second indexes are processed into a file in a second file format. The at least one processor is configured to execute instructions to aggregate the file in the second file format into an MPEG-G file.

[0018]

[0018] At least one processor is configured to execute instructions to generate access control policy information for the correlated information and integrate the access control policy information into a file in a second file format, the access control policy information including first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level. The extracted attributes may include at least one of chromosome attributes and genomic location attributes. [Brief explanation of the drawings]

[0019]

[0019] In the accompanying drawings, like reference numerals refer to identical or functionally similar elements throughout the various views. The accompanying drawings, together with the following detailed description, are incorporated into and form a part of the specification. The accompanying drawings serve to further explain exemplary embodiments of the concepts recited in the claims and illustrate various principles and advantages of those embodiments.

[0020]

[0020] The above and other more detailed and specific features are more fully disclosed in the following specification, taken in conjunction with the accompanying drawings. [Figure 1]

[0021] FIG. 1 illustrates an embodiment of a system for controlling the compression and decompression of genome annotation data. [Figure 2]

[0022] Figure 2 shows a method for controlling the compression and decompression of genome annotation data. [Figure 3]

[0023] FIG. 3 shows an embodiment of a unified and standardized file format for genome annotation data. [Figure 4]

[0024] FIG. 4 illustrates an embodiment for processing information into annotation files for genome annotation data. [Figure 5]

[0025] FIG. 5 illustrates an embodiment for processing information in the file header of a unified and standardized file. [Figure 6]

[0026] FIG. 6 illustrates an embodiment for processing information for a unified and standardized file. [Figure 7]

[0027] FIG. 7 illustrates an embodiment for processing compressed information for a consolidated and standardized file. [Figure 8]

[0028] FIG. 8 shows an embodiment for processing information for a one-dimensional array. [Figure 9]

[0029] FIG. 9 shows an embodiment for processing information into a two-dimensional array, along with arrays for dimension-specific attributes. [Figure 10]

[0030] FIG. 10 illustrates an embodiment for processing attribute information indexed for genome annotation data. [Figure 11]

[0031] FIG. 11 illustrates an embodiment for processing additional attribute-related information. [Figure 12]

[0032] FIG. 12 shows an embodiment for processing attribute information. [Figure 13]

[0033] FIG. 13 illustrates an embodiment for selectively compressing chunks of data. [Figure 14]

[0034] FIG. 14 illustrates an embodiment for processing compressed file index information. [Figure 15]

[0035] FIG. 15 illustrates an embodiment for processing index information. [Figure 16]

[0036] FIG. 16 shows an embodiment for processing chunks. [Figure 17]

[0037] FIG. 17 illustrates an embodiment for generating fixed-size chunks. [Figure 18]

[0038] FIG. 18 illustrates an embodiment for generating chunks of variable size. [Figure 19]

[0039] FIG. 19 illustrates an embodiment for processing chunk and attribute information. [Figure 20]

[0040] FIG. 20 illustrates an embodiment for processing chunk and attribute information. [Figure 21]

[0041] FIG. 21 illustrates an embodiment for processing a data payload. [Figure 22]

[0042] FIG. 22 illustrates an embodiment for selectively deploying genome annotation data in a unified and standardized format. [Figures 23A-23B]

[0043] 23A and 23B illustrate a method for converting a file format into a folder hierarchy according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0021]

[0044] It should be understood that the drawings are merely schematic and are not drawn to scale, and that the same reference numerals will be used throughout the different drawings to denote the same or similar parts.

[0022]

[0045] The description and drawings illustrate the principles of various exemplary embodiments. Thus, it will be appreciated that those skilled in the art will be able to devise various configurations that embody the principles of the present invention and are within the scope of the present invention, even though not explicitly described or shown herein. Furthermore, all examples described herein are expressly intended primarily for instructional purposes to aid the reader in understanding the principles of the present invention and the concepts provided by the inventors for the advancement of technology, and should not be construed as being limited to such specifically stated examples and conditions. Furthermore, the terms "or" and "or" as used herein refer to a non-exclusive disjunction (i.e., and / or) unless otherwise specified (e.g., "otherwise" or "or instead"). Furthermore, various exemplary embodiments are not necessarily mutually exclusive, and some exemplary embodiments may be combined with one or more other exemplary embodiments to form new exemplary embodiments. Descriptive terms such as "first," "second," and "third" are not intended to limit the order of elements discussed but are used to distinguish one element from the next and are generally interchangeable. Values ​​such as maximum or minimum values ​​may be predetermined and may be set to different values ​​based on the application.

[0023]

[0046] Exemplary embodiments describe systems and methods for managing compression of information, such as, but not limited to, tabular data and delimited text files. The information to be compressed may include genomic information (e.g., annotation, sequencing, or other forms of genome-related data). In other embodiments, other types of information unrelated to genomic information may be managed for compression.

[0024]

[0047] In genomic applications, systems and methods may manage the standardization of genomic data compression and decompression. This includes processing the data to conform to formats that reduce or eliminate issues related to interoperability and the need for frequent format conversions to make the data visible. In at least some applications, these features may result in improved and optimized compression algorithms for specific types of data.

[0025]

[0048] According to one or more embodiments, systems and methods may process data to enable selective compression of different fields, attributes, and / or sections of data. For example, in some implementations, all fields, attributes, or sections of data (including, e.g., those with different statistical characteristics) may not all be compressed using the same compression algorithm. Instead, systems and methods may be implemented to selectively compress one or more sections of data while not compressing other sections, or to selectively compress different sections using different compression techniques. The selected compression technique may, for example, be beneficial or optimal for different types of data within corresponding respective ones of the multiple sections.

[0026]

[0049] This selective compression technique may, in turn, enable selective decompression. For example, by selectively compressing data, the entire data file need not be decompressed. Instead, in at least one implementation, one or more sections of data of interest may be selectively decompressed without decompressing other sections of compressed data that are not of interest. This improves efficiency and the ability to avoid burdens and other delays during research.

[0027]

[0050] The processing performed by the systems and methods may also enable at least one embodiment to be generally applicable to all types of data, e.g., using multiple standard or customized compression algorithms selectively applied to different data sections. The systems and methods may also perform selective encryption of data sections that have been compressed (e.g., in different ways). The systems and methods may also be implemented to link multiple annotation datasets with each other and with sequencing data. The systems and methods may also process data to include metadata, linkage, and / or attribute-specific indexing.

[0028]

[0051] 1 illustrates an embodiment of a system for managing information compression. The system may be implemented, at least in part, in, for example, a laboratory workstation, a research facility, a data center, a university or corporate environment, or another location where data is processed in connection with an intended application. In another embodiment, the system may be implemented on one computer and / or other device, or multiple connected or networked computers and / or other devices that communicate with each other to perform the processing operations described herein.

[0029]

[0052] Referring to FIG. 1 , the system includes a processor 110, a memory 120, and a storage area 130. The processor 110 executes instructions stored in the memory 120 to perform operations, including managing the compression of information. The memory 120 may be any form of non-transitory computer-readable medium that stores instructions for controlling the processor to perform the operations described herein. The information may be received from a data source 140. The data source may vary, for example, by the type of information being processed. For example, in one implementation, the data source 140 may be a software-based tool for generating genome annotation data. In another implementation, the data source may provide another type of genome-related data, or possibly data unrelated to genome data.

[0030]

[0053] The processor 110 may be connected to the data source locally or over a network. In the latter case, the information processed by the system may be obtained from anywhere in the world, provided a network connection can be established between the processor and the data source. The network connection may be a local connection, an Internet connection, a virtual private network connection, a connection to a cloud computing or cloud-based storage device, and / or another type of connection.

[0031]

[0054] The storage area 130 may store the processed results of the processor 110. As described in more detail below, the processed results may include tabular data or delimited text files conforming to a unified and standardized file format that includes multiple fields, attributes, sections, or other portions that may be selectively compressed and / or otherwise manipulated. The storage area may be or be included in a database, an archive, a storage area network, a cloud-based storage system, memory included in or coupled to the processor's computer, system, or device, or another type of storage area.

[0032]

[0055] 1 may include or be coupled to one or more additional features for compressing the processed information into a unified and standardized file format. These features may include a parser 150, a compression manager 160, and an aggregator 170.

[0033]

[0056] Parser 150 may parse files that have been stored and later retrieved from storage area 130. For example, processor 110 may receive a request for data based on user input or from another device. The processor may, for example, determine one or more files stored in storage area 130 that correspond to the requested data and retrieve the determined one or more files. Processor 110 may, for example, determine which files correspond to the requested data using an index that links the contents of the files to one or more identifiers specified in the request. If a file is found, it is retrieved from storage area 130 and sent to the parser.

[0034]

[0057] The parser may parse the file in various ways. For example, the file may contain delimiters that separate different sections, parts, attributes, fields, etc. that contain different pieces of data (or other types of content) within the file. Using these delimiters as a guide, the parser 150 may parse the file into individual sections (or chunks) that correspond to the request. In one embodiment, the parser may perform these operations according to a predetermined compression scheme that indicates how the file is to be parsed for compression.

[0035]

[0058] The compression manager 160 receives files in a unified standard format when the files are retrieved from the storage area 130. The compression manager comprises multiple compressors 1611, 1612, ..., 161 Nwhere N≧2. Each compressor may implement a different compression algorithm to compress different sections of the parsed file. The compression algorithm applied by each compressor may be predetermined based on the parsed data or content of the file. For example, one compression algorithm may be better suited to compressing certain types of information, for efficiency reasons, or other reasons. Other compression algorithms may be better suited to compressing other types of information. The compression scheme may control which compression algorithm is used to compress particular data or content within the file. (The parser may also route different parsed sections of the file to particular compressors based on instructions derived from the compression scheme.) Thus, in this manner, compression manager 160 can selectively compress different parsed portions of the file.

[0036]

[0059] Aggregator 170 aggregates or combines multiple different compressed sections into a compressed file according to a unified and standardized file format for the compressed file, or according to different formats. After compression, the compressed file (e.g., a compressed tabular data file or a compressed delimited text file) may optionally be stored in storage area 130, stored in a different storage area, sent to a predetermined location or device, or routed according to another scheme, such as a scheme determined by predetermined control instructions for the system.

[0037] Uniform / standardized data processing

[0060] According to one or more embodiments, the processor 110 processes information from the data sources to conform to a unified and / or standardized file format that allows for selective compression (and therefore subsequent selective decompression) of different portions of the file. The processing may be performed to allow for storage of the genome annotation data in a file format that allows for support of one or more predetermined features. Examples of these features include fast querying, random access, multiple resolutions (e.g., zooming), selective encryption, authentication, access control, and traceability.

[0038]

[0061] In one embodiment, the genomic application data is processed into a format that allows for significant compression gains. This may be achieved, for example, by separating different attributes of the data into multiple sections (e.g., multiple tables). Multiple compressors may then be used to compress each of the separated attributes in the genomic annotation data. In one embodiment, all or some of the compressors may be specialized or customized compressors designed to compress specific attributes of the files assigned to them. The processing performed by processor 110 may also generate metadata and linkage for the sequencing data associated with the annotations, as well as linkage to other annotation data that may or may not be related to the sequencing data. The metadata and / or linkage may be stored in the same unified and standardized format as the genomic application data, allowing for seamless integration with existing file formats for sequencing data (e.g., MPEG-G).

[0039]

[0062]

[0013] Figure 2 illustrates a method for managing compression of information according to a unified and standardized file format. The method may be performed, for example, by the system of Figure 1 or another system according to instructions stored in a memory for controlling one or more processors to perform the operations described below. For purposes of explanation, it is assumed that the method is performed in part or in whole by the system of Figure 1 and that the data is genome annotation data.

[0040]

[0063] 2, the method includes, at 210, receiving data (e.g., genome annotation data) from a data source 140. The genome annotation data may be received in an existing format, such as generated by an annotation tool used to generate the data. One example of an existing format is MPEG-G format, although in another embodiment, the annotation data may be received in a different existing format. The data may be received by the processor 110 from the data source via a local connection or a network connection.

[0041]

[0064] At 220, portions of the genome annotation data are retrieved for processing into a unified and standardized file format. Because the genome annotation data is organized according to an existing known format, the content retrieved in each portion of the known format has a particular type of data known to processor 110. Thus, instructions controlling processor 110 can retrieve the particular type of data in each portion of the known format to generate a new file in a unified and standardized format according to one or more embodiments.

[0042]

[0065] At 230, processor 110 executes instructions in memory 120 to process the genome annotation data to conform to a unified standard file format 300, for example, as shown in Figure 3. For example, the processor may process the data to generate a file header section 310, a file protection section 320, a file metadata section 330, a file traceability section 340, a table information and index section 350, a section 360 indicating one or more compression parameters, and one or more tables 370. The chart in Figure 4 illustrates these sections and corresponding descriptions, along with other information that will be described in more detail below.

[0043]

[0066] The file header section 310 includes various types of identifying information. For example, the file header section may include a file name field 311, a file type field 312, and a file version field 313 (see, e.g., FIG. 5). The file name field may include a string that names the file, for example, generated by an annotation tool or based on information from a user. The file type field may include a string that indicates the type of file (e.g., variant, gene expression, etc.). The file version field may include a string that identifies the purpose and / or version of the file (e.g., this file is for recording updates).

[0044]

[0067] The file protection section 320 contains information related to the file's access control policy, which may be determined, for example, by a user, administrator, data owner, or other entity managing access to and / or distribution of data. The access control policy may indicate the same level of access for the entire file, or different levels of access for different portions of the file. In one embodiment, the file protection section may include a payload size field 321 and a payload field 322 (see, for example, FIG. 6). The payload size field may include one or more integer values ​​indicating the size of the protection information payload, which may allow this field or its associated information to be skipped or taken into account, for example. The payload field contains information about the file's protection and access control policy.

[0045]

[0068] The file metadata section 330 and the file traceability section 340 may include information as described in more detail below. In one embodiment, the file protection information, metadata, versioning, and traceability information may be compressed using, for example, a common compression algorithm, such as 7zip. In some implementations, a JSON / XML / XACML-based schema may be used to compress this information along with a uniform resource identifier (URI) notation (e.g., as done in MPEG-G part 3).

[0046]

[0069] The table information and index section 350 may include an nTables field 351 that indicates the number of tables (n) in the file. In one embodiment, the unified and standardized format may store annotation data in multiple tables (e.g., n≧2). In this case, different tables in the multiple tables may store various types of annotated data. In some cases, different tables in the file may be used to store the same genome annotation data at different resolutions. This is provided only as an example of table content. In other embodiments, tables may store other types of annotation data.

[0047]

[0070] In addition to the nTables field 351, the table information and index section may store a Table ID[i] section 352, a TableInfo[i] section 353, and a ByteOffset[i] section 354. The Table ID[i] section 352 may store one or more integer values ​​indicating a unique identifier for the corresponding table i of the n tables in the file. The TableInfo[i] section 353 may store information indicating the resolution of the corresponding table i of the n tables. The ByteOffset[i] section 354 stores one or more integer values ​​indicating the byte offset of each of the tables i of the n tables in the file. Using this format, basic information (e.g., resolution level) about one or more of the tables in the file may be extracted without reading the entire file, using the TableInfo field in a format such as JSON / XML. Similarly, the byte offset of a table in the compressed file may be available to jump directly to a specific table.

[0048]

[0071] At 240, the genomic data in the received (source) file is divided into chunks or chunk groups. The chunks may have the same fixed size or may have variable sizes depending, for example, on whether the chunks are attribute-dependent and / or based on other parameters associated with the data. The chunks may be generated, for example, by parser 150, as described above.

[0049]

[0072] Different compressors (or compression algorithms) are specified to compress the divided chunks or chunk groups at 250. The compressors may be external to the unified and standardized file and / or may be initiated internally, for example by embedded code or by a selectable option stored in a compression parameter table.

[0050]

[0073] Thus, compression parameters section 360 may include field 361 that indicates the number of different compressors (or compression algorithms) used to compress different information (e.g., attributes) contained within the table of the file. This information may be represented, for example, as one or more integer values. Section 360 may also include field 362 that contains information listing different compressors indexed by unique identifiers, which can be referenced within the table, thus avoiding repeated descriptions of compressors used in multiple tables or for multiple attributes.

[0051]

[0074] In one embodiment, as shown in FIG. 7, field 362 (or fields separate from or hierarchically related to field 362 in compression parameters section 360) may include a CompressorID field 363, an nDependencies field 364, a CompressorNameList field 365, and a CompressorParametersList field 366.

[0052]

[0075] The CompressorID field 363 may contain information (e.g., a string) that indicates a unique identifier for the corresponding compressor among multiple compressors. Thus, the CompressorID field may be used to identify the compressor used to compress each attribute or other portion of the file. In one embodiment, the unique identifier may be used in a table to point to the corresponding compressor.

[0053]

[0076] The nDependencies field 364 may contain information (e.g., one or more integer values) indicating that compression of an attribute is performed based on other attributes. For example, a system processor may process information for compressing one attribute using one or more other attributes as side information (provided there are no circular dependencies). In that case, this information may be included in the nDependencies field 364 of the unified and standardized file format. In one embodiment, the index of the nDependencies variable may indicate the number of dependency attributes used for decompression (e.g., the attribute information structure in FIG. 12 shows an example of the corresponding attributeID). Compressors such as context-based arithmetic coding may easily support the incorporation of side information. Another mechanism for incorporating side information from other attributes is to sort or split the values ​​of the current attribute based on the values ​​of other attributes. This allows similar values ​​to be grouped together to provide better compression. The parameter describes the dependency used by each compressor in the list.

[0054]

[0077] The CompressorNameList field 365 may contain information (e.g., a string) indicating a list of compressor names. The compressor names may refer to one or more compressors external to the file (e.g., implemented by the compression module or the system processor 110), one or more compressors embedded in the file (e.g., via embedded or executable code), or both. One or more of the compressor names and parameters may correspond to standard compressors or describe a decompression mechanism. In one embodiment, one or more of the compressor executable code and parameters may be indicated in CompressorParameters, for example, by setting CompressorName to “EMBEDDED.” In some embodiments, multiple compressors may be used to compress the same attribute or file portion. In this case, multiple compressors may be applied in sequence and appear in a list included in a unified and standardized file format.

[0055]

[0078] The CompressorParametersList field 366 may contain information (e.g., in a list or other format) that indicates one or more parameters that should be used to perform decompression of information compressed by all or part of a specified compressor.

[0056]

[0079] At 260, the method includes processing the received (source) file and / or policy information generated by the data owner or custodian to generate additional information including metadata, traceability information, linkage information, indexes, and protection information that is stored in corresponding fields as described herein.

[0057]

[0080] At 270, the unified and standardized genome annotation data file is saved for later retrieval. Because multiple different data chunks have been selectively compressed, only designated portions of the compressed file may be extracted to access specific information within the file without having to extract other information within the file. This improves processing speed and user convenience by targeting only portions of the genome data that may be of particular interest at any given time.

[0058]

[0081] All or part of the information specified in the unified and standardized file may correspond to correlated information set forth in tables, structures, indexes, or other types of correlated information that enable selective compression and expansion of, and access to, genome annotation data. Examples of tables within correlated information are described in more detail below.

[0059] table

[0082] The tables 370 in each unified and standardized file may be related to or independent of each other in terms of the attributes and / or other information stored. Each table contains multiple cells, and each cell may contain one or more attributes. Each attribute has a specific data type and may be compressed using one of several compressors. If a cell in a table contains multiple attributes, the data corresponding to the multiple attributes in each cell of the table may be compressed separately to improve compression and enable selective access to the attributes during subsequent decompression. In a genome functional annotation file, attributes may include, for example, a chromosome, a start position, an end position, a feature ID, and a feature name.

[0060]

[0083] Compression may be performed in different ways in different embodiments, e.g., per attribute, per cell, per table, etc. For example, for these purposes, groups of cells within the same table may be compressed separately from other groups of cells. Cells grouped together may have attributes corresponding to the same data, while other groups of cells may have different attributes corresponding to the same or different data. In one implementation, attributes within each cell may be compressed using different compression algorithms, or attributes within one or more cells may be compressed with a first compression algorithm, attributes within one or more other cells may be compressed using a second compression algorithm, attributes of one or more other cells may be compressed using a third compression algorithm, and so on.

[0061]

[0084] In some additional implementations, all or some of the compressors 1611, 1612, ..., 161N may execute different compression algorithms specialized for different data / attribute types within each of the multiple cells. Although an example of a unified and standardized file format is discussed below as including multiple tables, in one embodiment, the file format may include only one table.

[0062]

[0085] In some examples, the content within each cell is described as an attribute, but the content may include other types of data (e.g., types of data, attributes, characteristics, etc.) or combinations of multiple types of data in other embodiments. For example, tables 370 within one file may all store content or data representing the same genome sequence, but each table within the sequence may represent that content or data at a different resolution.

[0063]

[0086] The system processor 110 can process and store the information in the file tables 370 in various ways. For example, processing can be performed to generate one-dimensional tables and / or multidimensional tables. All tables in a given standardized file can be of the same type or can contain a combination of types, such as both one-dimensional and multidimensional tables. In genome annotation applications, one-dimensional tables can contain, for example, genome annotation data or quantitative browser tracks, while multidimensional tables can contain, for example, variant call data and gene expression data for one or more samples. In one embodiment, one-dimensional tables can contain multiple attributes.

[0064]

[0087] 8 shows an example of a table that stores data in a one-dimensional array 810 of cells. Each cell 820 in the array contains multiple attributes, labeled Attribute 1, Attribute 2, and Attribute 3. Each cell in the table has the same number of attributes, although in some embodiments, the number of attributes in the cells of the table may vary. Example table structures and examples of how a system processor may process content for inclusion in the table structure are described in more detail below.

[0065]

[0088] 9 shows an example of a multi-dimensional table that stores data in a two-dimensional array of cells 910. Each cell 920 in the table contains one or more attributes that correspond to the data. The number of attributes in a cell may be the same or different. Example table structures and examples of how a system processor may process content for inclusion in the table structure are described in more detail below.

[0066]

[0089] In one embodiment, one or more of the multidimensional tables 370 in each file may store dimension-specific attributes in addition to the two-dimensional array 910. An exemplary configuration for storing dimension-specific attributes is also shown in FIG. 9. Here, in addition to the attributes stored in array 910, table 370 stores dimension-specific attributes in two additional tables. A first array 940 of dimension-specific attributes contains cells that store row attributes. A second array 950 of dimension-specific attributes contains cells that store column attributes. Thus, each array 940 and 950 can be considered part of an additional one-dimensional table within table 370. One example of a table 370 containing these features is intended to represent variant data for multiple samples in two-dimensional array table 910, with sample genotypes and sample-level likelihoods being attributes of each cell in the 2D array. Additionally, array table 940 may store dimension-specific attributes corresponding to variant positions, and array table 950 may store dimension-specific attributes corresponding to sample names. An example of this table structure and an example of how the system processor may process the content for inclusion in the table structure are described in more detail below.

[0067]

[0090] Figure 10 shows an embodiment of how the system processor 110 may process genome annotation data into a particular table format, and Figure 11 shows an embodiment of a table format for the information processed to generate the table.

[0068]

[0091] 10 and 11 , the tabular format includes a number of sections similar to those of unified and standardized file formats. For example, the tabular format may include a table header section 1010, a table protection section 1020, and a table metadata section 1030. The table header section 1010 may include a TableID field 1105 that indicates the name of the table and a TableInfo field 1110 that indicates the payload (data) type and / or other information. The table protection section 1020 may include information that indicates the access control policy for the table. The table metadata section 1030 may include linkage, traceability, and / or other information.

[0069]

[0092] In addition to these features, the table may include a summary statistics section 1040, an attribute section 1050, an index section 1060, and a data section 1070. The summary statistics section 1040 may include information indicating averages, counts, distributions, and / or other information related to the data / attributes stored in the table, which may correspond to one or more key values. The statistics may be used, for example, to enable fast access to relevant statistics. This information corresponds to reference numeral 1140 in FIG. 11.

[0070]

[0093] Attribute section 1050 stores various attributes for one-dimensional and / or multi-dimensional array tables as described herein, and may or may not have intrinsic dimension attributes. For example, if dimension-specific attributes are included in the table, section 1050 may include field 1150 indicating the size, name, and metadata of each dimension i (i=1, . . . , N, N≧2), as well as the dimension-specific attributes corresponding to each dimension as shown in the table. For two-dimensional data, a SymmetryFlag field may also store information indicating whether the dimensional array is symmetric, e.g., whether it is symmetric Hi-C data.

[0071]

[0094] The table includes an nAttributesMain field 1160 indicating the number of attributes in the main array (or two-dimensional array), followed by attribute information for each attribute. The attribute information may have content and structure described in more detail herein. Field 1160 may also include byte offset information for the attributes / data in the 2D array table. Selective compression of attributes based on different algorithms and the byte offsets corresponding to the attributes may allow selective access to the attributes (specific portions of the genome annotation data) during decompression without having to decompress other attributes, table sections, or other data contained in the unified and standardized file. Index section 1060 stores indexes as described in more detail herein, and data section 1070 may store data, including, for example, chunks or chunk groups as described herein.

[0072]

[0095] FIG. 12 illustrates an embodiment of how the system processor processes genome annotation data to generate attribute information and associated structures that may be included within tables or other sections of the unified and standardized file.

[0073]

[0096] 12, attribute information may be processed into a structure including an AttributeInfoSize field 1205, an Attribute ID field 1210, an AttributeName field 1215, an AttributeMetadata 1220 field, and an Attribute Type field 1225. The AttributeInfoSize field contains information indicating the size of this information, which allows a processor to skip this structure. The Attribute ID field may contain a unique identifier for the attribute. The AttributeName field contains information indicating the name of the attribute, and the Attribute Metadata field may contain information for linking or grouping the attribute with one or more other attributes.

[0074]

[0097] The Attribute Type field 1225 may contain information that identifies the attribute as one of two types: the first type is a base type, which indicates whether the attribute corresponds to a character, a string (null-terminated), a float, a double, a Boolean, or signed and unsigned integers with different bit widths; and the second type is a derived type, which indicates whether the attribute corresponds to a variable-length array or a fixed-length array.

[0075]

[0098] In addition to these features, the attribute information structure may include a DefaultValue field 1230, a Summary Statistics field 1235, and a CompressorID field 1240. The DefaultValue field indicates whether most values ​​of the attribute are equal to a predetermined default value. If so, a predetermined type of encoding (e.g., sparse encoding) may be used to encode at least the corresponding attribute in the file. The Summary Statistics field may contain information indicating the mean, count, distribution, and / or other statistical data that may enable fast analysis of the attribute.

[0076]

[0099] The Compression ID field 1240 contains information indicating the type of compression algorithm to be used for the attribute. This information may, for example, identify one of the compressors 1611, 1612, ..., 161N assigned to compress the attribute. This information controls which decompression algorithm should be used to restore the attribute when retrieved in compressed form by the system processor. If the compressor uses side information / context during the decompression process, corresponding dependency attributes may also be specified in the attribute information. For multidimensional arrays, the side information may be obtained from multidimensional array attributes or from dimension-specific attributes. For example, in a VCF file, variant-specific fields (e.g., dimension-specific attributes) may be used as side information for compressing genotype data (which is an attribute of a two-dimensional array table). In one embodiment, the system processor may process the attribute information to include additional data required for decompression in the variable CompressorCommonData. This additional data may, for example, be common to all chunks. This additional data may be useful for storing codebooks, dictionaries, or statistical models computed from all or part of the genome annotation data file. Chunks and chunk-related processing are described in more detail below.

[0077] Compressor

[0100] Information (e.g., attributes, data, and other information) received by compression manager 160 may be selectively compressed using, for example, different compressors 1611, 1612,..., 161N of the system shown in Figure 1. A non-limiting list of these compressors includes: Run-length compression: This compression algorithm is used, for example, for long runs of the same value and may involve performing a substitution operation based on the value and length of the run. Differential coding: this compression algorithm is used, for example, to increase a sequence of numerical values ​​and may involve a substitution operation based on the difference between successive values. Dictionary-based / enumeration: This compression algorithm may be used, for example, for attributes whose values ​​correspond to a small set of options. In this case, the algorithm may perform a substitution operation based on an index within the set and store the resulting dictionary in the CompressorCommonData field. Sparse: This compression algorithm may be used to compress attributes that differ slightly (e.g., by less than a predetermined value) from their default values, such as those shown in the attribute information of FIG. 12. This type of compression may involve representing the attributes as coordinate locations and values ​​for non-default values. To improve compression, the coordinate locations may further be differentially encoded within each chunk; for example, in a two-dimensional sparse array, row indices may be differentially encoded and column indices may be differentially encoded within each row. Variable length arrays: This compression algorithm may be used, for example, to separate a variable length array into a stream of values ​​and a stream of lengths. Tokenization: This compression algorithm is used, for example, for structured string attributes and may involve splitting the attribute into different types of tokens and encoding each token based on its previous value (e.g., match with previous token, delta, new value, etc.). General-purpose compression / entropy coding methods: These types of compression methods include, for example, gzip, bzip2, 7-zip, adaptive arithmetic coding, and other lossless data compression (e.g., but not limited to, BSC's lossless block sorting compression).

[0078]

[0101] The above list of compression algorithms is not intended to be an exhaustive list. In one embodiment, one or more specialized or customized compression algorithms may be included in addition to or instead of one or more of the above compressors. One example of such a specialized compression algorithm is GTC (genotype compressor), which may be used to compress variant call data. Such compression may support fast random access to the information contained in the rows / columns of the compressed table. In some cases, for example, portions of the file may not be compressed to allow for faster selective access.

[0079]

[0102] One or more of the compressors 1611, 1612, ..., 161N may generate multiple streams. An example is a sparse compressor that generates a stream of coordinates and a stream of values. In one embodiment, these streams may be compressed using, for example, different entropy coders implemented with appropriately specified parameters. For example, consider the following: CompressorNameList=['sparse','gzip','7-zip'] CompressorParameterList=[ {“outStreams”:[“coordinate”,“value”]}, {“inStreams” :[“coordinate”]}, {“inStreams” :[“value”]} ]

[0080]

[0103] In this case, gzip compression may be applied to the coordinate stream and 7-zip to the value stream (e.g., JSON may be used to represent the parameters). This allows the application of a compressor per data stream, which may produce predetermined or optimal results. If no stream is specified, the same compression method may be applied to all received streams.

[0081]

[0104] As described above, in addition to or instead of one or more external compressors, a compressed file (processed into a unified and standardized file format) may include one or more embedded compressors. That is, the system processor may process the genome annotation data to include an embedded compressor executable within the unified and standardized file format, preferably with appropriate security protection. In the case of an embedded compressor, a corresponding unpacking executable code may be included in the compression parameters, along with one or more digital signatures as proof of origin and authenticity to protect against malicious software. For interoperability across different platforms, a standardized virtual machine bytecode may be used for the unpacking executable code.

[0082] Chunking and Indexing

[0105] 13 illustrates an embodiment of a method for processing information organized into a unified and standardized file format, in which compressed files are processed to allow efficient random access to the compressed information during decompression and information recovery.

[0083]

[0106] 13, the method includes, at 1310, dividing attributes, data, (1D and / or 2D) tables, and / or other information in each table 370 (or the entire file) into chunks. The chunks may all have the same fixed size, or may have variable sizes. The same chunks may or may not be used for all attributes. In one embodiment, the attributes in each table 370 are divided into rectangular chunks to allow efficient random access.

[0084]

[0107] At 1320, after division, the chunks are selectively compressed by different ones of a plurality of compressors (eg, as directed by instructions executed by the system processor 110).

[0085]

[0108] Next, at 1330, the processor generates at least one index for each compressed chunk or group of chunks to be included in the unified and standardized format compressed file. The index may include information for efficiently determining the location of each particular chunk within the compressed file. Processing the data into an index may enable fast access to any location within the data by simply selectively decompressing the corresponding chunk without decompressing other chunks that are unrelated or irrelevant to the chunk or chunks of interest. To support fast random access based on the value of a particular attribute, the system processor may also process the data to generate attribute-specific indexes. In one embodiment, each compressed file may also include information that enables sharing of a codebook or statistical model for performing decompression across multiple chunks.

[0086]

[0109] FIG. 14 shows an example of an index structure generated by a processor. This example corresponds to an index structure for a one-dimensional table when the flag AttributeDependentChunks field is false, as described below. The index includes a first section 1410 and a second section 1420. The first section 1410 includes information corresponding to multiple chunks, each of which includes starting index information 1411, ending index information 1412, and byte offset information 1413. The second section 1420 includes multiple additional indexes for attributes included in the chunk or other portions of the associated compressed file. Each additional index may include indexed attribute information 1421, index type information 1422, and index data information 1423. The index file structure may be included, for example, in the index section of each table (e.g., as shown in FIG. 10) or in selected tables of the compressed, consolidated, and normalized file.

[0087]

[0110] FIG. 15 illustrates an example index structure according to one or more embodiments, and FIG. 16 illustrates an example chunk structure that may correspond to and provide a better explanation of the index structure of FIG. 14 according to one or more embodiments.

[0088]

[0111] Referring first to FIG. 16 , the chunk structure may include an nChunks field 1605 and a VariableSizeChunks field 1610. The nChunks field 1605 contains information (e.g., an integer) indicating the number of chunks corresponding to a particular attribute. The VariableSizeChunks field 1610 contains a flag indicating whether the nChunks have the same fixed size or different sizes. While fixed-size chunks are simpler to process (especially when included in a multidimensional array table), variable-size chunks may be useful when the choice of a fixed chunk size is not optimal, for example, because the sparsity of the data varies widely. In some cases, variable-size chunks may be better suited to certain attributes of genome annotation data (e.g., chromosomes, genome locations, etc.) and therefore may enable faster random access for these attributes.

[0089]

[0112] If the chunk structure indicates that variable-sized chunks are being used, then the start index 1615 and end index 1620 of each chunk along each dimension may be included within the chunk structure. This information is also shown in the compressed file index structure of Figure 14. For fixed-sized chunks, the chunk size may be indicated along each dimension. In either case (fixed or variable size), byte offset information 1625 may be included within the file for each chunk to allow random access.

[0090]

[0113] Figure 17 shows an example of a portion of a genome annotation data file associated with a particular attribute. In this example, the data is arranged in a two-dimensional array and divided into chunks 1710 of the same fixed size (e.g., ChunkSize(1) is 5 and ChunkSize(2) is 11) with a particular ByteOffset[j]. The portion of data shown contains a total of 15 chunks (nChunks=15), with the VariableSizeChunks field indicating a false value to specify the fixed size.

[0091]

[0114] FIG. 18 shows an example of a portion of a genome annotation data file related to a particular attribute. In this example, the data is arranged in a two-dimensional array and divided into chunks 1810-1840, each with a variable size. Because the chunk sizes are variable, a start index and an end index are provided for each block, along with an associated ByteOffset. For example, StartIndex[1][1]=1 and EndIndex[1][1]=17 are shown for the first direction of the first rectangular chunk (Chunk 1), and StartIndex[1][2]=1 and EndIndex[1][2]=17 are shown for the second direction of the first rectangular chunk. Corresponding ByteOffset information is also provided for Chunk 1. Similarly, StartIndex[2][1]=1 and EndIndex[2][1]=11 are shown for the first direction of the second rectangular chunk (Chunk 2), and StartIndex[2][2]=18 and EndIndex[2][2]=31 are shown for the second direction of the second rectangular chunk. This data portion as shown includes a total of four chunks (nChunks=4), with the VariableSizeChunks field indicating a true value to specify variable size. The remaining chunks 3 and 4 may be specified similarly. Information corresponding to all four chunks for the attribute may be incorporated into the chunk structure, index structure, corresponding tables, and / or other information associated with a unified and standardized file of genome annotation data.

[0092]

[0115] 15, the index structure includes information including an AttributeDependentChunks field 1510. This field may include a flag indicating whether the chunk size is dependent on one or more attributes, or whether the same chunking (e.g., the same chunk size) is used for all attributes associated with the file. If the flag has a first value, then the same chunking is used for all attributes at 1515. Using the same chunking for all attributes significantly reduces the size of the index structure and may be useful, for example, if all attributes in a chunk are queried most of the time.

[0093]

[0116] 19 shows an example of information included in a data payload structure for the one-dimensional case when the AttributeDependentChunks flag has a first value (e.g., false). In this example, data 1910 is divided into multiple chunks 1920 (e.g., nChunks=5), with each chunk corresponding to one or more attributes 1930. In this case, chunk 1920 corresponds to three attributes. Each attribute also includes information indicating the payload size and the payload.

[0094]

[0117] Referring again to Figure 15, if the flag has a second value, then at 1520, attribute-dependent chunking is used as shown in the chart of Figure 16. Attribute-dependent chunking may be useful, for example, when the optimal chunk sizes of different attributes for compression and random access differ significantly (e.g., above a predetermined value). For example, if some attributes are sparse and other attributes are dense, using the same chunk size may not optimize compression. Attribute-dependent chunking may also be useful when all chunks of a single attribute are queried frequently, for example, above a predetermined frequency value.

[0095]

[0118] Figure 20 shows an example of a data payload structure for the one-dimensional case when the AttributeDependentChunks flag has a second value (e.g., true). In this example, data 2010 corresponds to one or more attributes 2020. Each attribute includes or corresponds to one or more chunks 2030 obtained by dividing the data. Each chunk also includes information indicating the payload size and payload. As is clear from a comparison of Figures 15 and 20, according to one embodiment, the organization of chunks and attributes depends on the mode of operation, for example, whether attribute-dependent chunking is performed or not.

[0096]

[0119] In one embodiment, random access during the selective unfolding process may be performed without using row and / or column numbers. This embodiment may be implemented, for example, for applications where random access is performed with respect to a particular attribute, such as for applications where random access is performed with respect to a genomic location.

[0097]

[0120] In these cases, the index structure of FIG. 15 may include one or more attribute-specific indexes 1525. For example, if attribute-dependent chunking is performed, nAdditionalIndexes may be used (n≧1). To perform faster queries by attribute rather than by row / column, the index structure may include one or more attribute indexes. For example, in the case of genome annotation data, indexes for chromosome and location attributes may be generated by the system processor 110 and included in the index structure of FIG. 15. When a query is performed by the system processor, chunks or chunk numbers that match the chromosome and location attributes of the query may be returned for selective expansion. In other embodiments, different attributes and / or a different number of attributes may be used.

[0098]

[0121] The index structure may also include other information related to the nAdditional index. For example, for each attribute-specific index, the index structure may include an AttributeIDsIndexed field 1530, an IndexType field 1535, an IndexSize field 1540, and an IndexData field 1545. The AttributeIDsIndexed field 1530 may include a list of one or more indexed attributes (e.g., chromosomes, genomic locations, etc.).

[0099]

[0122] The IndexType field 1535 may contain information indicating the type of indexing performed for each attribute listed in the AttributeIDsIndexed field. In one embodiment, the type of indexing for one or more attributes may differ from the standard set. For example, an R-tree or CSI index may be used for each chromosome attribute and genome location attribute, as well as for range queries. A B-tree index may be used for database-type queries. Genomic range indexing can store the coordinates of the left and right edges of each chunk, which allows for fast identification of chunks that overlap with the queried range. Similarly, a B-tree index can store a map from attribute values ​​to chunks that contain the value and the value's location within the chunk.

[0100]

[0123] The IndexSize field 1540 may contain information indicating the size of the index generated by the system processor for each listed attribute.

[0101]

[0124] The IndexData field 1545 may contain the actual indexing data in a predetermined (e.g., binary) format. The type of format may depend, for example, on the type of index created for the attribute.

[0102]

[0125] FIG. 21 illustrates exemplary logic that may be executed by the system processor 110 to generate payload information (e.g., as shown in the data of FIG. 19 and / or FIG. 20). This logic is implemented based on the value in the AttributeDependentChunks field. More specifically, as is evident from a comparison of FIG. 19 and FIG. 20, the structure of the payload differs depending on the value of the AttributeDependentChunks field. If the chunks of data in the payload are attribute-independent, the system processor implements logic to generate the structure shown in FIG. 19, as described above. If the chunks of data in the payload are attribute-dependent, the system processor implements logic to generate the structure shown in FIG. 20, as described above.

[0103]

[0126] FIG. 22 illustrates one embodiment of a query method that may be performed by the system processor 110 to selectively identify and expand annotated genomic data based on an index structure (e.g., but not limited to, the index structure of FIG. 15 ). The query may be performed based on a lookup operation implemented as follows: First, at 2210, the system processor 110 receives a user query including information indicating one or more attributes of interest. At 2220, the system processor parses the query and identifies one or more attributes in the query. Within the query, the one or more attributes may be specified in various ways, such as a string such as “abcd” or a specific attribute number in a series of numbers identifying each attribute. At 2230, the system processor searches the attribute-specific index to determine one or more chunks or one or more chunk numbers corresponding to the attributes identified in the query. At 2240, the system processor searches the genome annotation data to retrieve (or otherwise obtain access to) one or more chunks or one or more chunk numbers that match the query attributes. By using the chunk index to restore the chunks, values ​​that match the condition can be filtered (as the chunks may contain non-matching values).

[0104]

[0127] At 2250, the decompressed chunks may be selectively expanded (e.g., expanded without expanding other chunks unrelated to the specified attribute). Selective expansion is possible because multiple data chunks were previously selectively compressed independently of one another. The expansion may be performed based on the algorithms shown in the attribute information structure of FIG. 12 and / or the index, table, or other structures described herein. If the chunks are globally compressed, the globally compressed data may be shared among the chunks using, for example, the CompressorCommonData mechanism shown in the attribute information structure. In one embodiment, in the case of a symmetric two-dimensional array table (e.g., when the SymmetryFlag for the table in FIG. 11 is true), the chunks may cover only a portion, such as the lower triangular portion and diagonal. In this case, the expansion process may address the upper diagonal values ​​by filling them with the corresponding lower triangular values. In other cases, the chunks may cover the entire range of the index without overlapping.

[0105] interoperability

[0128] The unified and standardized file format for genome annotation data may have various additional features. One additional feature relates to the linkage and interoperability associated with files generated in this format. In one embodiment, a system processor can process the file information into the unified and standardized format as described herein so as to be independent of any encoding or formatting of the source file (e.g., data provided by data source 140 of FIG. 1).

[0106]

[0129] In another embodiment, the system processor may process the file information into a unified and standardized format that is linked to the encoding and / or formatting of the source file and is interoperable. The data source file may be in MPEG-G format / encoding or another type of format or encoding.

[0107]

[0130] Once processed into a linked, interpretable state, the files can be processed as part of a source file (e.g., an MPEG-G file) by storing the associated information in a dataset. The MPEG-G file stores data from an entire study, with each dataset group corresponding to a different individual. Each MPEG-G dataset group is further divided into multiple datasets corresponding to different sequencing runs.

[0108]

[0131] To store data corresponding to a single individual, multiple different annotation files are assembled into multiple separate datasets, each dataset containing a single annotation file or sequencing data. Examples of separate datasets are shown below: Dataset group (single individual) → Dataset 1 (sequencing data) Dataset 2 (sequencing data) Dataset 3 (variant call data) Dataset 4 (gene expression data) ...

[0109]

[0132] When collecting annotation data from a large-scale survey, the dataset may be organized as follows: Dataset Group (Large-scale Survey) → Dataset 1 (variant call data) → Annotation file (sample 1) Annotation file (sample 2) ... Dataset 2 (gene expression data) → Annotation file (sample 1) Annotation file (sample 2) ... ...

[0110]

[0133] In one embodiment, multiple different annotation files may be merged to improve compression and analysis performance, for example, as shown below. Dataset Group (Large-scale Survey) → Dataset 1 (variant call data) → Annotation files (all samples) Dataset 2 (gene expression data) → Annotation files (all samples) ...

[0111]

[0134] To perform this implementation, the system processor may extend the existing dataset header structure with additional fields to support the data type (sequencing / variant / gene expression / ...), the number of annotation files in the dataset, and the byte offset of each of these files. If a compressor is shared between annotation files or datasets, the compressor parameters may be stored at the dataset level or dataset group level, respectively. In one embodiment, one or more of the annotation files contains a compressor structure with compressor name "POINTER" and compression parameters that store the position, e.g., {"DatasetGroupId":1, "DatasetId":2, "CompressorId":5} indicates that the compressor is as specified in dataset group 1, dataset 2, the fifth compressor.

[0112] linkage

[0135] In addition to the above features, the system processor may process the data to generate a unified and standardized file in a format that includes linkage between different types of annotation data and corresponding sequencing data. In one embodiment, this linkage may be provided based on metadata stored in or associated with the file.

[0113]

[0136] This may be achieved by the system processor specifying the dataset group or datasets that store the sequencing data or associated annotation data in the FileMetadata or TableMetadata fields of Figure 4, for example, using the uniform resource identifier (URI) notation described in MPEG-G part 3, or by using JSON. For example, the following JSON may be used in FileMetadata to provide linkage to the sequencing datasets: “Linkages”:[{ “DataType” :“Sequencing”, “DatasetGroup”:5, “Dataset”:2 }] Although the above example shows only a single linkage, in other embodiments multiple linkages may be provided.

[0114]

[0137] Additionally or alternatively, the system processor may generate table-level linkages. In one embodiment, the system processor can generate table-level linkages by index. In this case, for example, the nth row (column) of one table may correspond to the mth row (column) of another table. This type of linkage can prevent repetition when multiple annotation files / tables share the same row / column (e.g., multiple VCFs that have not yet been merged and consist of the same variant). Similarly, this type of linkage can be useful when sample-related information is stored in a single table, to which both the VCF table and the gene expression table link.

[0115]

[0138] In another embodiment, the system processor can generate table-level linkages by index. In this case, a particular attribute can be linked by matching its value with an attribute in another table. For example, gene expression data can include a gene name without detailed information about the gene, which is available in a separate file. An example of the use of such linkages could be a query requesting gene expression data for all genes in the MHC (major histocompatibility complex) that correspond to an autoimmune disease and specify a coordinate range within human chromosome 6. To address this query, multiple gene names for the coordinate range can be retrieved from the gene information file based on the genome coordinate index, and the required data can be obtained by querying these names within the gene expression file. The following example relates to these features, specifically, a row (dimension 1) can be linked to a row in another table (e.g., table 3 in the same annotation file). “Linkages”:[{ “Type” : “byIndex”, “DimensionInCurrentTable”:1, “Table”: 3, “DimensionInLinkedTable” :1 }]

[0116]

[0139] An example of linking a column (dimension 2) to a row in another table by attribute value (attribute 2 in dimension 2 of the current table is linked to attribute 5 in dimension 1 of table 3 in dataset 4 of file 2) can be performed by the system processor as follows: “Linkages”:[{ “Type” : “byValue”, “DimensionInCurrentTable”:2, “AttributeInCurrentTable”:4, “Dataset”:4, “AnnotationFile” :2, “Table”: 3, “DimensionInLinkedTable” :1, “AttributeInLinkedTable” :5, }]

[0117]

[0140] Because the metadata structure supports arbitrary information storage, the system processor may further extend the linkage to link three or more tables, for example, using a standardized format (e.g., table 3 may convert gene IDs used in table 1 to gene names in table 2). Although the above example uses a particular JSON-based format for the linkage, another format may be used, for example, but not limited to, XML.

[0118]

[0141] In one embodiment, attribute-based linkage may be implemented. While metadata-based linkage may be useful for high-level linkage, in some cases, row / column-wise linkage may be beneficial. For example, in a VCF file with multiple samples, sequencing data corresponding to a particular sample may be linked by adding the attributes SequencingDatasetGroup and SequencingDataset to the column attributes. For such linkage attributes, the "LinkageAttributeFlag" may be set to true in the metadata so that the decompressor can distinguish linkage attributes from regular attributes.

[0119]

[0142] In some cases, the system processor may perform mapping operations between annotation datasets according to genomic regions. This may be achieved, for example, by separately indexing each dataset. For example, to find sequencing data corresponding to a region in a VCF file, one or more appropriate access units may be determined by accessing a master index table of sequencing data. Using separate indexing for different datasets allows for the selection of predetermined (e.g., optimal) chunk sizes and other parameters for each dataset. Furthermore, in some cases, variants may not be directly linked to AUs due to different AU classes. Similarly, in a VCF file with multiple samples, variants are mapped to access units across multiple datasets, and storing this information can occupy significant storage space. In one embodiment, the AUId or byte offset of the sequencing data is stored as a row attribute in the VCF file, enabling fast retrieval of the access unit corresponding to the current variant. Genes may also be mapped to lists of variants by using a list attribute for genes.

[0120]

[0143] In one embodiment, access control policies use a predetermined format, such as XACML, and may be specified at the file level (e.g., in the annotation file of FIG. 4), the table level (e.g., in the information of FIGS. 10 and 11), or both. Certain users may have access to all data, while other users may only have access to coarser-resolution data (recall that different resolutions are stored in different tables). This type of policy may be specified at the file level. Policies specific to attributes within a table may also be specified at the table level, for example. This may include access to only a subset of attributes, or to only certain chunks based on the value of an attribute. Another type of policy may allow access to metadata and information but not the actual data.

[0121] Deployment

[0144] The process for extracting selected portions of a file or the entire file in the unified and standardized format described herein may first include executing one or more types of queries. The queries and / or extraction algorithms may not be mutually exclusive and may be combined in some embodiments. For example, both metadata and specific attributes may be extracted, or selected attributes from selected chunks may be extracted. The extraction may be performed by a system processor or by another processing entity coupled to the system processor. In some cases, some of these queries may be restricted by access control policies. An application programming interface (API) may be used to support these. Such an API may be similar to MPEG-G part 3 or may be a different type of API.

[0122]

[0145] Metadata / Information Queries: These types of queries may query only the metadata and information corresponding to the requested table (e.g., resolution level), compressor, attribute, and / or chunk. When performing a query, the top-level information may first be accessed directly, as listed at the beginning of the annotation file in Figure 4. Table-specific metadata / attribute details may be selectively accessed using the ByteOffset of the table in Figure 4.

[0123]

[0146] Full Data Expansion: This type of expansion may involve expanding the entire data, including all tables and attributes. This may be performed by first reading the file's top-level metadata and table information, as described above. Next, the compression parameters are loaded into the expansion manager. Then, for each table, the table information, dimensions, and attributes are read, followed by an index to determine the chunk's location along each dimension. Next, the data payload is selectively expanded (serial or parallel) for each chunk and each attribute. If an attribute is compressed using another attribute as a dependency / context, expansion may be performed by first expanding that other attribute. If the attribute uses CompressorCommonData, this information may be loaded before expanding the chunk. For two-dimensional symmetric arrays (e.g., see SymmetryFlag above), only the diagonal and lower triangular matrices may be expanded, and the upper triangular portion may be filled in using symmetry.

[0124]

[0147] Extracting only one table: This type of extraction is similar to the full data extraction, except that the ByteOffset field in the annotation file (e.g., in Figure 4) is used to jump to the requested table, and only that table is extracted.

[0125]

[0148] Querying Selected Attributes of a Table: This method is similar to extracting only one table, except that only the information corresponding to the requested attributes is read. Other attributes are skipped based on the AttributeInfoSize variable in the attribute information structure of FIG. 12. In one embodiment, only the requested attributes are extracted by skipping other attributes based on the Payload Size[i][j] information of FIG. 22. If attribute-dependent chunks are used, all chunks for a given attribute may be stored together, for example, as shown in the data payload structure of FIG. 20.

[0126]

[0149] Querying only a selected index range within an array: This method is similar to unpacking only one table, except that the indexes are loaded and chunks that overlap with the requested range are determined depending on the chunking type (fixed size / variable size). The ByteOffset information in the chunk structure table (Figure 16) may be used to jump to the payload of the chunk. In some cases, the method may be implemented more efficiently if attribute-dependent chunking is not used and all attributes of a given chunk are stored together. The requested chunk is unpacked and only the overlapping indexes are returned. If the compressor for a given attribute allows efficient random access within the chunk, this random access can be used as a basis to further increase unpacking speed. An example scenario in which this might occur is a specialized compressor for sparse arrays or genotypes such as GTC.

[0127]

[0150] Querying based on the value / range of a specific attribute: This approach is similar to querying only a selected index range in an array, except that if an additional attribute-specific index (e.g., as shown in the index structure of FIG. 15) is available for the chunk(s) of interest, this index may be used to determine the chunk(s). If such an index is not available, the attribute of interest may be expanded for all chunks, and then the associated chunks may be determined. Even without using additional indexes, the expansion speed may be improved because only some attributes are expanded for all chunks. In one embodiment, the remaining attributes may be expanded only for the associated chunks.

[0128] Folder Structure and Editing

[0151] The processing of genome annotation data into a unified and standardized file format by a system processor as described above provides many advantages in terms of efficiency, convenience, reduced system requirements, fast querying, and data accessibility. Additional variations and adjustments may be made to the embodiments to better suit particular applications.

[0129]

[0152] For example, if data is stored on a single machine and is frequently edited, the data may be stored in a directory / folder hierarchy using a file manager. This hierarchy may make it easy to manipulate portions of the data by modifying only the file corresponding to a single chunk and attributes, rather than overwriting the entire file. When editing is complete and the data is sent, the file may be converted back to a single file format. This can be done by recalculating the index based on the data payload size and collapsing the folder hierarchy back into a single file.

[0130]

[0153] An example of how to convert a file format (e.g., Figure 23A) to a folder hierarchy (e.g., Figure 23B) and back to a single table is shown. In this example, the conversion is performed by treating each table in the table section 2310 of the file format as a folder in the folder hierarchy, and each chunk of data in the data section 2320 of the file format as a folder in the folder hierarchy (e.g., assuming a false value is stored in the AttributeDependentChunks field). In the folder hierarchy, blocks 2331-2343 correspond to files, and blocks 2321-2326 correspond to folders. In the folder hierarchy, since the chunks are already stored in separate folders, the index only needs to store attribute-specific indexes. A simple example is shown in Figure 9.

[0131]

[0154] When considered as a folder hierarchy, the proposed scheme can be loosely related to the scheme proposed in the above-referenced related patent application by changing the terminology as follows: "attribute": "area with one or more rows / columns" and "chunk": "block".

[0132]

[0155] To illustrate how one or more of the above embodiments may be implemented, the following two examples for storing various annotation data while providing related functionality are discussed: (1) variant call data, and (2) genomic functional annotation data. [Table 1]

[0133]

[0156] Table 1 above shows a section of a VCF file containing five variants and three samples. VCF files can be processed by the system processor 110 into a unified and standardized file format, both to hold data and to provide additional associated functionality. Such files can have the following features:

[0134]

[0157] Metadata: Comment lines (starting with ##) may be kept as part of the FileMetadata. If this is saved as part of an MPEG-G file along with the sequencing data, the metadata may also include the corresponding dataset group that contains the sequencing data that corresponds to this variant call data.

[0135]

[0158] Traceability: When traceability information is stored as part of an MPEG-G file along with the sequencing data, it may include one or more commands for generating one or more variant calls, starting with the URI of the raw sequencing data and the tools and their versions used. The traceability information may be used to validate the file in a reproducible manner.

[0136]

[0159] Table: If the variant data is stored at a single resolution, the variant data may be stored in a single table in a unified file format (nDimensions=2).

[0137]

[0160] Dimension Attributes: For the first dimension (variant), there may be multiple dimension attributes such as CHROM, POS, ID, REF, ALT, QUAL, FILTER, and INFO fields. As described in the comments, the INFO field may be split into multiple attributes such as NS, DP, and AF. The types of these attributes may also be mentioned in the comments field. Attribute metadata may be used to group them (e.g., NS, DP, and AF may belong to the group INFO). The default value depends on the attribute; for example, for the FILTER attribute, the default value may be set to "PASS".

[0138]

[0161] For the second dimension (sample), the sample name (e.g., NA00001) may be the only attribute present in the original VCF file. Additional attributes, such as the dataset group and dataset containing the sequencing data corresponding to this sample, may be added to support linkage to sequencing data. Additional dimension attributes may be added to support fast access to specific quantities, such as counts or average quantities corresponding to specific variants. A description of the INFO attribute in a comment may be saved as part of the AttributeMetadata.

[0139]

[0162] 2D Table Attributes: These attributes are described in FORMAT fields such as GT, GQ, DP, etc., each of which is a two-dimensional array. The type of these attributes may also be described in comments. If most variants are not represented, the default value of the GT attribute may be set to, for example, 0 / 0. The description of these attributes in comments may be stored as part of the AttributeMetadata.

[0140]

[0163] Compressors: Compressors for attributes may be selected based on the attribute's type and characteristics. For example, CHROM may be compressed using gzip after an enumeration-based scheme, and POS may be compressed using gzip after differential encoding. Sample names (e.g., NA00001) may be efficiently compressed using a tokenization-based string compressor, for example. Some INFO fields may be encoded with a sparse representation because they may only be present for a small number of variants. Similarly, genotypes (GT) may be encoded with a sparse representation or a genotype-specific compressor (e.g., GTC).

[0141]

[0164] The length of a particular variable-length attribute may depend on one or more other attributes. For example, the length of the AF (allele frequency) attribute may be equal to the number of variant alleles. In such cases, the compressor's nDependencies may be set to 1, and this dependency may be exploited to enhance compression. Similarly, the value of the GT field may be used as side information for the compression of other FORMAT fields, exploiting the dependency between them.

[0142]

[0165] Chunking and Indexing: Chunking of the main 2D array may be performed depending on the access pattern. If most accesses are to variants within a particular region, each chunk may contain all samples and a few variants (e.g., horizontal chunks). If most accesses are to all variants of a particular sample, a chunk may contain all variants and a few samples (e.g., vertical chunks). If both types of queries occur frequently, it may be better in some cases to use rectangular chunks containing a few variants and samples. By increasing the chunk size, random access performance may be traded off for compression ratio.

[0143]

[0166] For random access based on genomic regions, additional indexes may be used (e.g., based on CSI indexing) as shown in Table 2. [Table 2]

[0144]

[0167] Instead of specifying an actual file location as is done with CSI, a list of chunk IDs that overlap with the genomic region of interest may be returned (or indicated). The locations of these chunks within the file can then be determined from a default index structure. If indel or structural variants are common, CSI indexing may be based on both the START and END positions of the variants. More attributes may be indexed to enable fast random-access queries. For example, indexing the FILTER attribute may enable faster filtering of variants based on whether FILTER=PASS.

[0145]

[0168] Protection: Access control policies can take various forms, depending on, for example, the use case. For example, certain users may have access to all data, while others may only have access to variants within specific genomic regions (e.g., specified by CHROM and POS). Similarly, access may be restricted to only specific samples. In some cases, this may require chunks to be selected accordingly. Access control may also be imposed at the attribute level, for example, allowing access to INFO fields but not individual sample data. [Table 3]

[0146]

[0169] Table 3 above shows the sections of a BED file with annotation data. A system processor may process the information in this file to conform to a unified and standardized file, preserving the data and providing additional functionality, as described below.

[0147]

[0170] Metadata: Comment lines (e.g., the first three lines) may be retained as part of the FileMetadata. If this is saved as part of an MPEG-G file along with the sequencing data, the metadata may also include the corresponding dataset group that contains the sequencing data that corresponds to this annotation data.

[0148]

[0171] Tables: To allow data to be displayed at different scales or resolutions, multiple tables may be generated by the system processor with pre-calculated values ​​for different resolutions. For example, a TableInfo field may store parameters indicating the resolution and other information in a predefined format. This may allow a user to query a list of available resolutions without reading the entire file. Also, the desired resolution may be directly accessible by using a ByteOffset variable for each table. One or more of the multiple tables may, for example, have a single dimension.

[0149]

[0172] Attributes: In one embodiment, each column can serve as an attribute as follows: chrom(string), chromStart(integer), chromEnd(integer), name(string), score(integer), strand(character), thickStart(integer), thickEnd(integer), itemRGB(8-bit integer array of length 3).

[0150]

[0173] Compressor: A compressor for an attribute may be selected based on the type and characteristics of the attribute. For example, chrom may be compressed using an enumeration-based scheme followed by gzip or run-length compression, and chromStart and chromEnd may be compressed using gzip followed by differential encoding. If the values ​​of thickStart and thickEnd are close to chromStart and chromEnd, compression may be improved by using these values ​​as side information, for example.

[0151]

[0174] In this example, the value of chromStart matches the value of chromEnd from the previous line. One way to take advantage of this is to consider chromStart and chromEnd as a single attribute of type "integer array of length 2". This can be done, for example, if the visualization tool understands this alternative representation.

[0152]

[0175] Chunking and Indexing: For random access based on genomic regions, an additional index may be used (based on CSI indexing) as shown in Table 4. Instead of specifying the actual file location as is done in CSI, a list of chunk IDs that overlap with the genomic region of interest may be indicated. The location of these chunks within the file can then be determined from the default index structure. [Table 4]

[0153]

[0176] Instead of specifying the actual file location as is done in CSI, a list of chunk IDs that overlap with the genomic region of interest may be given, and the location of these chunks within the file can be determined from the default index structure.

[0154]

[0177] Protection: Access control policies can also take different forms depending on the use case. For example, certain users may have access to all data, while others may only have access to coarser resolution data (recall, for example, that different resolutions may be stored in different tables). Similarly, access may be restricted to only certain genomic regions; in this case, chunks may be selected accordingly.

[0155] Single-cell RNAseq expression data

[0178] Single-cell RNAseq expression data may contain a sparse two-dimensional matrix of integer / float expression values. Each row corresponds to a gene, and each column corresponds to a barcode representing a cell. In this case, expression values ​​are stored as a two-dimensional sparse attribute array, while gene-related information is a dimension-specific row attribute and barcode-related information is a dimension-specific column attribute. The sparse array may be split into a coordinate stream and a value stream. These are compressed separately, and the row coordinates and column coordinates of each row are differentially encoded before compression. To enable fast random access, the data may be chunked into a fixed number of genes (rows) per chunk. Finally, an additional B-tree index may be used to perform selective queries based on gene ID. This index maps gene IDs to chunks containing the gene ID and the location of the gene ID within the chunk.

[0156]

[0179] We applied this method to a dataset consisting of 10,000 brain cells from E18 mice. The sparse array contained approximately 31,000 genes, 6,800,000 barcodes, and 40 million integer entries. The proposed method, using BSC as the final compression layer, reduces the size from 750 MB (uncompressed) to 67 MB, which is less than half the size after compression when applying gzip or BSC directly to the original file without splitting columns into attributes or differentially encoding coordinates.

[0157]

[0180] The methods, processes, and / or operations described herein may be performed by code or instructions executed by a computer, processor, controller, or other signal processing device. The code or instructions may be stored on a non-transitory computer-readable medium according to one or more embodiments. Because the algorithms underlying the methods (or the operations of a computer, processor, controller, or other signal processing device) are described in detail, the code or instructions for implementing the operations of the method embodiments may transform a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods described herein.

[0158]

[0181] The processors, compressors, decompressors, managers, selectors, parsers, and other information generating, processing, and computational features of the embodiments disclosed herein may be implemented, for example, in logic that may include hardware, software, or both. When implemented at least partially in hardware, the processors, compressors, decompressors, managers, selectors, parsers, and other information generating, processing, and computational features may, for example, be any of a variety of integrated circuits, including, but not limited to, an application specific integrated circuit, a field programmable gate array, a combination of logic gates, a system on a chip, a microprocessor, or another type of processing or control circuitry.

[0159]

[0182] When implemented at least in part in software, the processor, compressor, decompressor, manager, selector, parser, and other information generating, processing, and computational features may include memory or other storage devices for storing code or instructions executed by, for example, a computer, processor, microprocessor, controller, or other signal processing device, etc. Because the algorithms underlying the methods (or the operation of a computer, processor, microprocessor, controller, or other signal processing device) have been described in detail, the code or instructions for implementing the operations of embodiments of the methods may transform a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods described herein.

[0160]

[0183] From the above description, it will be apparent that various exemplary embodiments of the present invention may be implemented in hardware or firmware. Various exemplary embodiments may also be implemented as instructions stored on a machine-readable storage medium. These instructions may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a machine-readable storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and similar storage media.

[0161]

[0184] Those skilled in the art will appreciate that the block diagrams presented herein are conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, any flowcharts, flow diagrams, state diagrams, pseudocode, and the like are tangibly represented in a machine-readable medium and thus represent various processes that may be performed by a computer or processor (whether or not a computer or processor is explicitly shown).

[0162]

[0185] While various exemplary embodiments have been described in detail with particular reference to certain illustrative aspects of those exemplary embodiments, it should be understood that the invention is capable of other embodiments and its details are susceptible to modification in various obvious respects. One or more embodiments may be combined with one or more other embodiments to form new embodiments. As will be readily apparent to those skilled in the art, variations and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the foregoing disclosure, description, and drawings are for illustrative purposes only and do not in any way limit the invention, which is defined solely by the claims.

[0163]

[0186] According to one or more embodiments, a system and method for controlling selective compression and decompression of genome annotation data includes processing information from one of multiple incompatible file formats into a unified file that controls selective compression and decompression. The annotation data is processed, and information is extracted and further processed to support features such as fast querying, random access, multiple resolutions (zooms), selective encryption, authentication, access control, and traceability. Processing may also be performed to enable significant compression gains by separating different attributes of the data and enabling the use of specialized compressors. Additional processing may be performed to support metadata and linkage to sequencing data associated with the annotations, as well as linkage to other annotation data from the same study. This allows for seamless integration with existing MPEG-G file formats for sequencing data.

[0164]

[0187] In one or more embodiments, additional processing may be performed to generate protection (access control) information at multiple levels in the hierarchy, enabling fine-grained security configuration. Similarly, metadata and attributes allow for effective linking of multiple different types of annotation data and sequencing datasets. The files generated by the processing may be used as standalone files or as part of MPEG-G files. Furthermore, the generation of such files, particularly for genome annotation data, provides sufficient flexibility to achieve state-of-the-art compression performance for various data types by incorporating compression techniques appropriate for the attributes of interest.

Claims

1. 1. A method for controlling compression of data, the method comprising: accessing tabular data in a first file format of a plurality of first file formats; extracting a plurality of attributes from the tabular data; splitting the tabular data into a number of chunks; processing the extracted attributes and the chunks into correlated information; selecting different compressors and dependency attributes for the attributes and chunks identified in the correlated information; storing global data for the compressor that is shared between the chunks; generating a file in a second file format including the correlated information and information indicating the different compressors for the chunks and the attributes indicated in the correlated information; wherein the first file formats are incompatible with one another and differ from the second file format, and the information indicative of the different compressors is processed into the second file format to enable selective expansion of the attributes and chunks indicated in the correlated information.

2. The correlated information includes: First information indicative of one or more of the attributes; and second information corresponding to chunks associated with the one or more attributes indicated in the first information; The method of claim 1 including at least one table.

3. The method of claim 2 , wherein the first information comprises a two-dimensional cell array, each cell identifying one or more corresponding attributes contained within the chunk that corresponds to the two-dimensional cell array.

4. The method of claim 3 , wherein the first information comprises at least one one-dimensional table of dimension-specific attributes associated with the chunk.

5. The method of claim 1 , wherein at least one of the different compressors is embedded in the file.

6. The method of claim 1 , wherein at least one of the different compressors encodes one or more attributes of the tabular data as a sparse array.

7. The method of claim 1 , wherein the tabular data is divided into chunks of different sizes.

8. The method further comprises: generating one or more attribute-specific indexes; and incorporating the one or more attribute-specific indexes into the file in the second file format, the attribute-specific indexes enabling identification of chunks of the tabular data that contain values ​​or ranges of particular attributes.

9. The method further comprises:

2. The method of claim 1, further comprising the step of: merging said files of said second file format into an MPEG-G file.

10. The method further comprises: generating access control policy information for the correlated information; and integrating the access control policy information into the file of the second file format; 2. The method of claim 1, wherein the access control policy information includes first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level.

11. 1. An apparatus for controlling compression of data, the apparatus comprising: a memory configured to store instructions; and at least one processor configured to execute the instructions to perform operations, the operations including: accessing tabular data in a first file format of a plurality of first file formats; extracting a plurality of attributes from the tabular data; splitting the tabular data into a number of chunks; processing the extracted attributes and the chunks into correlated information; selecting different compressors and dependency attributes for the attributes and chunks identified in the correlated information; storing global data for the compressor that is shared between the chunks; generating a file in a second file format including the correlated information and information indicating the different compressors for the chunks and the attributes indicated in the correlated information; the plurality of first file formats are incompatible with one another and differ from the second file format, and the information indicative of the different compressors is processed into the second file format to enable selective expansion of the attributes and the chunks indicated in the correlated information.

12. The correlated information includes: First information indicative of one or more of the attributes; and The apparatus of claim 11 , further comprising at least one table including second information corresponding to chunks associated with the one or more attributes indicated in the first information.

13. The apparatus of claim 12 , wherein the first information comprises a two-dimensional cell array, each cell identifying one or more corresponding attributes contained within the chunk that corresponds to the two-dimensional cell array.

14. The apparatus of claim 13 , wherein the first information comprises at least one one-dimensional table of dimension-specific attributes associated with the chunks.

15. The method of claim 1 , wherein at least one of the different compressors is embedded in the file.

16. The method of claim 1 , wherein at least one of the different compressors encodes one or more attributes of the tabular data as a sparse array.

17. The apparatus of claim 11 , wherein the tabular data is divided into chunks of different sizes.

18. The at least one processor executes the instructions to: generating one or more attribute-specific indexes; 12. The apparatus of claim 11, configured to incorporate the one or more attribute-specific indexes into the file of the second file format, the attribute-specific indexes enabling identification of chunks of the tabular data that contain values ​​or ranges of particular attributes.

19. 12. The apparatus of claim 11, wherein the at least one processor is configured to execute the instructions to merge the files of the second file format into an MPEG-G file.

20. The at least one processor executes the instructions to: generating access control policy information for the correlated information; configured to integrate the access control policy information into the file in the second file format; 12. The apparatus of claim 11, wherein the access control policy information includes first information indicating a first access level for a first portion of the correlated information and second information indicating a second access level for a second portion of the correlated information, the second access level being different from the first access level.

Images