System and method for an automatically configurable data compression framework

Abstract

1. A method (100) for compressing and decompressing a data file, the method (100) comprising: (i) receiving a data file for compression including a plurality of different attributes (120); (ii) identifying a first attribute of the plurality of different attributes (130); (iii) selecting a plurality of compression types and / or configurations (140); (iv) compressing at least a portion of data from the received data file for the identified first attribute using each of the selected plurality of compression types and / or configurations (150); (v) determining which of the selected plurality of compression types and / or configurations is most suitable for compression (160); (vi) generating a compression parameter data structure (170) including an identification of the selected plurality of compression types and / or configurations; (vii) compressing data from the received data file for the first attribute to generate a compressed data file (180); and (viii) storing the compression parameter data structure and the compressed data file (190).

Description

Technical Field

[0001]

[0001] This disclosure generally relates to methods and systems for compressing data files using a file compression / decompression system.

Background Art

[0002]

[0002] Considering that the sizes of many data files continue to increase, efficient file compression methods and systems are becoming increasingly important. A single genomic data file, for example, can include sequencing results for millions or billions of locations across the genomes of multiple samples, along with annotations and other data attributes. A Variant Call Format (VCF) file, for example, can include genetic sequence variations with millions of rows containing data attributes and a large number of columns. These data files must be stored over a long period of time and can become extremely large, such as 100 gigabytes or 1000 gigabytes.

Summary of the Invention

Problems to be Solved by the Invention

[0003]

[0003] Compression of data files generally involves one or more data compressors. There are many common compressors that can be directly applied to data files, but their compression performance is usually suboptimal because they do not fully utilize the similarity or predictability of the data in individual fields within a dataset for the effective removal of data redundancy. One technique to improve the compression ratio is, therefore, to first divide the dataset into fields with distinct characteristics, and then compress each field individually using a compressor that can make the most use of the particular data model of the field. However, this generally requires that the data fields and their associated compressors be predefined for each file type, and that the encoding and decoding processes be hardcoded accordingly in the software used for compression and decompression. The lack of flexibility of such a technique makes it difficult to adapt to emerging data types or to adopt new algorithms. [Means for solving the problem]

[0004]

[0004] There is an ongoing need for methods and systems for efficient and adaptable compression and decompression of data files. This disclosure relates to inventive methods and systems for compressing data files using automated identification and / or configuration of a data compressor based on compression performance. A file compression / decompression system receives a data file for compression, which contains multiple different attributes for each of multiple different samples. The system identifies a first attribute from the multiple different attributes and, based on the first attribute, selects multiple different compression types and / or compression configurations from a predefined set of compression types and configurations. The system compresses the data from the received genome data file for the identified first attribute individually using each of the selected compression types and / or compression configurations and determines which of the selected compression types and / or compression configurations is most suitable for compression. Next, the system generates a compressed parameter data structure having an identification of the most suitable compression type and / or compression configuration from among a selection of compression types and / or compression configurations for compressing the identified first attribute, and compresses the data from the received genome data file for the first attribute using the identified compression type and / or compression configuration to generate a compressed genome data file. The compressed parameter data structure and the compressed genome data file can then be stored and used for decompression.

[0005]

[0005] Generally, in one embodiment, a method is provided for compressing / decompressing a genome data file using a file compression / decompression system. The method includes (i) receiving a data file for compression in which the data file contains a plurality of different attributes; (ii) identifying a first attribute from the plurality of different attributes and storing the specifications of the identified first attribute in an attribute parameter data structure; (iii) selecting a plurality of compression types and / or compression configurations from a plurality of predefined compression types and / or compression configurations based on the identified first attribute; (iv) compressing at least a portion of the data from the received data file for the identified first attribute individually using each of the selected plurality of compression types and / or compression configurations, wherein one or more metrics are measured for each individual compression; and (v) selecting a plurality of metrics based on one or more metrics for each of the individual compressions. (vi) determining which of several compression types and / or compression configurations is most suitable for compression; (vii) generating a compression parameter data structure having an identification of the compression type and / or compression configuration most suitable for compression of an identified first attribute from among the selected multiple compression types and / or compression configurations; (vii) compressing data from a received data file for the first attribute using the identified compression type in order to generate a compressed data file; and (viii) storing the attribute parameter data structure, the compression parameter data structure and the compressed data file, wherein the stored compression parameter data structure is configured to be retrieved and used for decompressing the compressed data file.

[0006]

[0006] According to one embodiment, the method further includes repeating the steps of identifying, selecting, compressing, and determining for at least a second attribute among a plurality of different attributes, wherein the compression parameter data structure includes a specification of the compression type and / or compression configuration that is most suitable for compressing the identified second attribute among the plurality of selected compression types and / or compression configurations.

[0007]

[0007] According to one embodiment, the method further includes decompressing a stored compressed file, the decompression comprising examining a compression parameter data structure to identify a decompression type for a first attribute.

[0008]

[0008] According to one embodiment, each of the selected compression types and / or compression configurations is selected because it is capable of compressing data from an received data file for an identified first attribute.

[0009]

[0009] According to one embodiment, the data file is a genome data file.

[0010]

[0010] According to one embodiment, one or more metrics include one or more of the compression ratio and processing time for the compression.

[0011]

[0011] According to one embodiment, the compression type is most preferred when the corresponding compression has a higher compression ratio and / or faster processing time than other compression types and / or compression configurations among a plurality of selected compression types and / or compression configurations.

[0012]

[0012] According to one embodiment, the method further includes providing information about the generated compressed parameter data structure and / or the compressed data file via a user interface.

[0013]

[0013] According to one embodiment, the method further includes providing the user with information about one or more of the identified compression types and / or compression configurations via a user interface so that the user can select the compression type and / or configuration to be applied to compress the attributes.

[0014]

[0014] The second aspect is a system for compressing / decompressing a data file. The system includes a data file for compression containing a plurality of different attributes, and a processor, the processor (i) identifies a first attribute from the plurality of different attributes and stores the specifications of the identified first attribute in an attribute parameter data structure, (ii) selects a plurality of compression types and / or compression settings from a plurality of predefined compression types and / or compression settings based on the identified first attribute, (iii) compresses at least a portion of the data from the received data file for the identified first attribute individually using each of the selected plurality of compression types and / or compression settings, wherein one or more metrics are measured for each individual compression, and (iv) selects a plurality of metrics based on one or more metrics for each of the individual compressions. (v) determine which of the compression types and / or compression settings is most suitable for compression; (vi) generate a compression parameter data structure having an identification of the compression type and / or compression setting most suitable for compression of an identified first attribute from among a plurality of selected compression types and / or compression settings; (vi) compress data from a received data file for the first attribute using the identified compression type in order to generate a compressed data file; and (vii) store the attribute parameter data structure, the compression parameter data structure and the compressed data file, wherein the stored compression parameter data structure is configured to be retrieved and used for decompressing the compressed data file.

[0015]

[0015] According to one embodiment, the system includes a user interface configured to provide information about the generated compressed parameter data structure and / or the compressed data file.

[0016]

[0016] In various implementations, a processor or controller may be associated with one or more storage media (volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, and magnetic tapes, which are commonly referred to herein as “memory”). In some implementations, the storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform at least some of the functions described herein. Various storage media may be fixed within a processor or controller or be portable so that one or more programs stored thereon can be loaded into the processor or controller to implement the various embodiments described herein. The terms “program” or “computer program” are used herein in a general sense to refer to any type of computer code (e.g., software or microcode) that can be employed to program one or more processors or controllers.

[0017]

[0017] It should be understood that all combinations of the above concepts, and any additional concepts described in more detail below, are intended to be part of the inventive subject matter disclosed herein (unless such concepts are mutually contradictory). In particular, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the inventive subject matter disclosed herein. It should also be understood that any terms expressly adopted herein that may appear in any disclosure incorporated by reference should be given meanings that best correspond to any particular concept disclosed herein.

[0018]

[0018] These and other aspects of the various embodiments will become apparent from the embodiments described below and will be clarified by referring to the embodiments described below.

[0019]

[0019] In the drawings, the same reference numerals across different views generally refer to the same part. Also, the drawings are not necessarily to a fixed scale, and rather, the emphasis is generally on illustrating the principles of various embodiments. [Brief explanation of the drawing]

[0020] [Figure 1]

[0020] This is a flowchart of a method for compressing genome data files using a file compression / decompression system according to one embodiment. [Figure 2]

[0021] This is a schematic representation of a file compression / decompression system according to one embodiment. [Figure 3]

[0022] This is a schematic representation of a data file according to one embodiment. [Modes for carrying out the invention]

[0021]

[0023] This disclosure describes various embodiments of systems and methods for efficient and adaptable compression of data files. The applicant recognizes and understands that it would be beneficial to provide data file compression methods and systems that employ automated identification and / or configuration of data compressors based on compression performance. A data file compression / decompression system receives a genomic data file for compression, the genomic data file contains multiple different attributes (or data fields) for each of multiple different genomic samples. The system identifies a first attribute from the multiple different attributes and, based on that attribute, selects multiple compression types and / or compression configurations from a set of predefined compression types and / or compression configurations, each of which is capable of compressing data from the received genomic data file for the identified first attribute. The data file compression system individually compresses at least a portion of the data from the received genomic data file for the identified first attribute using each of the selected compression types and / or compression configurations, one or more metrics being measured for each individual compression, one or more of which include a compression ratio and processing time for the compression. Based on one or more metrics of each individual compression, the system determines which of the selected compression types and / or compression configurations is most suitable for compression. According to one embodiment, the compression type is most suitable when the corresponding compression has a higher compression ratio and / or faster processing time than the other compression types and / or configurations among the selected compression types and / or configurations. The system then generates a compression parameter data structure containing the identification of the most suitable compression types and / or configurations among the selected compression types and / or configurations for compressing an identified first attribute, and / or processing instructions for those compression types and / or configurations, and compresses the data from the received genome data file for the first attribute to generate a compressed genome data file.The compressed parameter data structure and the compressed genomic data file are stored, where the stored compressed parameter data structure is retrieved and configured to be used for the decompression of the compressed genomic data file.

[0022]

[0024] According to one embodiment, the novel methods and systems described or otherwise contemplated herein provide a framework for the automatic configuration of a compressor for a data field based on the performance of the compressor, enabling the compressor to be explicitly specified and stored along with the compressed data. To access the attribute data, the decoder can execute the decompression and inverse transformation algorithms based on the processing steps and parameters specified in the compressor configuration associated with the attribute. The novel methods and systems described or otherwise contemplated herein provide numerous advantages. For example, the method and system provide flexibility for customizing the compressor configuration for a file, facilitating the adoption of new transformation and compression algorithms. Second, the method and system adapt to any new file type by identifying and providing the compressor configuration. The processing instructions are included in the compressor configuration and are not hard-coded in software, so the decoder can easily process the file without the need for code modification. Third, the automatic configuration of the compressor settings helps improve compression performance without requiring user knowledge or experience. The method and system also provide many other advantages.

[0023]

[0025] According to one embodiment, the compressor configuration framework system includes one or more of the following components: (i) a uniform data interface for specifying a compressor, such as a compression process consisting of a sequence of data transformations, compression steps, and their default parameters; (ii) a uniform data interface for specifying data attributes and associating data attributes with a compressor using customized parameters; (iii) an algorithm for encoding attribute data based on processing steps specified in the associated compressor configuration; (iv) an algorithm for decoding attribute data based on processing steps specified in the associated compressor configuration; and (v) a mechanism for automatic determination of a performance-based compressor configuration.

[0024]

[0026] Referring to Figure 1, one embodiment is a flowchart of Method 100 for compressing a genome data file using a file compression / decompression system. The method described with respect to the figure should be understood to be given merely as an example and not to limit the scope of this disclosure. The file compression / decompression system may be any of the systems described or otherwise assumed herein. The file compression / decompression system may be a single system or several different systems.

[0025]

[0027] In step 110 of the present method, a file compression / decompression system 200 is provided. Referring to an embodiment of the file compression / decompression system 200 shown in FIG. 2, for example, the system includes one or more of a processor 220, a memory 230, a user interface 240, a communication interface 250, and a storage 260 interconnected via one or more system buses 212. It will be understood that FIG. 2 constitutes an abstraction in some respects and that the actual organization of the components of system 200 may be different and more complex than shown. Further, the file compression / decompression system 200 can be any of the systems described or otherwise envisioned herein. Other elements and components of the file compression / decompression system 200 are disclosed and / or envisioned elsewhere in this specification.

[0026]

[0028] According to one embodiment, the file compression / decompression system 200 includes a file database 270 or communicates directly or indirectly with a file database 270. The file database includes one or more files that can be compressed or have been compressed using the file compression / decompression system 200, among other possible data. According to one embodiment, the one or more files include genomic files, which include genetic sequence variations having millions of rows and a large number of columns containing data attributes for potentially many different genomic samples. The file database can be any such database, including but not limited to the databases and systems described or otherwise envisioned herein.

[0027]

[0029] According to one embodiment, the file compression / decompression system 200 includes or communicates directly or indirectly with a compression and decompression algorithm database 280. The compression and decompression algorithm database includes a library of different compression and decompression algorithms. These compression and decompression algorithms can be used by a data file system to compress and decompress data files. New compression and decompression algorithms can be added to the database at any time. The compression and decompression algorithm database may be any such database, including, but not limited to, the databases and systems described or otherwise assumed herein.

[0028]

[0030] In step 120 of this method, the file compression / decompression system receives an uncompressed data file for compression, the data file containing multiple different attributes for each of multiple different samples. For example, in the case of a genome data file, the genome data file contains multiple different attributes for each of multiple different genome samples. The data file may be received locally or remotely from any source, including the file database 270, which may, but is not limited to, itself be a local or remote database.

[0029]

[0031] In step 130 of the method, the file data system identifies a first attribute among several different attributes in the data file. An attribute can be any information from the data file. Referring to Figure 3, a non-limiting example of a data file is a portion 300 of a genome data file containing several columns 310, each of which is an attribute. The file data system may be configured to recognize known attributes in the data file, or to predict, estimate, or otherwise determine new or unrecognized attributes in the data file. Thus, the file data system may be configured to identify a first attribute among several different attributes in the data file using a wide variety of different methods. According to one embodiment, after identifying an attribute, the system specifies the attribute in an attribute parameter data structure (such as the attribute parameter set described herein).

[0030]

[0032] In step 140, the data file compression / decompression system selects a plurality of compression types or algorithms ("compressors") and / or compression configurations from a plurality of predefined compression types, algorithms, and configurations. A compressor or compression configuration may include, for example, a sequence of compression algorithms and parameters associated with the compression algorithms. Compression types, algorithms, and compression configurations may be selected locally or remotely from any source, including, but not limited to, a compression and decompression algorithm database 280, which itself may be a local or remote database. According to one embodiment, one or more of the plurality of compression types, algorithms, and compression configurations are selected from a plurality of predefined compression types, algorithms, and compression configurations based at least in part on an identified attribute. For example, a compressor or configuration may be selected based on the compressor's ability to compress data from an received data file for a first identified attribute. Thus, the system may preferentially identify compressors or configurations capable of compressing the first attribute data and / or preferentially exclude compressors that are not capable of compressing the first attribute data.

[0031]

[0033] In step 150, the file data system compresses the data using identified compressors. According to one embodiment, the file compression / decompression system compresses the first attribute data individually. This individual compression by multiple compressors can be performed sequentially or simultaneously. According to one embodiment, each compressor compresses only a subset of the first attribute data, and the subsets are equivalent for each compressor to allow for a more objective comparison of compression metrics among multiple compressors. According to another embodiment, each compressor compresses all of the first attribute data.

[0032]

[0034] According to one embodiment, the file compression / decompression system measures, or otherwise obtains or identifies, one or more compression metrics for each individual compression by each compressor. This enables an objective comparison of compression performance among multiple compressors and the selection of a preferred compressor for first attribute data. According to one embodiment, the compression metrics may be compression ratio, processing time, compression accuracy, and / or any other compression metrics.

[0033]

[0035] In step 160 of the method, the file compression / decompression system uses the measured or acquired metrics to determine which of the selected compression types, algorithms, or compression configurations is best suited for compression. Thus, the file compression / decompression system compares, ranks, or otherwise utilizes the metrics corresponding to each individual compressor to select the preferred compressor. For example, a compressor or configuration may be best suited or preferred for compression if it has a higher or best compression ratio and / or faster processing time than the other compression types and / or compression configurations among the selected compression types or configurations. According to the first embodiment, the file compression / decompression system is configured to select the compressor with the best compression ratio. According to the second embodiment, the file compression / decompression system is configured to select the compressor with the fastest compression or processing time. According to the third embodiment, the file compression / decompression system is configured to select the compressor with the best combination of compression ratio and compression or processing time. These configurations may, among other options, be pre-programmed, learned, or determined by the user of the system.

[0034]

[0036] According to one embodiment, the file compression / decompression system repeats the steps of identifying, selecting, compressing, and determining for one or more additional attributes. For example, the system may be configured to identify a preferred compressor for some or all of the attributes found in the data file.

[0035]

[0037] In step 170 of this method, the file compression / decompression system generates a compression parameter data structure, such as a compression parameter data file. According to one embodiment, the generated compression parameter data structure includes the identification of a selected compressor and / or compression configuration for at least a first identified attribute, and / or specifications for such a compressor and / or compression configuration, and optionally, the identification of a selected compressor and / or compression configuration for multiple attributes in the data file (e.g., specifying the use of a compressor for multiple attributes). The compression parameter data file may include any information necessary to utilize the compressor, including compressor settings and other information.

[0036]

[0038] In step 180 of the method, the file compression / decompression system compresses data from the received data file to produce a compressed file using the identified compressor and / or compression configuration. This may include compression of data for a first attribute, data for multiple attributes, or data for all attributes in the file. Once the system produces the compressed file, the file may be immediately available, such as by sending, transmitting, or using the compressed file, or the compressed file may be stored in local or remote storage for use in a further step of the method. Many other options are possible.

[0037]

[0039] In step 190 of the method, the generated compression parameter data structure is stored so that the stored file can be retrieved and used for decompressing the compressed data file. The compression parameter data structure may be stored in a remote or local database that is a component of the file compression / decompression system 200 or can communicate with the file compression / decompression system 200. For example, the compression parameter data structure may be stored with the compressed data file or with references to or from the compressed data file.

[0038]

[0040] Referring to Figure 2, in one embodiment, it is a schematic representation of a file compression / decompression system 200. System 200 may be any of the systems described or otherwise assumed herein and may comprise any of the components described or otherwise assumed herein. It will be understood that Figure 2 constitutes an abstract in some respects, and that the actual organization of the components of system 200 may differ and be more complex than shown.

[0039]

[0041] According to one embodiment, the system 200 includes a processor 220 capable of executing instructions stored in memory 230 or storage 260, or otherwise processing data to perform, for example, one or more steps of a method. The processor 220 may be formed from one or more modules. The processor 220 may take any preferred form, but is not limited to, a microprocessor, a microcontroller, multiple microcontrollers, a circuit, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a single processor, or multiple processors.

[0040]

[0042] Memory 230 can take any preferred form, including non-volatile memory and / or RAM. Memory 230 may include various types of memory, such as L1, L2, or L3 caches or system memory. Thus, memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM), or other similar memory devices. The memory can, among other things, store the operating system. RAM is used by the processor for temporary storage of data. According to one embodiment, the operating system may include code that controls the operation of one or more components of the system 200 when executed by the processor. In embodiments in which the processor implements one or more of the functions described herein in hardware, it will be apparent that the software described as corresponding to such functions in other embodiments may be omitted.

[0041]

[0043] The user interface 240 may include one or more devices for enabling communication with the user. The user interface may be any device or system that enables the transmission and / or reception of information, and may include a display, mouse, and / or keyboard for receiving user commands. In some embodiments, the user interface 240 may include a command-line interface or a graphical user interface that can be presented to a remote terminal via the communication interface 250. The user interface may be located together with one or more other components of the system, or it may be located remotely from the system and communicate via a wired and / or wireless communication network.

[0042]

[0044] The communication interface 250 may include one or more devices to enable communication with other hardware devices. For example, the communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Furthermore, the communication interface 250 may implement a TCP / IP stack for communication according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the communication interface 250 will become apparent.

[0043]

[0045] The storage 260 may include one or more machine-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media. In various embodiments, the storage 260 may store instructions for execution by the processor 220, or data to which the processor 220 can operate. For example, the storage 260 may store an operating system 261 for controlling various operations of the system 200.

[0044]

[0046] It will become clear that various types of information described as being stored in storage 260 may be stored in memory 230 as an addition or replacement. In this respect, memory 230 may be considered to constitute a storage device, and storage 260 may be considered to be memory. Various other arrangements will become clear. Furthermore, both memory 230 and storage 260 may be considered to be non-temporary machine-readable media. As used herein, the term non-temporary will be understood to include all forms of storage, including both volatile and non-volatile memory, but excluding temporary signals.

[0045]

[0047] Although System 200 is shown as including one of the components described, various components may overlap in various embodiments. For example, Processor 220 may include multiple microprocessors configured to perform the method described herein individually, or to perform steps or subroutines of the method described herein so that multiple processors cooperate to achieve the function described herein. Furthermore, if one or more components of System 200 are implemented in a cloud computing system, various hardware components may belong to separate physical systems. For example, Processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0046]

[0048] According to one embodiment, the file compression / decompression system 200 includes or communicates directly or indirectly with a file database 270. The file database includes one or more files, among possible data, that can or have been compressed using the file compression / decompression system 200. According to one embodiment, one or more files include a genome file, which includes gene sequence variations having millions of rows and numerous columns, containing data attributes for potentially many different genome samples.

[0047]

[0049] According to one embodiment, the file compression / decompression system 200 includes or communicates directly or indirectly with a compression and decompression algorithm database 280. The compression and decompression algorithm database includes a library of different compression and decompression algorithms. These compression and decompression algorithms can be used by a data file system to compress and decompress data files. New compression and decompression algorithms can be added to the database at any time.

[0048]

[0050] According to one embodiment, the storage 260 of the system 200 may store one or more algorithms, modules, and / or instructions for performing one or more functions or steps of the methods described herein or otherwise assumed. For example, among other instructions or data, the storage 260 may include an attribute identification instruction 262, a compressor formulation instruction 263, a compressor selection instruction 264, a storage instruction 265, and / or a reporting instruction 266.

[0049]

[0051] According to one embodiment, the attribute identification command 262 instructs the system to identify one attribute from among several different attributes in a data file received by the system. The attribute may be any information from the data file. The file data system may be configured to recognize known attributes in the data file, or to predict, estimate, or otherwise determine new or unrecognized attributes in the data file. Thus, the file data system may be configured to identify a first attribute from among several different attributes in the data file using a wide variety of different methods.

[0050]

[0052] According to one embodiment, the compressor build instruction 263 instructs the system to select multiple compressors from a predefined set of compression types, algorithms, and / or compression configurations. The compression types, algorithms, and / or compression configurations may be selected locally or remotely from any source, including, but not limited to, a compression and decompression algorithm database 280. The compressor build instruction 263 also instructs the system to individually compress some or all of the attribute data using the identified compressors and / or compression configurations. This individual compression by the multiple compressors may be performed sequentially or simultaneously. According to one embodiment, each compressor compresses only a subset of the first attribute data, and the subsets are equivalent for each compressor to allow for a more objective comparison of compression metrics among the multiple compressors. According to another embodiment, each compressor compresses all of the first attribute data.

[0051]

[0053] According to one embodiment, the compressor construction instruction 263 also instructs the system to measure, or otherwise acquire or identify, one or more compression metrics for each individual compression by each compressor. This enables an objective comparison of compression performance among multiple compressors and / or compression configurations, and the selection of a preferred compressor and / or compression configuration for first attribute data. According to one embodiment, the compression metrics may be compression ratio, processing time, compression accuracy, and / or any other compression metrics.

[0052]

[0054] According to one embodiment, the compressor selection command 264 instructs the system to use measured or acquired metrics to determine which of a selection of compression types, algorithms, and / or compression configurations is most suitable for compression. The file compression / decompression system may compare, rank, or otherwise utilize the metrics corresponding to each individual compressor and / or compression configuration to select the preferred compressor and / or compression configuration. For example, a compressor and associated configuration may be most suitable or preferred for compression when the compression involves a higher or better compression ratio and / or faster processing time than other compression types among the selection of compression types. According to one embodiment, different compressor configurations and corresponding metrics can be presented to the user via a user interface to facilitate manual selection of the compressor to be applied to an attribute.

[0053]

[0055] According to one embodiment, the storage instruction 265 instructs the system to generate a compressed parameter data structure, such as a compressed parameter data file. According to one embodiment, the generated compressed parameter data structure includes the identification of a selected compressor and / or compression configuration for at least a first identified attribute, and / or specifications for such a compressor and / or compression configuration. The compressed parameter data file may include any information necessary to utilize the compressor, including compressor settings and other information. The storage instruction 265 can also instruct the system to store the generated compressed parameter data structure so that the stored file can be retrieved and used for decompressing a compressed data file. The compressed parameter data structure may be a component of the file compression / decompression system 200, or may communicate with the file compression / decompression system 200, and may be stored in a remote or local database. For example, the compressed parameter data structure may be stored together with the compressed data file, or with references to or from the compressed data file.

[0054]

[0056] According to one embodiment, the reporting command 266 instructs the system to provide information about compression, data files, and / or the compressor via the system's user interface 240. The information provided may be any information described or otherwise assumed herein. The system may provide the user with this information through any mechanism, including, but not limited to, visual displays, audible notifications, text messages, emails, pages, or any other notification method.

[0055]

[0057] Uniform interface for specifying compressors

[0058] According to one embodiment, the file compression / decompression system includes a compressor parameter set. The compressor parameter set is a data structure containing compressor configurations, which include all instructions and default parameters required for encoding and decoding attribute data, and which should be stored together with the compressed data. Each compressor configuration carries a unique ID to be used to specify the association of each compressor configuration with an attribute, and a sequence of compressor steps consisting of one or more of the following elements:

[0056]

[0059] (i) A unique ID for the compressor step.

[0057]

[0060] (ii) The type and ID of the algorithm to be applied in step. According to one embodiment, there are two broad types of algorithms, namely transformation and compression, which involve changing the format of the data representation and transforming fixed-length symbols into a compressed bitstream by probabilistic modeling. The algorithm ID is a predefined code that refers to a known algorithm of the specified type. All referenced algorithms must be installed and made accessible to the encoder or decoder in order to process the data.

[0058]

[0061] (iii) A list of default parameters for the algorithm. Each parameter consists of a parameter name, a data type, the number of values ​​in the parameter, and a parameter value.

[0059]

[0062] (iv) The number of input variables to the algorithm and the names of the input variables. Note that the variable names are used only to refer to the data objects that should be passed from one compressor step to another. For the algorithm, it is the order of the input data objects, specified by the order of the recognized input variable names. For the first step, the input data must be raw attribute values, so it is not necessary to specify input variable names, and the number of input variables can be set to 0. From the second step onward, each input variable name must match the output variable name in the previous step.

[0060]

[0063] (v) The number of output variables for holding an ordered list of data objects generated by the algorithm, as well as the state and name of each output variable. If an output variable does not require further processing, its state is complete, and the data object should proceed directly to the compressor's output buffer, so its variable name can be omitted. If an output variable needs to be processed by a subsequent step, its variable state is incomplete, and the variable name must be specified.

[0061]

[0064] (vi) The number of dependency attributes whose values ​​are required to execute the algorithm. The IDs of the dependency attributes should be given during the configuration of the associated attributes.

[0062]

[0065] Referring to Table 1, an exemplary syntax for a compressor parameter set capturing the key elements of the compressor configuration described above, followed by a semantic explanation. Note that the names, ordering, and data types of individual fields can be changed without affecting functionality. Regarding data type notation, u(n) represents an unsigned integer using n bits, u(v) represents an unsigned integer with a number of bits corresponding to the values ​​of other syntax elements, f(n) represents a fixed pattern bit string using n bits, and st(v) represents a null-terminated string with variable length.

[0063]

[0066] Table 1. Compressor parameter set syntax [Table 1]

[0064]

[0067] According to one embodiment, the compressor parameter set semantics are as follows:

[0065]

[0068] The compressor_ID is a unique identifier for the compressor in the annotation table. Note that a value of 0 is reserved for no compression.

[0066]

[0069] n_compressor_steps is the total number of processing steps in this compressor configuration. For each compressor step i, the following fields are specified: (i) compressor_step_ID[i] is the identifier of the compressor step. (ii) algorithm_ID[i] is the identifier of the algorithm to be applied in this step. Each algorithm type should have its own ID map. (iii) use_default_pars[i] is a flag; if set to 1, it indicates that the algorithm's default parameters will be used, and therefore no changes to the parameters are necessary.

[0067]

[0070] n_in_vars[i] is the number of input variables for the algorithm. In the first step (i==0), the input must be a raw attribute value, so its value should be set to 0.

[0068]

[0071] For each input variable j for the algorithm, the following fields are specified: (i) in_var_ID[i][j] is a predefined ID of an input variable known to the algorithm, and (ii) prev_step_ID[i][j] and prev_out_var_ID[i][j] are the ID of the previous compressor step and the ID of the corresponding output variable that contains the data to be passed to the current algorithm through the input variable specified by in_var_ID[i][j], respectively.

[0069]

[0072] n_completed_out_vars[i] is the number of output variables for which the algorithm has completed and which can proceed directly to the compressor's output buffer without requiring further processing.

[0070]

[0073] completed_out_var_ID[i][j] specifies a predefined ID of an output variable known to the algorithm that is complete and does not require further processing.

[0071]

[0074] `nesting_zero_bit` is a single bit set to 0 that is inserted as needed to ensure byte alignment of the data structure.

[0072]

[0075] According to one embodiment, the algorithm parameter is a data structure for specifying the parameter settings of the algorithm referenced by the algorithm_ID. Referring to Table 2, there is a table of exemplary syntax for the algorithm parameter.

[0073]

[0076] Table 2. Algorithm parameter syntax [Table 2]

[0074]

[0077] According to one embodiment, the algorithm parameter semantics are as follows:

[0075]

[0078] n_pars is the number of parameters that need to be modified to run the algorithm referenced by algorithm_ID. Its value must not exceed the number of parameters defined in the algorithm.

[0076]

[0079] For each parameter i being modified, the following fields are required: (i) par_ID[i] is the ID of one of the parameters defined in the algorithm, (ii) par_type[i] is the data type ID of the parameter, (iii) par_num_array_dims[i] is the number of dimensions of the parameter, which is 0 if it is a scalar value, (iv) par_array_dims[i][] contains elements that specify the size of each array dimension. This is omitted if attribute_num_array_dims=0. (v) par_val[i][][][] contains the parameter value. Its number of dimensions is equal to (par_array_dims+1), and extra dimensions can be omitted.

[0077]

[0080] A uniform interface for specifying data attributes and associated compressor configurations.

[0081] According to one embodiment, the file compression / decompression system includes an attribute parameter set. The attribute parameter set is a data structure that includes the configuration of an attribute, which includes some basic information and a compressor configuration for processing it. The following are important elements: (i) the unique ID and name of the attribute, (ii) data type information, as well as the default and missing values ​​of the attribute, (iii) the ID of the compressor for processing the attribute, which must be one of the compressor IDs defined in the compressor parameter set, (iv) the modification of the selected parameters of the compressor step, and (v) the ID of a dependency variable (attribute or descriptor) whose data is required for processing by several compressor steps.

[0078]

[0082] Refer to Table 3, which shows an illustrative syntax of an attribute parameter set that captures the key elements of the attribute configuration described above, with a semantic explanation provided after the table. Note that the names, ordering, and data types of individual fields can be changed without affecting functionality. The data type notation is the same as in Table 1. The attribute parameter set includes definitions of additional attributes, including those parameters, grouping, ordering, and associated compressors.

[0079]

[0083] Table 3. Attribute Parameter Set Syntax [Table 3]

[0080]

[0084] According to one embodiment, the attribute parameter set semantics are as follows:

[0081]

[0085] `attribute_ID` is a unique attribute identifier within the annotation parameter set.

[0082]

[0086] `attribute_name_len` is the length of the attribute name.

[0083]

[0087] `attribute_name` is the name of the attribute with length `attribute_name_len`.

[0084]

[0088] `attribute_type` specifies the data type of the attribute.

[0085]

[0089] `attribute_num_array_dims` specifies the number of array dimensions for attribute entries; it is 0 if it is a scalar value.

[0086]

[0090] The `attribute_array_dims[]` array contains elements that specify the size of each array dimension. This is omitted when `attribute_num_array_dims=0`.

[0087]

[0091] `attribute_default_value` is the default value for the attribute (i.e., it can be used for sparse coding when most values ​​that are equal to the default are excluded).

[0088]

[0092] `attribute_missing_value` is the missing value of the attribute that should be used in place of a null value in the output after the decoding process.

[0089]

[0093] The compressor_ID is the compressor_ID defined in the compressor_parameter_set used for processing the current attribute.

[0090]

[0094] n_steps_with_dependencies is the number of steps in the compressor, referenced by the compressor_ID with its dependency variables.

[0091]

[0095] For each compressor step i that has dependency variables that need to be specified, (i) compressor_step_ID[i] is the ID of the compressor step, and (ii) n_dependencies[i] is the number of dependency variables required by the algorithm of the compressor step.

[0092]

[0096] For each dependency variable j in the algorithm, (i) dependency_var_ID[i][j] is a predefined ID of the dependency variable known to the algorithm, (ii) dependency_is_attribute[i][j] is a flag that, if set to 1, indicates that the dependency data is stored in an attribute, and otherwise, the dependency data is stored as a descriptor, and (iii) dependency_ID[i][j] specifies the ID of the attribute or descriptor that contains the dependency data. Note that the length of the ID is 16 bits for attributes and 8 bits for descriptors.

[0093]

[0097] `nesting_zero_bit` is a single bit set to 0 that is inserted as needed to ensure byte alignment of the data structure.

[0094]

[0098] Encoding algorithms based on compressor configuration

[0099] According to one embodiment, encoding of raw attribute data is performed by sequential execution of compressor steps defined in the relevant compressor configuration. The encoding algorithm is outlined below.

[0095]

[0100] 1. Extract the configuration (parameter set) of the attribute being processed and its associated compressor.

[0096]

[0101] 2. Obtain a compressor configuration customized for an attribute by overriding the default parameter values ​​from the compressor configuration using attribute-specific parameter values ​​in the attribute configuration.

[0097]

[0102] 3. Each compressor step is executed continuously from beginning to end. For each compressor step i, a. Look up the conversion / compression algorithm using the algorithm ID. b. If the algorithm ID cannot be found, the user will be prompted to grant permission to download and install the corresponding algorithm from an authenticated software repository via a secure transmission channel. c. If there are dependency variables defined for step, obtain the decoded data of the dependency variables from the corresponding attribute or descriptor specified by dependency_is_attribute[i][] and dependency_ID[i][]. d. Apply the transformation / compression algorithm to the input variable data object using the parameter values ​​associated with the step and the data objects of the dependent variables. If it is the first step, the raw attribute data is the only input data object being processed. The generated output consists of a list of data objects and their corresponding variable IDs. e. If an output variable is specified as complete, its data object should be placed in the compressor's output buffer. Otherwise, its data object should be stored in a data map so that it can be looked up by subsequent steps through its step and variable ID for further processing. Note that each encoded data block / object is stored using a typed_data structure, as specified herein, to store its data type and array format.

[0098]

[0103] Table 4. Typed Data Syntax [Table 4] f. In the final step, return the contents of the output buffer, which is a concatenation of encoded blocks.

[0099]

[0104] Table 5 shows pseudocode for the attribute data encoding algorithm described above, based on the relevant compressor configuration according to one embodiment.

[0100]

[0105] Table 5. Pseudocode of attribute data encoding algorithms [Table 5] TIFF0007893250000006.tif225170TIFF0007893250000007.tif59170

[0101]

[0106] Decoding algorithm based on compressor configuration

[0107] According to one embodiment, the file compression / decompression system includes an attribute decryption process. Decryption of the compressed attribute data involves the reverse of the compression steps defined in the relevant compression configuration, by a stepwise inverse operation of the algorithm starting from the last step and going back to the first step. The decryption algorithm is outlined below.

[0102]

[0108] 1. Extract the configuration (parameter set) of the attribute being processed and its associated compressor.

[0103]

[0109] 2. Extract one or more encoded blocks from the compressed attribute stream, each corresponding to a completed output variable defined in the compressor step. Each encoded block is stored using the typed_data structure specified in Table 4 to preserve its data type and array format.

[0104]

[0110] 3. Starting from the last step, perform the reverse operation of each compressor step in reverse order until reaching the first step. For each compressor step i, a. Collect all input variable data for the current step from either the encoded block in step 2 above, or the output variable of its upstream decoding step. b. If there are dependency variables defined for the current step, obtain the decoded data of the dependency variables from the corresponding attribute or descriptor specified by dependency_is_attribute[i][] and dependency_ID[i][]. c. Extract the changes from the compressor parameter set to the default parameter settings for the current step. d. Using the relevant parameter settings and any dependent variable data, apply the inverse operation of the specified algorithm for the current step to the input variable data. e. If it has not reached the first step, use the step ID and variable ID pair as the key for the lookup to store the output variable data that will be further processed by the subsequent decoding step. f. If it has reached the first step, fill in any missing values ​​and return the inverse operation's output variable data as attribute data decoded by the compressor.

[0105]

[0111] The steps of the attribute decoding process based on the relevant compressor configuration are given in the table below. Note that the initial value of attribute_stream is the encoded attribute data in the tile referenced by its index. Other variables not declared or initialized in the code are defined in the compressor parameter set referenced by the compressor ID associated with the attribute.

[0106]

[0112] Table 6. Attribute Decryption Process [Table 6] TIFF0007893250000009.tif232170TIFF0007893250000010.tif75170

[0107]

[0113] Mechanism for automatic determination of compressor configuration

[0114] For well-known and popular file types, a group of data compression experts can pre-define a compressor configuration and adopt it for community use. However, there are times when specifying a compressor configuration is necessary, for example, when dealing with new or exclusive file types, or when a known file type allows for the inclusion of unintended attributes. One possible approach is to have the user manually specify the compressor configuration, sometimes with the help of a graph tool that provides different options and explanations. This approach, however, requires considerable user knowledge and experience regarding data compression and often results in suboptimal configurations.

[0108]

[0115] Therefore, a mechanism is provided for the automatic determination of a performance-based compressor configuration. This system may include a list of available transformation and compression algorithms that are available for selection, for each of which the types of data that the algorithm can process and the possible connections between the algorithms are known. The mechanism can work as follows: For each attribute in the file,

[0116] 1. Identify all possible compressor configurations for the attribute, given that the available algorithms and additional constraints such as the maximum allowed number of compressor steps and prohibition of repetition in the step sequence are known.

[0117] 2. Evaluate the performance of each compressor configuration using metrics such as compression ratio and processing time. This can be done on a subset of data from the files being compressed, or on training files to build a compressor configuration template for a specific file type. To speed up the process, evaluations can be performed in parallel.

[0118] 3. Select the best compressor configuration for the attribute based on performance criteria, such as a compressor configuration with the highest compression ratio and a processing time that places it in the first quarter of all configurations.

[0119] After selecting the best compressor configuration for each attribute,

[0120] 4. Generate an attribute and compressor parameter set data structure containing all the instructions and parameters required for file processing.

[0121] 5. If the generated attributes and compression configuration are applicable to other files of the same type, their data structures can be stored in the repository and looked up for future use by file type or file extension.

[0122] 6. To compress the file, the encoder can use the generated configuration to apply the above-described encoding algorithm to each attribute and store the configuration data structure along with the compressed attribute data.

[0123] 7. To decrypt a compressed file, the decoder can use the configuration data associated with the file to apply the above-described decryption algorithm to each attribute.

[0109]

[0124] Example 1 - Compressor configuration for sparse representation data

[0125] The following is a non-restrictive example of applying attributes and compressor configurations for compressing sparse single-cell RNA representation data. Two compressor configurations were evaluated.

[0110]

[0126] Table 7 - Compressor configuration 1 for compressing single-cell represented data [Table 7]

[0111]

[0127] In the case of Compressor Configuration 1 summarized in Table 7, the input matrix of representation values ​​first undergoes a sparse transformation that registers only cells with non-zero values, generating three output variables: row_idx, col_idx, and value. Then, BSC compression is applied individually to each of the output variables from step 1, resulting in an encoded data stream consisting of three data blocks concatenated in the order they were defined in the compressor step.

[0112]

[0128] Table 8 - Compressor configuration 2 for compressing single-cell represented data [Table 8]

[0113]

[0129] In the case of compression configuration 2 summarized in Table 8, as in the first configuration, the input matrix of the representation values ​​first undergoes a sparse transformation that generates three output variables, namely row_idx, col_idx, and value. A delta transformation is applied to row_idx to generate the delta value stored in the variable row_idx_delta, and the delta value is then compressed by the BSC algorithm. Run-length encoding (RLE) is applied to col_idx to generate two output variables, namely col_idx_val and col_idx_count, and col_idx_val and col_idx_count are then compressed individually by BSC. The data in the value variable from step 1 is compressed directly by BSC. This results in an encoded data stream consisting of four data blocks concatenated as they were defined in the compressor step.

[0114]

[0130] To evaluate performance, two compression configurations were applied to single-cell RNA representation data stored in a Market Matrix format file, which had a size of 152 MB before compression. After direct compression with Gzip and BSC, the file sizes were reduced to 47 MB ​​and 42 MB, respectively.

[0115]

[0131] Table 9 summarizes the various data sizes after applying compressor configurations 1 and 2 using the encoding algorithm. For a 42MB file size after direct compression with BSC, the compression ratios of compressor configurations 1 and 2 are 5.19 and 3.68, respectively. Since compressor configuration 1 has better performance in terms of compression ratio, it should generally be chosen for compressing sparse representation data.

[0116]

[0132] Table 9 - Compressed data size for Compressor configurations 1 and 2 [Table 9]

[0117]

[0133] This embodiment demonstrates that the present method significantly reduces the size of compressed data files compared to prior art methods (such as direct compression by BSC). This improves the functionality and efficiency of file compression / decompression systems and revolutionizes the storage capacity of computer systems or databases. Furthermore, when processing files containing millions or billions of data points for each of one or more attributes, compression involves millions or billions of calculations that are beyond the capabilities of the human mind even with pen and pencil.

[0118]

[0134] It should be understood that all definitions defined and used herein govern dictionary definitions, definitions in documents incorporated by reference, and / or the ordinary meanings of the terms defined.

[0119]

[0135] As used herein in the specification and claims, singular elements should be understood to mean “at least one” unless otherwise clearly indicated.

[0120]

[0136] As used herein in the specification and claims, the phrase “and / or” should be understood to mean “either or both” of the elements thus combined, i.e., elements that are sometimes conjunctive and, in other cases, disjunctive. Multiple elements described using “and / or” should similarly be interpreted as “one or more” of the elements thus combined. Other elements other than those explicitly identified by the “and / or” clause may be present, whether related to those explicitly identified elements or not.

[0121]

[0137] When used herein in the specification and claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as inclusive, that is, including several elements or lists of elements, and optionally, at least one of several unlisted items or lists of such items, but also including two or more of them. Only terms that are clearly indicated in the opposite way, such as “exactly one of” or “strictly one of” or, when used in the claims, “consisting of,” refer to including exactly one element of several elements or lists of elements. In general, the term “or” as used herein shall be interpreted only as indicating an exclusive substitution (i.e., “one or the other, but not both”) when followed by terms of exclusivity, such as “either,” “one of,” “exactly one of,” or “strictly one of.”

[0122]

[0138] When used herein in the specification and claims, the phrase “at least one” means, with respect to a list of one or more elements, at least one element selected from any one or more elements in the list of elements, but it should be understood that this does not necessarily include at least one of all elements explicitly stated in the list of elements, and does not necessarily exclude any combination of elements in the list of elements. This definition also allows for the existence of elements other than those explicitly identified in the list of elements to which the phrase “at least one” refers, whether or not they are related to those explicitly identified elements.

[0123]

[0139] Furthermore, unless otherwise clearly indicated, it should be understood that in any method claimed herein, including two or more steps or actions, the order of steps or actions of a method is not necessarily limited to the order in which the steps or actions of a method are presented.

[0124]

[0140] In the claims and in the above specification, all transitional phrases such as “having,” “including,” “carrying,” “having,” “containing,” “accompanying,” “holding,” and “consisting of” should be understood to be open-ended, meaning “not limiting, but including.” Only the transitional phrases “consisting of” and “essentially consisting of” are closed or semi-closed transitional phrases, respectively.

[0125]

[0141] While several inventive embodiments have been described and illustrated herein, those skilled in the art will readily conceive of various other means and / or structures for performing the functions described herein and / or obtaining one or more of the results and / or advantages described herein, and each of such changes and / or modifications will be considered to fall within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials, and configurations described herein are illustrative, and that actual parameters, dimensions, materials, and / or configurations depend on the specific application or application for which the inventive teaching is used. Those skilled in the art will be able to recognize or verify many equivalents of the specific inventive embodiments described herein by means of routine experimentation. It should be understood that the above embodiments are presented merely as examples, and that inventive embodiments other than those specifically described and claimed may be implemented within the scope of the appended claims and their equivalents. The inventive embodiments of this disclosure cover each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the inventive scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not inconsistent with each other.

Claims

1. A method for compressing a genome data file using a file compression system to generate a compressed genome data file, wherein the method is: The steps include receiving a genome data file for compression that contains multiple different attributes, The steps include identifying a first attribute from among the plurality of different attributes, and storing the specifications of the identified first attribute in an attribute parameter data structure. The steps include selecting a plurality of compression types and / or compression configurations from a plurality of predetermined compression types and compression configurations based on the identified first attribute, A step of individually compressing at least a portion of the data from the received genome data file for the identified first attribute using each of the selected compression types and / or compression configurations, wherein one or more metrics are measured for each individual compression. A step of determining which of the selected compression types and / or compression configurations is most suitable for compression based on the one or more metrics of each of the individual compressions, A step of generating a compression parameter data structure having an identification of the compression type and / or compression configuration most suitable for compressing the identified first attribute from among the selected plurality of compression types and / or compression configurations, wherein the compression parameter data structure is stored together with the compressed genome data file for a plurality of compression steps to be computed by sequencing. The unique ID of the compressor step; The type and ID of the algorithm to which the compressor step should be applied; The number of input variables, and the names of the input variables for the compressor steps after the first compressor step; The number of output variables, the status of the output variables, and the names of the output variables whose status is not yet complete; A step of encoding at least one compression configuration including, To generate a compressed genome data file, the steps include compressing the data from the received genome data file for the first attribute using the identified compression type and / or compression configuration, A step of storing the attribute parameter data structure, the compressed parameter data structure, and the compressed genome data file, A method having.

2. The method according to claim 1, wherein the one or more metrics include one or more of the compression ratio and processing time for the compression.

3. The method according to claim 1 or 2, further comprising the step of storing a list of default parameters for the algorithm in the compression configuration.

4. The method according to claim 1, further comprising the step of providing information about the generated compressed parameter data structure and / or the compressed genome data file via a user interface.

5. The method according to claim 1, further comprising the step of providing a user with information about the identified compressor type and / or compressor configuration via a user interface so that the user can select a compressor type and / or compressor configuration to be applied to compress attributes.

6. The method according to claim 1, further comprising the step of communicating with a compression and decompression algorithm database having a library of different compression and decompression algorithms.

7. A genome data file for compression containing multiple different attributes, Processor and A system for compressing genome data files, including, The aforementioned processor, (i) Identifying a first attribute among the plurality of different attributes, and storing the specifications of the identified first attribute in an attribute parameter data structure, (ii) Selecting a plurality of compression types and / or compression configurations from a plurality of predetermined compression types and / or compression configurations based on the identified first attribute, (iii) Individually compressing at least a portion of the data from the received genome data file for the identified first attribute using each of the selected compression types and / or compression configurations, wherein one or more metrics are measured for each individual compression, (iv) Determining which of the selected compression types and / or compression configurations is most suitable for compression based on one or more metrics of each of the individual compressions, (v) To generate a compression parameter data structure having an identification of the compression type and / or compression configuration that is most suitable for compressing the identified first attribute from among the selected plurality of compression types and / or compression configurations, (vi) Compressing the data from the received genome data file for the first attribute using the identified compression type and / or compression configuration in order to generate a compressed genome data file, (vii) storing the attribute parameter data structure, the compressed parameter data structure, and the compressed genome data file, It is configured to do the following: The processor provides for a plurality of compressor steps to be calculated in sequence, The unique ID of the compressor step; The type and ID of the algorithm to which the compressor step should be applied; The number of input variables, and the names of the input variables for the compressor steps after the first compressor step; The number of output variables, the status of the output variables, and the names of the output variables whose status is not yet complete; A system configured to encode at least one compression configuration, including the following, in the compression parameter data structure.

8. The system according to claim 7, further comprising a user interface configured to provide information about the generated compressed parameter data structure and / or the compressed genome data file.

Images