Method for efficient data compression in MPEG-G, genome encoder, genome decoder and computer-readable medium

Abstract

1. A computer-implemented method for storing or transmitting a representation of genome sequencing data in a genome file format including annotation data associated with the genome sequencing data, the genome sequencing data including nucleotide sequence reads, the method comprising: aligning (10) the reads to one or more reference sequences to generate aligned reads; classifying (14) the aligned reads according to classification rules based on mapping of the aligned reads to the one or more reference sequences to create (18) classes of the aligned reads; entropy coding the classified aligned reads as a block of multiple descriptors; structuring the blocks of descriptors with header information to create a first class of access units (119) containing genome sequencing data; The method further includes encoding the annotation data (12) into different access units (122) of a second class and indexing the data into a master annotation index (MAI, 123, 211), the indexed data representing a coded form of the annotation string data obtained by using at least one compressed string indexing algorithm (28) on the annotation string data (212), the MAI associating the coded annotation strings with the access units of the second class; The method further includes jointly coding the first and second category of access units and the MAI.

Description

[Technical Field]

[0001] The present invention relates to the field of MPEG-G data compression.

[0002] MPEG, the Moving Picture Experts Group (MPEG), is a working group of data compression experts formed by ISO and IEC to set standards for audio and video compression and transmission.

[0003] This group has been developing standards for video-efficient video compression since the early 1990s. MPEG techniques essentially consist of reducing the entropy of video and audio source data, allowing for high compression ratios for efficient storage and transmission.

[0004] Because there is significant expertise in data compression within the MPEG expert group, it was decided to develop a standard for the compression of genomic information in order to overcome the limitations of existing solutions in the art (e.g., CRAM and BAM file formats).

[0005] Thus, even though MPEG-G relates to the compression of genomic data, the main idea of ​​exploiting data redundancy is taken from the field of video and audio compression, the technical field closest to the present application.

[0006] In effect, the present invention applies syntax element construction to genomic data in a similar manner to how syntax elements are applied to the compression of video and audio data in MPEG.

[0007] However, given the fact that genomic data is quite different from audio and video data, data classification and syntax elements differ from those used in MPEG video and audio standards. Indeed, it is necessary to take advantage of the redundancies present in genomic data, which are different from multimedia data.

[0008] Thus, the present invention deals with the compression of genomic data in an efficient way, to obtain files of reduced size and that are easy to randomly access even in the compressed domain.

[0009] The present invention is based on the encoding and decoding methods, systems, and computer programs disclosed in patent applications WO2018 / 068827A1, WO2018 / 068828A1, WO2018 / 068829A1, and WO2018 / 068830A1. These disclosures regarding entropy coding of genomic data may be essential for understanding some aspects of the present invention. Therefore, the disclosures of the aforementioned documents are deemed to be incorporated into the present invention by reference.

[0010] The present disclosure provides a novel method for representing annotations and metadata associated with genome sequencing data that reduces storage space used, provides a single syntax for several metadata formats, and improves data access performance by providing new indexing capabilities not available in known prior art representation methods.

[0011] The methods disclosed in this invention provide higher compression rates for genome sequencing data and associated annotations by: Representing genome sequencing data and associated annotations in terms of the syntax of numeric and textual descriptors defined in this disclosure Compressing non-indexed text descriptors separately from indexed ones Applying non-indexed descriptor transformations such as differential coding, run-length coding, byte separation, and entropy coders such as CABAC, Huffman coding, arithmetic coding, and range coding Applying compressed full-text string indexing algorithms, such as compressed string pattern matching data structures, compressed suffix arrays, FM indexes, and hash tables, to indexed text descriptors by eliminating the redundancy of having both an index and a compressed payload, as in existing methods.

[0012] The advantage of compressing non-indexed descriptors separately from indexed text descriptors is that these two classes of data, when grouped separately, exhibit lower entropy than when coded together, allowing for higher compression ratios.

[0013] By using a compressed full-text string indexing algorithm, the method described in this invention eliminates the need to have both a compressed payload of genomic information to support selective access and an index of the information, thus achieving better compression ratios. The compressed full-text string indexing algorithm is both an index and compressed information that can be used both to perform selective access and to decompress to obtain the desired information. The present invention overcomes the need to have both an index and a compressed payload currently required by existing solutions in the art.

[0014] This method also allows previously unrelated concepts related to genome annotation to be described hierarchically and stored in a compressed format, making it possible to encode previously unaccountable relationships between concepts and enabling new ways of describing and exchanging data. [Background technology]

[0015] Genomic or proteomic information generated by DNA, RNA, or protein sequencing machines undergoes various stages of data processing, resulting in heterogeneous data. In prior art solutions, this data is currently stored in computer files with different and unrelated structures. This makes archiving, transferring, and elaborating this information extremely difficult.

[0016] Genomic or proteomic sequences referred to in the present invention include, but are not limited to, for example, nucleotide sequences, deoxyribonucleic acid (DNA) sequences, ribonucleic acid (RNA), and amino acid sequences.

[0017] Sequence alignment refers to the process of arranging sequence reads by finding regions of similarity that may be the result of functional, structural, or evolutionary relationships between sequences. When alignment is performed with reference to an existing nucleotide sequence, called a "reference sequence," the process is called "mapping." Prior art solutions store such information in "SAM," "BAM," or "CRAM" files. The process of performing sequence alignment is also called "alignment."

[0018] The concept of aligning sequences to reconstruct partial or complete genomes is shown in Figure 2 of WO2018068827 A1, the disclosure of which is incorporated herein by reference.

[0019] There is a clear need to provide an appropriate genome sequencing data and metadata representation (Genomic File Format) by organizing and partitioning the data, which maximizes data and metadata compression and efficiently enables several features such as support for selective access and incremental updates, as well as data processing capabilities that are useful at various stages in the genomic data lifecycle.

[0020] Furthermore, as genome sequencing data generated by high-throughput sequencing machines is analyzed by processing pipelines and analysts, annotations of various regions of the genome are generated, expressing numerous and diverse characteristics, currently expressed in heterogeneous text formats. While the various types of generated results and annotations are conceptually related to each other and ideally should be accessed and used collaboratively, current solutions used in the art place these metadata in the form of independent, separate text files, separate from the coded data associated with the genome. These formats do not support any type of link between elements in one file and elements in other files that are conceptually linked, potentially sharing common biological meaning.

[0021] At best, the lack of such explicit connections means that processing and using genomic data and annotation information is highly inefficient when searching for specific information and associated metadata, possibly through time-consuming parsing of large text files. At worst, the inability to describe connections hinders the development of effective bioinformatics workflows and databases for downstream applications such as biomedical research and personalized medicine.

[0022] For example, RNA-seq reads aligned to a gene (typically consisting of a series of intervals on a reference genome) must be counted to measure the degree of expression of the gene in the biological conditions used in the experiment. Different biological conditions (generating different sets of reads generated by different experiments) are typically compared in the context of a specific experiment, with the goal of finding paths linking genotype to phenotype. The process of generating and aggregating information related to single reads and their alignment to a reference genome into results with more general genetic and biological meaning is called "secondary analysis."

[0023] Various types of annotations (meta-information) generated by secondary analyses using genome sequence reads can be conceptually associated with genome sequence reads aligned to one or more intervals of the genome sequence used as a reference.

[0024] A genomic interval can be uniquely identified by specifying a reference assembly (i.e., a chromosome, gene, set of consecutive bases, single base, etc. within the genome), a forward or reverse molecular strand, and a sequence of nucleotides at the start and end positions that specify the range of bases (a.k.a. nucleotides) included in the interval. [Table 1]

[0025] Genomic interval features such as variants, the number of aligned reads at a particular position (also known as "coverage"), parts of the genome to which proteins bind, the nature and location of genes, and regions associated with specific genetic functions are uniquely identified and associated with genomic intervals. Intervals can be as short as a single base or span thousands of nucleotides or more.

[0026] Complex analyses of genome sequencing data can be constructed through the integration of numerous experiments. Different sequencing-derived protocols typically characterize each experiment, often used to sample different cellular functions or compartments. The results generated by each experiment's primary analysis (i.e., alignment of reads to a reference) and secondary analysis (integration and statistical studies performed on the alignment results) can be visualized in graphical form using software applications called genome browsers, allowing one-dimensional navigation of the genome along nucleotide positions. Information derived from secondary analyses associated with each position or interval in the genome is typically visualized in the form of plots (or "tracks"), which are distinct for each sequencing experiment, representing the presence and structure of transcripts, individual or population sequence variants, sequencing read coverage, and the strength of protein binding to each position in the genome.

[0027] Modern genome annotation formats produced by analytical tools represent all the aforementioned results (also known as "features") using a multitude of heterogeneous, independently defined and maintained formats. Such formats are typically characterized by poor and inconsistent syntax and semantics, leading to a proliferation of slightly different, incompatible file formats for each type of analytical result. A drawback of all currently existing solutions is that scientists working on integrated analyses of genomic data who need to collaboratively access and study a set of experiments are forced to systematically transcode the various formats using a complex concatenation of text processing tools and programs. This proliferation of different formats reduces the interoperability and reproducibility of results across different groups of scientists, even when slight differences in representation or associated semantics exist.

[0028] The most used formats for representing genome annotations generated by genome sequencing data analysis and used in the art are: · Variant calling format (VCF) represents variants relative to a reference genome that are present in either a single individual or a population of individuals. Browser Extended Data (BED) A format that supports the representation of data lines that appear in annotation tracks typically displayed in genome browsers. http: / / genome.ucsc.edu / FAQ / FAQformat#format1 · Generic Feature Format (GFF) represents genomic features in a text file characterized by nine columns and tab delimiters. · Gene Transfer Format (GTF) is a backward compatible extension of GFF. The BigWig format is used to display high-density continuous data as graphs in genome browsers. Furthermore, the fact that such heterogeneous data cannot be described by a unified hierarchy means that it is also not possible to describe the relationships between features belonging to different categories at all, making progress in this field even more difficult. Summary of the Invention [Means for solving the problem]

[0029] In order to solve the above-mentioned problems of the existing prior art, the subject matter of claims 1, 9, 12, 14 and 16 is proposed. Advantageous modifications are set out in the dependent claims.

[0030] More specifically, the present disclosure provides a computer-implemented method for encoding, storing, and / or transmitting a representation of genome sequencing data in a genome file format including annotation data associated with said genome sequencing data, said genome sequencing data including nucleotide sequence reads, said method comprising the steps of: A computer-implemented method for encoding, storing, and / or transmitting a representation of genome sequencing data in a genome file format including annotation data associated with the genome sequencing data, comprising: aligning the reads to one or more reference sequences to create aligned reads; classifying the aligned reads according to classification rules based on mapping the aligned reads to the one or more reference sequences to create classes of the aligned reads; entropy-coding the classified aligned reads as blocks of multiple descriptors; structuring the blocks of descriptors with header information to create access units of a first classification including genome sequencing data; the method further comprises encoding the annotation data into different access units of a second classification and indexing the data into a master annotation index, wherein the indexed data represents a coded form of the annotation string data obtained by using at least one compressed string indexing algorithm on the annotation string data, and the MAI associates the coded annotation strings with the access units of the second classification. Preferably, the method further comprises jointly coding the first classification, the access units of the second classification, and the MAI.

[0031] The method may further include storing or transmitting the encoded genome sequencing data to a computer-readable storage medium, or transmitting the genome sequencing data over a data network or another data infrastructure, thereby making the encoded genome sequencing data available to a user in other manners known in the art.

[0032] In the context of the present disclosure, a descriptor may be implemented, for example, as a genome annotation descriptor, as defined in the detailed description below.

[0033] Preferably, the access units of the second class comprising genomic annotation data further comprise information data identifying genomic intervals, which identify nucleotide sequences in one or more reference sequences such that the annotation data contained in the access units of the second class are associated with associated coded reads of the genomic sequences contained in the access units of the first class comprising genomic sequencing data.

[0034] According to a (further) preferred embodiment, encoding the annotation data and indexing data comprises encoding genome annotation data as genome annotation descriptors, said genome annotation descriptors comprising numerical descriptors and textual descriptors, said encoding comprising: - selecting a subset of textual descriptors from said textual descriptors in accordance with configuration parameters provided by a user, - transforming the subset of text descriptors by generating string indices using a first string transformation method; - converting and encoding the string indexes using a string index conversion method to generate master annotation index data; - converting said textual descriptors that are not included in said subset of numeric descriptors and textual descriptors using at least one second conversion method different from the first conversion method; - encoding the numeric descriptors and text descriptors not included in the subset of text descriptors into separate access units of a second classification using at least one first entropy encoder for the numeric descriptors and at least one second entropy encoder for the text descriptors not included in the subset of text descriptors.

[0035] The first character string conversion method includes: - inserting a string terminator character after each text descriptor to signal the end of each text descriptor; - concatenate the text descriptors, - further preferably including interleaving genome annotation record index data to associate said text descriptors with positions of genome annotation records within access units of the second classification.

[0036] (Further) According to a preferred embodiment, the string index transformation method is one of: string pattern matching, suffix array, FM index, hash table.

[0037] Preferably, the at least one second transformation method is one of differential coding, run-length coding, byte separation, and entropy coders such as CABAC, Huffman coding, arithmetic coding, range transformation, etc.

[0038] (Further) According to a preferred embodiment, said Master Annotation Index contains in its header the number of AU types and the number of indices for each AU type.

[0039] More preferably, the above method further comprises coding the classified unaligned reads.

[0040] The object of the present invention is further solved by a method for the sequencing and extraction of nucleotide and genome annotation data encoded according to the method described above, said method comprising: Parsing the genome data multiplex into a genome layer of syntactic elements, Analyze the compressed annotation data, Parse the master annotation index, Expanding the genome layer into classified nucleotide sequence reads; selectively decoding the classified nucleotide sequence reads on one or more reference sequences to generate uncompressed nucleotide sequence reads; and selectively decoding the annotation data associated with the classified reads.

[0041] Preferably, the method further comprises decoding information data relating to genomic intervals that identify nucleotide sequences in the one or more reference sequences such that the annotation data is associated with coded reads of the associated genomic sequence.

[0042] It is further preferred that the method further comprises decoding data encoded according to the method for storing or transmitting a representation of genome sequencing data in a genome file format including annotation data associated with the genome sequencing data described above.

[0043] According to a further aspect of the present disclosure, there is proposed a genome encoder for compressing genome sequence data in a genome file format including annotation data associated with said genome sequencing data, said genome sequence data including nucleotide sequence reads, said encoder comprising: - an alignment unit for aligning said reads to one or more reference sequences, thereby generating aligned reads; - a data classification unit for classifying said aligned reads according to a classification rule based on mapping of said aligned reads on said one or more reference sequences, thereby creating classes of aligned reads; - an entropy coding unit for entropy coding the aligned reads classified into a number of descriptor blocks; - an access unit coding unit that uses header information to construct said block of descriptors, thereby creating an access unit of a first classification that includes genome sequencing data; - a genome annotation encoding unit that encodes annotation data into different access units of a second class and indexes the data into a master annotation index, wherein the indexed data represents a coded form of the annotation string data obtained by using at least one compressed string indexing algorithm on the annotation string data, and wherein the MAI associates the coded annotation strings with the access units of the second class.

[0044] Preferably, the encoding device includes means for jointly encoding the access units of the first and second categories and the MAI.

[0045] According to a (further) preferred embodiment, the genomic encoder comprises encoding means for carrying out the steps of the encoding method described above.

[0046] The present disclosure further refers to a genome decoder device for decoding the nucleotide sequence and genome annotation data encoded by the above encoder, said decoder comprising: - a means for parsing a genomic data multiplex into a genomic layer of syntactic elements; - means for analyzing said compressed annotation data; - means for parsing the master annotation index; - means for extending said genome layer into classified nucleotide sequence reads; - means for selectively decoding said classified nucleotide sequence reads on one or more reference sequences to generate uncompressed nucleotide sequence reads; and - means for selectively decoding the annotation data associated with the classified reads.

[0047] Preferably, the genome decoder further comprises decoding means for carrying out the steps of the above decoding method.

[0048] According to a further aspect of the present disclosure, a computer-readable medium is proposed, the computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform the above-mentioned method.

[0049] term In this disclosure, the following terms and phrases are used:

[0050] Bitstream Syntax: The structure of data coded as a series of bits (also known as a bitstream) in digital data storage or communication applications. The term typically refers to the format of a coded bitstream that is produced by a coding application (also known as an encoder) and processed as input to a decoding application (also known as a decoder) to reconstruct the uncompressed data when compression is used. Bitstream syntax uses several syntactic elements to represent the information coded in the bitstream.

[0051] Syntax element: A component of the bitstream syntax that represents one or more features of the coded information. In the bitstream produced by an encoder, syntax elements may or may not be compressed.

[0052] Source Model: In information theory, the expression "source model" refers to the definition of the sequence of events generated by a source, their context, and the probability associated with each event and the corresponding context. Data compression uses knowledge about the source of the information to be coded to define a source model that can reduce the entropy of the model and, consequently, the number of bits (i.e., code) required to represent the information generated by the source.

[0053] Sequencing data: The set of sequence reads generated by a sequencing protocol.

[0054] Sequencing read (a.k.a. read): In sequencing, a read is an estimated sequence of base pairs (or base pair probabilities) corresponding to all or part of a nucleic acid molecule.

[0055] Genomic interval: A series of bases (also known as nucleotides) contained between a start and end position on a nucleotide sequence, such as a chromosome, gene, transcriptome, or other nucleotide sequence.

[0056] Genomic feature: A set of genomic intervals that share a biological property.

[0057] Annotation Data: Quantitative, qualitative, or sequence information associated with genomic features. These include variants, browser tracks, functional annotations, methylation patterns and levels, sequence coverage and statistics, feature expression matrices, contact matrices, and protein affinity to nucleic acids.

[0058] Functional Annotation: Information related to genomic features, particularly the hierarchy of concepts related to biological transcriptional and translational genomic information (genes, transcripts, exons, coding sequences, etc.). Formats currently used to represent such information include GFF, GTF, BED, and all their derivatives.

[0059] Multiplexer: A coding module that takes as input a large number of access units of various types and produces a structured bitstream for use in streaming or file storage.

[0060] Genome Annotation Record: A data structure consisting of a set of genome annotation descriptors representing a genomic interval and annotation data related to genome functional annotations, browser tracks, genome variants, gene expression information, contact matrices, and other annotations associated with said genomic interval. One genome annotation record can be logically linked to other genome annotation records and related annotations.

[0061] String Data Structure: A data structure used to index strings, possibly in a compressed domain, to allow fast lookup.

[0062] Master Index Table (MIT): An index structure defined in ISO / IEC 23092-1 and W02018068827A1 and W02018152143A1. It is used to associate genomic intervals and classes of coded genome sequence reads with access units. Access units are used to carry compressed reads mapped to that interval and associated metadata.

[0063] Genomic compression of data blocks, access units, genome data layers, and genome data multiplexes The data structure further disclosed by the present invention relies on the following concepts:

[0064] A data block is defined as a set of descriptor vector elements of the same type (e.g., position, distance, reverse complement flag, position and mismatch type) that make up a layer. Typically, one layer is composed of many data blocks. A data block, which is incorporated herein by reference in co-pending patent application no. WO2018068830A1, consists of transmission units with a size typically specified according to communication channel requirements. Such segmentation capability is desirable to achieve transport efficiency using typical network communication protocols.

[0065] An access unit is defined as a subset of genome data that can be fully decoded independently of other access units by using only globally available data (e.g., decoder configuration) or by using information contained in other access units. An access unit consists of a header and the result of multiplexing data blocks from different layers. Multiple packets of the same type are encapsulated into a block, and multiple blocks are multiplexed into one access unit. These concepts are illustrated in Figures 5 and 6 of WO2018068827. For clarity, in this disclosure, access units containing compressed genome sequencing data are referred to as first-class access units, and access units containing compressed annotation data are referred to as second-class access units.

[0066] A genomic data layer is defined as a set of genomic data blocks that encode the same type of data (e.g., position blocks of reads that perfectly match the reference genome are encoded in the same layer).

[0067] A genome data stream is a packetized version of a genome data layer, where the encoded genome data is carried as the payload of a genome data packet that contains additional service data in its header. See Figure 7 of WO2018068827 for an example of packetizing three genome data layers into three genome data streams.

[0068] A genome data multiplex is defined as a sequence of genome access units used to convey genome data associated with one or more genome sequencing, analysis, or processing processes. Figure 7 of WO2018068827 provides a schematic diagram of the relationship between a genome multiplex carrying three genome data streams broken down into access units. The access units encapsulate data blocks belonging to the three streams and are divided into genome packets that are transmitted over a transmission network. [Brief explanation of the drawings]

[0069] [Figure 1] FIG. 1 shows the relationship between the present invention and the coding device described in ISO / IEC 23092. [Figure 2] FIG. 2 shows a coding apparatus for genome annotation that functions according to the principles of the present invention and extends the coding apparatus described in ISO / IEC 23092. [Figure 3] FIG. 3 illustrates a decoding apparatus for genome annotation that operates in accordance with the principles of the present invention and extends the decoding apparatus described in ISO / IEC 23092. [Figure 4] FIG. 4 illustrates a decoding apparatus for genome annotation that functions according to the principles of the present invention and extends the decoding apparatus described in ISO / IEC 23092 to allow partial decoding driven by a text query. [Figure 5] FIG. 5 shows an example of a possible uncompressed index layout for a string index, useful for explaining the string indexing algorithms presented in this disclosure. [Figure 6] Figure 6 shows how two families of string indexing algorithms can be combined to maximize compression and speed over using one family alone. [Figure 7] FIG. 7 shows the relationship between the present invention and the decoding device described in ISO / IEC23092. [Figure 8] FIG. 8 shows how the conceptual organization of data described in this invention prepares it for performing text queries. [Figure 9] FIG. 9 illustrates how the conceptual organization of data described in this invention allows searches across genomic intervals to be performed. DETAILED DESCRIPTION OF THE INVENTION

[0070] Key aspects of the disclosed solution are:

[0071] 1 To enable selective access to the encoded data according to criteria related to the alignment results, sequence reads are classified into different classes according to the results of their alignment with a reference sequence. This implies the specification of a file format that "contains" structured data elements in a compressed format. Such an approach can be seen as the opposite of prior art approaches, such as SAM and BAM, in which the data is structured in an uncompressed format and then the entire file is compressed. The first clear advantage of this approach is that it can efficiently and naturally provide various forms of selective access to data elements in the compressed domain, which is impossible or very cumbersome in prior art approaches.

[0072] 2. Decomposition of classified reads into layers of homogeneous metadata to reduce information entropy as much as possible. Decomposition of genomic information into specific "layers" of homogeneous data and metadata offers the great advantage of allowing the definition of different models of information sources characterized by low entropy. Such models can differ not only from layer to layer but also within each layer. This structuring allows the use of specific compressions optimal for each class or part of data or metadata, resulting in significant improvements in coding efficiency compared to prior art approaches.

[0073] 3. Structuring layers into access units, i.e., genome information that can be decoded individually using only globally available parameters (e.g., decoder configuration) or by using information contained in other access units. When the compressed data in a layer is divided into data blocks contained in access units, various models of information sources can be defined, characterized by low entropy.

[0074] 4 Information is structured so that relevant subsets of data used in genomic analysis applications can be efficiently and selectively accessed through appropriate interfaces. These features enable fast access to data, resulting in more efficient processing. Master and local index tables enable selective access to information carried by layers of coded (i.e., compressed) data without the need to decode the entire volume of compressed data. Furthermore, association mechanisms between various data layers are specified to enable selective access of possible combinations of subsets of semantically associated data and / or metadata layers without the need to decode all layers.

[0075] 5 Joint storage of master index table and access units.,The encoding scheme for genome reads is represented,by the encoder in Fig. 1.

[0076] Sequencing read classification The sequence reads generated by the sequencing device are classified by the disclosed invention into five different "classes" according to the results of alignment with one or more reference sequences, which are defined based on matching / mapping to a reference genome due to the presence of substitutions, insertions, deletions and clipped bases with the one or more reference sequences.

[0077] When aligning a DNA sequence of nucleotides against a reference sequence, there are five possible outcomes. 1. A region in the reference sequence is found to match the read sequence without any errors (perfect mapping). Such a nucleotide sequence is called a "perfect match read" or designated "Class P." 2. A region in the reference sequence is found to match sequence reads with many mismatches, consisting of many positions where the sequencing instrument was unable to call a base (or nucleotide). Such mismatches are indicated with an "N." Such sequences are called "N-mismatched reads" or "Class N." 3. A region in the reference sequence is found to match a sequence read with many mismatches, consisting of many positions where the sequencing instrument was unable to call a base (or nucleotide), or where a base was called that differs from that reported in the reference genome. These types of mismatches are called Single Nucleotide Variations (SNVs) or Single Nucleotide Polymorphisms (SNPs). This sequence is called an "M-mismatched read" or "Class M." 4. The fourth class is composed of sequence reads that exhibit mismatch types (also known as indels), including the same mismatches as class M plus the presence of insertions or deletions. An insertion is represented by a sequence of one or more nucleotides that are not present in the reference but are present in the read sequence. In the literature, if the inserted sequence is at the end of the sequence, it is called a "soft clip" (i.e., the nucleotides do not match the reference but are retained in the aligned read, as opposed to "hard clip" nucleotides, which are discarded). A deletion is a "hole" (missing nucleotide) in the aligned read relative to the reference. Such sequences are called "I-mismatched reads" or "class I." 5. The fifth class includes all reads that can find a valid mapping on the reference genome according to the specified alignment constraints. Such sequences are said to be unmapped and belong to "class U". Unmapped reads can be assembled into a single sequence using a de-novo assembly algorithm. Once a new sequence is generated, unmapped reads can be further mapped to it and classified into one of four classes P, N, M, and I.

[0078] Once the classification of the reads is completed with the definition of classes, further processing consists of defining a set of distinct syntactic elements that represent the remaining information that allows the reconstruction of the DNA read sequence when represented as mapped to a specific reference sequence. A DNA segment that references a particular reference sequence can be fully represented in the following way:

[0079] Syntactic elements used to code genomic reads Start position of the reference genome (pos) A flag indicating whether the read should be considered a reverse complement to the reference (rcomp) Distance to mate pair in case of paired reads (pair) The read length value if the sequencing technology produces variable length reads. If the read length is constant, the read length associated with each read can obviously be omitted and stored in the main file header. Additional flags describing specific characteristics of the read (duplicate read, first or second read in a pair, etc...) For each mismatch: Position of mismatch (nmis in class N, snpp in class M, and indp in class I) Type of mismatch (absent in class N, snpt in class M, indt in class I) Optional soft-clip nucleotide string, if present (indc for class I)

[0080] This classification creates a group of descriptors (syntactic elements) that can be used to unambiguously represent a genome sequence read.

[0081] For each layer of the genome data structure disclosed in the present invention, different coding algorithms can be used according to the specific features of the data or metadata carried by the layer and its statistical properties. A "coding algorithm" should be intended as an association of a specific "source model" of a descriptor with a specific "entropy coder." A specific "source model" can be specified and selected to obtain the most efficient coding of the data in terms of minimizing source entropy. The choice of entropy coder is driven by coding efficiency considerations and / or probability distribution functions, and related implementation issues. Each choice of a specific coding algorithm is called a "coding mode," which applies to the entire "layer" or all "data blocks" contained in the access unit. Each "source model" associated with a coding mode has the following characteristics:

[0082] Definition of the syntax elements published by each source (e.g., read position, read pairing information, mismatches with respect to the reference sequence, etc.). · Definition of the relevant probabilistic model. · Definition of the relevant entropy coder.

[0083] For each data layer, the source model employed by one access unit is independent of the source model used by other access units of the same data layer, allowing each access unit to use the most efficient source model for each data layer in terms of entropy minimization.

[0084] Genome annotation Genome annotations, browser tracks, variant information, gene expression matrices, and other annotations referred to in the present invention are associated with, for example, but not limited to, nucleotide sequences, deoxyribonucleic acid (DNA) sequences, ribonucleic acid (RNA), and amino acid sequences. While the description herein is described in considerable detail with respect to annotations to reference genomes in the form of nucleotide sequences, it will be understood that the methods and systems for compression can be similarly implemented for annotation of other genome or proteome sequences, with some variations as will be understood by those skilled in the art. Genomic functional annotations are defined as notes added as descriptions or annotations to identified locations of genes and coding or non-coding regions within a genome to explain the function of the genes and their transcripts.

[0085] Genomic variants (or variations) represent differences between a genomic sample and a reference genome. Variants are typically classified as small (e.g., substitutions, insertions, and deletions) and large (a.k.a. structural) variants (e.g., copy number variations and chromosomal rearrangements).

[0086] A genome browser track is a plot associated with aligned genome sequence reads visualized in a genome browser. Each point in the plot corresponds to a location in the reference genome and represents information related to that location. Typical information represented in a browser track includes the presence and structure of transcripts, individual or population sequence variants, sequence read coverage, and the strength of protein binding to each location in the genome.

[0087] A gene expression matrix is ​​a two-dimensional array in which rows represent genomic features (usually genes or transcripts), columns represent different samples such as tissues or experimental conditions, and numbers represent counts of the number of times each gene is expressed in a particular sample (the counters are also known as the "expression level" of a particular gene).

[0088] The contact matrix is ​​generated by a Hi-C experiment, where each i,j entry measures the strength of the physical interaction between two genomic regions i and j at the DNA level. At the smallest granularity, i and j denote two locations on the genome represented as one sequence across all chromosomes concatenated.

[0089] The limitations of current state-of-the-art technology To date, the above classes of annotation data have been represented using different, incompatible text formats, typically compressed using general-purpose text compression programs such as gzip or bzip2. In most cases, analysis programs process this information by first decompressing the entire file and then parsing the decoded text to look for and extract the required information, if present. It is quite common for each format used for each category of data to be modified independently, sometimes significantly, by different users or groups of users to produce several "variations" or "dialects" of the same format. This fact creates significant interoperability issues and requires that each file format variation be "sanitized" before the data can be exchanged.

[0090] Another limitation of the current format is that there is no support for establishing links between different types of annotation data when represented in a condensed format. For example, to associate a set of variants with a particular gene, it is necessary to:

[0091] 1) Unpack and parse the variant files (i.e. unpack BCF to VCF) 2) Extract and analyze gene annotation files (i.e., GTF / GFF) 3) Establishing links using the respective genomic locations of variants and genes as a result of both analysis operations on the whole file, which requires another ad-hoc format that does not exist at the time of this writing.

[0092] State-of-the-art formats have the disadvantage of being stored in separate files, which are inefficient as far as data compression is concerned and do not support an efficient process for running queries against compressed files. To obtain all variants associated with a particular gene XYZ, and possibly simultaneously obtain the expression of that gene in a sample set, one must unpack the entire associated file and parse all of its contents. Today, the described process of relating variants to genes can only be achieved by combining several inefficient operations of data unpacking, parsing, and processing, and by describing the relationships between various features using new ad-hoc formats that are not currently available or standardized.

[0093] Use case: Variant calling in a clinical setting By way of example and not limitation, the method disclosed in this document addresses the shortcomings of current solutions when attempting to determine clinically relevant variants using variant calling pipelines and visualizes results so that clinicians can easily inspect and verify them. The goal is to use genome resequencing to identify variants that may be associated with disease manifestation or a specific phenotype of interest. Variants are determined by first aligning genomic sequence reads to a reference genome and then using the alignment information at every position accumulated across all reads (the "pileup") to call genomic variants, such as single nucleotide polymorphisms (SNPs), through an appropriate variant calling program. Variant calling is a complex operation requiring a complex pipeline of tools to perform advanced processing. False-positive or false-negative results can occur due to many technical issues, such as variable coverage or variants located in repetitive genomic regions. Due to these issues, in clinical settings, variants of potential clinical significance are typically manually verified by human operators before being included in medical reports. However, data processing and validation typically require access to and correlation of numerous information elements (genome sequence, genome annotation, read alignment, sequencing coverage, and sequencing pileups in regions flanking variants), each stored in separate files and represented using different file formats. In particular, current technology makes it impossible to explicitly state relationships such as "this set of sequencing reads aligned to this range of positions (i.e., interval) in the genome supports this variant contained in this genomic feature," because the various entities (aligned reads, variants, and genomic features) are represented in separate, distinct files. Today, this result can only be achieved by:

[0094] 1) Unzip the various files to get the original textual representation of the information for the entire sample. 2) Analysis of text files to search for features of interest (e.g., genomic intervals, gene names, annotation names, etc.) 3) Perhaps mapping (slightly) different names used in different files to identify the same function (different naming conventions exist to identify the same genomic function) 4) Aggregate the retrieved information into a single container and expose it to the end user or processing application in an application-specific format. These various steps can take a significant amount of time depending on the size of the parsed text file, which can range from several gigabytes to hundreds of gigabytes.

[0095] The present invention aims to address these limitations by providing: 1. A unified, condensed representation of annotations that can represent the following information content: browser tracks, genomic variants, gene expression data, contact matrices, and other metadata associated with genome sequencing data. 2. High compression performance of the unified representation, resulting in higher compression ratios when compared to state-of-the-art solutions. 3. Built-in indexing functionality that provides explicit browsing of annotations and metadata within the compressed domain. This indexing functionality supports the execution of advanced queries that generate a hierarchy of related data structures including biologically linked annotations, browser tracks, genomic variants, gene expression information, contact matrices, and other annotations associated with intervals of aligned genome sequencing data. 4. A mechanism for explicitly linking indexed and compressed raw sequencing data and associated metadata with indexed and compressed annotations. Such a mechanism would enable selective access across domains of compressed annotations and associated related sequence reads by querying either the compressed raw data or the compressed annotation data.

[0096] In this example of variant calling in a clinical setting, data processing and visualization are accomplished by encoding two different compressed data structures (which may or may not be contained in the same file) linked by a bidirectional indexing mechanism. The data structures include:

[0097] 1. Genome sequencing reads and associated alignment information 2. Annotation information (annotations, browser tracks, genomic variants, gene expression information, contact matrices and other annotation data) as described in this disclosure.

[0098] In particular, the coded information, as described in this disclosure, is contained in a hierarchical structure that links: 1. Variants for included genes or genome functions, including details on the function and ontology of each gene (if any). 2. A variant for a supporting lead, i.e., a variant for a lead that supports the variant being called. 3. Each variant into a pileup profile obtained from reads supporting the variant. 4. Any other type of annotation information as described above.

[0099] The current state of the art allows for separate representation of the various sources required for genomic data annotation and variant calling (aligned reads using SAM / BAM / CRAM files, genome annotation using GTF / GFF3 files, variants using VCF / BCF files, and various index file formats required for implementing range searches). These do not support explicit representation of bidirectional relationships between different entities. Furthermore, software analysis workflows (or "pipelines") that perform variant calling must work with different file formats depending on the analysis stage, rather than the single data structure provided by this disclosure. While it is possible to represent the various sources in a single genome browser, this requires working with various file formats, and there is no way to specify to the genome browser that features belonging to different files are correlated.

[0100] Technical advantages of variant calling analysis. In one embodiment, the present invention offers important technical advantages for variant calling analysis use cases, as described in the following text.

[0101] With regard to efficient data mining for variant calling analysis, the advantages of our method over state-of-the-art solutions are as follows: 1. Applications that explicitly express the relationship between sequence reads and genomic features, such as genome browsers, should support and manage a single data container and associated bitstream format, rather than a multitude of potentially non-interoperable formats. 2. By using genome browsers or other similar means, clinicians and scientists can explore the relationship between variants, their supporting reads, and the names and functions of the genes involved. In particular, integrating different types of information allows clinicians and scientists to verify the accuracy of variant calls (e.g., excluding miscalls due to the presence of repeated reads and / or repeated reference regions, or due to the lack of realignment when multiple indels are present at different positions, or confirming the likely significance of the variant by the function of the gene containing the variant, or its presence in a database of known variants). 3. The possibility of performing text searches of the meta-information contained in the files allows clinicians or scientists to correlate the presence / absence of multiple variants based on gene function (e.g., by retrieving all variants contained in genes with similar functions or genes with multiple functional copies, or by retrieving all variants with similar clinical effects contained in known databases). 4. Analysis pipelines can selectively access and operate on a single coded data structure at all stages (from alignment to variant calling), resulting in much simpler and more economical software development / data access patterns and reduced operational costs. 5. Because relationships are explicitly established when encoding the data and all relationships are coded into a referable index, there is no need to decompress and parse the entire file and potentially disconnected files, irrelevant features can be discarded (e.g., variants present in known databases but not in the resequenced individual, or variants unrelated to the pathology being considered) to obtain higher compression ratios. 6. All processing steps 1 through 5 requiring data access can be performed utilizing an indexing mechanism embedded in the compressed data, supporting single-query retrieval of both sequence reads and all associated annotations from a single compressed file structure. The sequencing reads and associated annotations can also be separated and encapsulated in separate files to enable transfer of only the required portions of the data.

[0102] Limitations of state-of-the-art variant calling solutions State-of-the-art technologies support the representation of the various pieces of information required for the described use case using a variety of data structures and formats: sequence reads in SAM / BAM / CRAM file formats, genome annotations in GTF / GFF3 file formats, variants in VCF / BCF file formats, and various types of independent index file formats used to implement range searches only. These state-of-the-art technologies do not support explicit representation and linking of relationships between different pieces of information. Pipelines that perform variant calling must operate with different file formats depending on the analysis stage, rather than a single, selectively accessible, compressed data structure as proposed in current approaches. While genome browsers can be fed a variety of genomic information using current state-of-the-art technologies, this requires a complex preprocessing stage consisting of manipulating and parsing various file formats in uncompressed formats. Furthermore, there is no way to specify to genome browsers the appropriate display, annotation, biological features, and correlations between sequence data.

[0103] Use Case: Establishing and querying population-level libraries of genomic variant data By way of example, and not limitation, the method disclosed in this document addresses the shortcomings of existing solutions when attempting to compile a large database of genomic variants. This scenario is similar to the one considered in the previous case: a setting in which a researcher or clinician seeks to validate and collect genomic variants based on sequencing technologies. However, we now assume that the researcher or clinician is interested in cataloging a large number of variants—ideally, all variants in each genome—for a potentially very large number of individuals (one can think of initiatives that seek to cover an increasing portion of the population, with the ultimate goal being complete coverage). In this example, variant calling is performed first, typically following the analysis steps described in the previous use case. This process is repeated for all samples. Researchers then typically query information resulting from the data analysis, such as, "How many individuals have this particular variant?", "Is this variant consistently supported across all individuals considered?", or "How many individuals in a sample have any of the variants included in a given dataset of clinically relevant variants? And what is the list of such variants for each individual?" Currently, there are ways to save the list of variables, typically as a VCF / BCF file. However, the size of such population-level files is prohibitive, making querying them technically challenging and allowing only very limited query functionality (i.e., retrieving variants at specified genomic intervals).

[0104] Technical Advantages The advantages of the current method over state-of-the-art solutions are: 1. The possibility of storing large collections of variants in a more compact way, because the method disclosed in this document explicitly separates and describes the sources of information about the variants, allowing better compression techniques to be specified for each source. 2. The possibility of performing more complex queries in a compressed domain, in part because the data is separated by individuals and split into multiple streams with specified semantics, which allows for selective access and filtering in addition to range access based on genome coordinates. 3. The possibility of correlating information about variant calls with other types of information, such as functional annotations present at the variant position, sequence reads supporting each variant, and the intensity at that position of signals obtained from other sequencing technologies, such as ChIP-seq experiments.

[0105] The limits of cutting-edge solutions While it is possible to store large databases using currently available formats such as VCF / BCF, the complexity of the format complicates the process, and the resulting files are relatively bulky due to the common compression methods used and the mixing of different sources in the same record, which reduces the efficiency of compression. Furthermore, formats such as VCF / BCF were not designed with complex queries in mind. They can only be queried genomically to retrieve all variants present within a genomic range. Further filtering, such as selecting variants depending on whether they are present in a particular individual, must be performed separately. Finally, as described in the previous use case, there is no ability to intersect information about genomic variants with other sources, such as lists of supporting sequence reads or lists of functional genomic features.

[0106] Use Case: Correlating information from complex omics experiments By way of example and not limitation, the methods disclosed herein address the shortcomings and inefficiencies of current solutions when attempting to determine the biological mechanisms underlying a particular phenotype. This is achieved by encoding several pieces of information into the same compressed data structure (e.g., multiple "omics" sequencing-based experiments). Identifying complex molecular mechanisms requires the combination of multiple experimental techniques, each examining a different cellular compartment (e.g., ChIP-seq experiments investigating chromatin structure, bisulfite sequencing experiments determining genomic methylation, RNA-seq experiments determining how transcription is regulated, etc.).

[0107] The molecular mechanisms underlying genotypes are determined by analyzing the interactions and correlations between patterns that occur simultaneously in different cellular compartments when the same biological state is sequenced. Chromatin markers are obtained by accumulating alignments to a reference genome and are determined as peaks in ChIP-seq tracks. Methylation patterns are obtained by a specialized alignment pipeline capable of processing BS-seq data, since bisulfite treatment generates reads with altered bases not present in the original genome. RNA sequencing data is processed by an ad hoc alignment pipeline capable of performing splicing alignment. This depends on the specific "omics" experiment being considered, such as how the cellular machinery concatenates one or more blocks of genomic sequence ("exons") and discards sequences occurring between the blocks ("introns") to derive RNA sequences, resulting in sequences not present in the original genome.

[0108] The data generated by each "omics" experiment typically require complex analytical pipelines, each tailored to the type of sequences generated by the specific biological protocol employed (e.g., ChIP-seq, BS-seq, RNA-seq). Each pipeline typically requires different types of data (genomic sequence, genome annotation, sequencing reads, read alignments, sequencing coverage, sequencing pileup), each of which is typically stored in different files and represented, accounted for, and related using different file formats. In particular, current technology makes it impossible to explicitly state relationships such as "in a particular biological condition, this set of sequencing reads aligned to this range of positions in the genome supports this ChIP-seq peak, which correlates with specific patterns of RNA expression and genomic / histone methylation."

[0109] Technical advantages in data processing and visualization In one embodiment, the processing and visualization of genomic data is improved by the present invention by presenting it in the same compressed data structure. 1. Genome sequencing reads and associated alignment information 2. Annotation information as described in this disclosure (gene models, pileup profiles, methylation patterns called ChIP-seq peaks, expression levels derived from RNA sequencing).

[0110] In particular, the joint compressed data structure, as described in this disclosure, includes a hierarchical structure that links: 1. Methylation patterns, ChIP-seq peaks, and RNA expression in different biological conditions, along with the genes or genomic features included (if any) and the function and ontology details of each gene. 2. Methylation patterns, ChIP-seq peaks, and RNA expression in different biological conditions and their supporting reads, i.e., reads supporting each of the described functions. 3. Each feature to pile-up profile taken from leads supporting the feature.

[0111] The advantages of the present method over existing solutions regarding efficient data mining to correlate information obtained from several "omics" experiments are listed below. 1. To explicitly represent the relationships between sequence reads and "omics" features, and between different "omics" features, as current methods provide, applications such as genome browsers need to support and manage a single data container and associated bitstream format, rather than a multitude of non-interoperable formats. 2. Through browsers and other means, researchers can explore the relationships between various "omics" features, their supporting reads, and the names and functions of the genes they contain. In particular, integrating different types of information allows researchers to infer correlations / causal relationships between various "omics" features highlighted by experiments and flag interesting genomic regions for subsequent experimental validation. 3. The possibility of performing text searches of the annotations contained in the files allows researchers to correlate the presence or absence of multiple "omics" features based on gene function (e.g., by retrieving all features contained in genes with similar functions or genes with multiple functional copies). 4. The analysis pipeline can operate on a single compressed data structure for all stages (from alignment to variant calling) and all types of "omics" data, resulting in much simpler software development / data access patterns. 5. Because relationships are explicitly established when encoding files and all relationships are encoded in the same file rather than using disconnected files, irrelevant features (e.g., "omics" features occurring outside the region of interest) can be discarded, resulting in higher compression.

[0112] Limitations of existing solutions for linking different genome functions Existing technologies allow users to represent the various information sources required for this use case individually (aligned reads using SAM / BAM / CRAM files, genome annotation using GTF / GFF3 files, ChIP-seq peaks, RNA expression levels, and other "omics" features) in addition to the various index file formats required to implement range searches, along with other file types. However, they do not support explicit representation of relationships between different entities. Pipelines performing analysis of various "omics" data would need to work with different file formats depending on the analysis stage, rather than the single compressed data structure proposed in current approaches. While it is possible to present the various information sources as a single genome browser, this requires working with different file formats, and there is no way to tell the genome browser that features belonging to different files are correlated.

[0113] Concepts and Terminology Access Unit Throughout this disclosure, and with reference to WO2018 / 068827A1, WO / 2018 / 068828A1, and WO / 2018 / 068830A1, an access unit (AU) is defined as a logical data structure containing a coded representation of genomic information for ease of access and manipulation of the bitstream. It is the smallest data organization that can be decoded by a decoding device implementing the inventions described in this disclosure. An access unit is characterized by header information and a payload of compressed data structured as a series of blocks, each compressed using a different compression scheme. The inventions described in this document introduce a new access unit type that contains genome annotation data, such as genomic features, functional annotations, browser tracks, genomic variants, gene expression information, contact matrices, and genotype data.

[0114] In the context of this disclosure, the following definitions apply: Genome annotation record: A data structure consisting of a set of genome annotation descriptors that describe features of a genome, such as genome functional annotations, browser tracks, genome variants, gene expression information, contact matrices, genotype data, and other annotations associated with a genomic interval. Each genome annotation record is identified by a unique identifier, as shown in Table 1.

[0115] Genomic Features: Genomic features are intended as pieces of biologically meaningful information associated with genome sequencing data. By way of example and not limitation, genomic features include genome annotations, browser tracks, genome variants, gene expression information, and contact matrices.

[0116] Access unit start position: The smallest mapping position on a reference sequence (e.g., chromosome) where an access unit encodes genomic data or metadata.

[0117] Access unit end position: The maximum mapping position on a reference sequence (e.g., a chromosome). An access unit encodes genomic data or metadata.

[0118] Access unit range: The genome range contained between the access unit start position and the access unit end position.

[0119] Access unit size: The number of genome annotation records contained in an access unit.

[0120] Access unit coverage region: The genomic range contained between the start position of an access unit and the end position of the access unit.

[0121] In the context of this disclosure, one or more access units are organized into a structure called a genome dataset. A genome dataset is a compressed unit that includes a header and access units. The set of access units that make up a genome dataset constitutes a genome dataset payload. A collection of one or more genome datasets is called a dataset group.

[0122] Read Classes: ISO / IEC 23092 and WO2018 / 068827A1, WO / 2018 / 068828A1, WO / 2018 / 068830A1, and WO2018152143A1 specify how genome sequence reads are classified and coded according to the results of alignment of said reads on a reference genome. Depending on the type and number of mapping errors, each read or read pair is assigned to a different class.

[0123] AU class: Each AU contains reads that belong to a single class.

[0124] Annotation Data Type: In the context of this disclosure, an annotation data type characterizes a set of genome annotation information that falls into one of these categories: genome features, functional annotations, browser tracks, genome variants, gene expression information, contact matrix, genotype data, genome sample information.

[0125] Genome annotation descriptors In the context of this disclosure, a genome annotation descriptor is a syntactic element that represents some of the information (and also elements of the file format and / or bitstream syntactic structure) required to reconstruct (i.e., decode) the coded reference sequence, sequence reads, associated mapping information, annotations, browser tracks, genomic variants, gene expression information, contact matrices, and other annotations associated with genome sequencing data. Genome annotation descriptors common to all annotation data types disclosed in this invention are listed in Table 1.

[0126] Other descriptors specific to each annotation data type are disclosed in the syntax and semantics tables specific to each annotation data type. Textual descriptors are descriptors represented as strings, and numeric descriptors are descriptors represented as numbers.

[0127] There are three types of genome annotation descriptors: Numerical descriptors expressed as numbers A text descriptor represented as a string Attributes are data structures defined in this disclosure (section titled "Attributes")

[0128] [Table 2] Table 1 - Descriptors common to all annotation data types

[0129] According to the method disclosed in the present invention, genome annotations, browser tracks, genome variants, gene expression information, contact matrices, and other annotation data types associated with genome sequencing data are coded using a subset of the descriptors listed in Table 1. Each descriptor is then entropy coded using multiple entropy coders according to its unique statistical properties. This means that different types of descriptors are grouped together and coded with different entropy coders, achieving higher compression ratios. Blocks of compressed descriptors with similar statistical properties are structured into access units that represent the smallest coded representation of one or more genomic features that can be manipulated by a device implementing the invention described in this disclosure.

[0130] Genome annotation descriptors are organized into blocks and streams as defined below.

[0131] A block is defined as a data unit consisting of a header and a payload made up of some compressed descriptors of the same type.

[0132] A descriptor stream is defined as a sequence of coded descriptor blocks used to decode the descriptors of a particular data class.

[0133] This disclosure specifies a genomic information representation format in which relevant information is efficiently compressed to make it easily accessible, transferable, storable, and viewable, and in which redundant information is de-weighted.

[0134] The main innovative aspects of the disclosed invention are as follows: Annotations, browser tracking, genomic variants, gene expression information, contact matrices, and other metadata associated with genome sequencing data are compressed into a unified hierarchical data structure that allows for fast transfer, economical storage, and selective access to the encoded data according to criteria such as genomic interval / location, gene name, variant location and genotype, variant identifier, comments within the annotation, annotation type, and by genomic interval pair (in the case of matrix data connecting genomic locations to other locations). 2 Annotations, browser tracks, genome variants, gene expression information, contact matrices, and other annotation data associated with genome sequencing data are represented by genome annotation descriptors grouped into blocks with uniform statistical properties, allowing the identification of distinct sources of information characterized by low information entropy. 3. The possibility of modeling each isolated information source using a separate source model that matches the statistical characteristics of each annotation descriptor, and the possibility of modifying the source model within each annotation descriptor of each annotation data type and within each descriptor block of each individually accessible data unit (access unit). Depending on the statistical characteristics of each source model of the annotation descriptor, appropriate transformations, binarizations, context-adaptive probability models, and associated entropy coders are employed. 4 Definition of correspondences and dependencies between descriptor blocks that allow selective access to sequence data and associated metadata without the need to decode all descriptor blocks when only part of the information is needed. 5. Transmission of configuration parameters that control both the encoding and decoding processes through data structures embedded in the compressed genome data in the form of header information. Such configuration parameters can be updated during the encoding process to improve compression performance. Such updates are conveyed in the compressed content in the form of updated configuration data structures. Each of the above aspects is described in further detail below.

[0135] Genome annotation descriptors for each specific annotation data type Genomic variants Data on genomic variants are coded using the general descriptors introduced above and the specific descriptors listed below.

[0136] [Table 3]

[0137] Functional annotation Functional annotation data describes a gene and its contents: the spliced ​​transcript with biological function in terms of its constituent exons, its breakdown into UTRs, where applicable, its start and stop codons, the coding sequence, and other information about the transcript. It is coded using the general descriptors introduced above and the specific descriptors listed below.

[0138] [Table 4] The sizeof() function returns the number of bits required to represent each attribute value according to the type_ID defined in the attribute type.

[0139] track The data in a track represent a numerical value associated with each position in the genome. A typical example is the coverage of sequence reads at each position generated by an RNA or ChIP-sequencing experiment. The data can be provided at various pre-calculated zoom levels, which is appropriate when the information is displayed in a genome browser. The data is coded using the general descriptors introduced above and the specific descriptors listed below.

[0140] [Table 5]

[0141] Genotype information Genotype information data represents the set of genomic variants present at each location in the genome of an individual or population of individuals, and is coded using the general descriptors introduced above and the specific descriptors listed below.

[0142] [Table 6] TIFF0007775215000007.tif73170

[0143] Sample information Sample information describes meta-information about the specific biological sample on which the sequencing experiment was performed, such as collection date and location, sequencing date, etc. Sample information data is coded using specific descriptors shown below.

[0144] [Table 7]

[0145] Expression information Expression information associates some genomic region (usually corresponding to a gene, transcript, or another feature of the genome) with one or more numerical values. Each value corresponds to a biological condition tested in a separate experiment. Expression data are coded using specific descriptors, as described below.

[0146] [Table 8]

[0147] Contact Matrix Information Contact information data is coded using specific descriptors as follows:

[0148] [Table 9]

[0149] Bitstream Structure The present invention introduces a condensed representation of annotation data associated with genome sequencing data in the form of a bitstream syntax described below. The syntax is described in terms of a concatenation of data structures made up of elements characterized by data types.

[0150] Syntax Notation In the following description, the following syntax notation is used:

[0151] [Table 10]

[0152] Extensions to ISO / IEC 23092-1 The present disclosure extends the data structures specified in ISO / IEC 23092-1 to support the transport of genome annotations encoded in the bitstream syntax specified in ISO / IEC 23092-1.

[0153] Dataset Group The syntax of the dataset group is the same as that specified in ISO / IEC 23092-1.

[0154] [Table 11]

[0155] Dataset In ISO / IEC 23092-1, a dataset is a data structure that contains a header, master configuration parameters in a parameter set, an index structure, and a collection of access units that encode genomic data. The dataset type is extended to hold different types of genomic annotation data, specified by different "dataset_type" values.

[0156] [Table 12] TIFF0007775215000014.tif54170

[0157] [Table 13]

[0158] [Table 14]

[0159] Dataset Header This is a box that describes the contents of the dataset.

[0160] [Table 15] TIFF0007775215000018.tif231160TIFF0007775215000019.tif231160TIFF0007775215000020.tif160160

[0161] References This data structure extends the reference data structure specified in ISO / IEC 23092 to support the bitstream syntax specified in this disclosure.

[0162] [Table 16] TIFF0007775215000022.tif232170TIFF0007775215000023.tif55170

[0163] Annotation Indexing This disclosure describes methods for encoding (i.e., compressing) annotation data portions consisting of genome sequencing reads, other non-textual genome annotations, and textual information elements associated with sequences derived from genomes, so that the text elements are searchable in a compressed domain.

[0164] Information on functional features of the genome (e.g., gene name, gene description, gene annotation, gene ontology, variant name, variant description, clinical significance of the variant) Nucleic acid sequences (e.g., partial sequences of a reference genome, sequences of RNA molecules transcribed from a reference genome, or sequencing reads from a genome) Protein sequences (e.g., sequences corresponding to translation of messenger RNA molecules) Sample metadata and methodological information (e.g., name, collection date / time / location, experimental technique used to perform sequencing, analytical techniques used to perform functional annotation and variant calling).

[0165] The information is compressed using an appropriate data structure, such as, by way of example and not limitation, a compressed string pattern matching data structure. Representative examples of compressed string pattern matching data structures are, by way of example and not limitation, compressed suffix arrays, FM indexes, and several categories of hash tables. Such (compressed) data structures are used to perform string pattern matching and to carry in compressed form the text portion of the annotation data that is added to the compressed bitstream as a file header or access unit payload. For clarity, in this disclosure, all algorithms that belong to one of these data structure categories will be referred to as "string indexing algorithms."

[0166] By way of example and not limitation, the present disclosure describes a method for encoding different annotation data types and text portions of genome reads by using a combination of compressed string indexing algorithms. There are several families of string indexing algorithms, and each family can be parameterized by a number of parameters that specify the balance between compression performance and query speed. Compression uses a predefined set of compressed string indexing algorithms. Each algorithm is specified by selecting a compressed string indexing algorithm family and a selection of parameters for that family. The set of algorithms is sorted by the compression level achieved, and one specific algorithm can be selected at encoding time depending on the desired tradeoff between compression ratio and query speed. This selection is specified in a parameter set for the compressed bitstream.

[0167] By way of example and not limitation, the selected compression string indexing algorithm may be applied to the following concatenations, either individually or together: Gene name Gene description Sequences of genomic transcripts and their protein products (if present) Variant Name Variant Description Sample Name Genome sequence reads, represented as a sequence of one symbol for each nucleotide and other text information associated with the genomic interval Additional information that encodes the relationship between textual information and genomic intervals.

[0168] Applying a compressed string indexing algorithm to the information produces a compressed and indexed representation that can be queried for the presence of any substring. Specifically, a combination of exact substring searches can be used to perform imprecise substring searches, such as a search that retrieves all occurrences of a substring up to a specified number of deviations (mismatches / errors) from a specified pattern. This process allows for a single query to query text information considered or generated by genome annotation during analysis and reanalysis of sequence data. This is possible when: 1. Genomic information associated with a genomic interval is represented as a data structure called a genome annotation record, which contains information about the sequence of nucleotides contained in that interval. 2. Genome annotation records associated with genomic intervals related to consecutive positions on the reference genome are compressed into the same access unit. 3. All text portions of the annotation information are compressed using a compressed string indexing algorithm selected from the available set.

[0169] The following text and data structures describe an embodiment of this method for indexing and searching genome annotation data that has been compressed and embedded in access units of an MPEG-G (ISO / IEC 23092) compliant bitstream.

[0170] The table below shows the text information indexed and compressed using the string indexing algorithm for each genome annotation type, according to the method described in this document. For each access unit, the text descriptors of each type are concatenated using string delimiters, the index information is recorded, and then compressed using the string indexing algorithm, as shown in Figure 5.

[0171] [Table 17]

[0172] Indexing criteria access unit type per genome annotation This table describes the indexing criteria and indexing tools that apply to access units of each genome annotation data type.

[0173] [Table 18] TIFF0007775215000026.tif203170

[0174] The Master Annotation Index (MAI) is an indexing tool that provides indexing capabilities for MIT sequence reads as defined in ISO / IEC 23092-1 and WO2018 / 068827A1, WO / 2018 / 068828A1, and WO / 2018 / 068830A1 for annotation data.

[0175] [Table 19] Table 2 - Master Annotation Index

[0176] Master Annotation Index Header [Table 20] Table 3 - Master Annotation Index Header

[0177] Semantics num_mai_AU_types is the number of AU types indexed by the MAI. A value of 0 indicates that no indexing is provided by the MAI. mai_AU_type[i] is the i-th AU type indexed by MAI. The array mai_AU_type[] contains unique values, i.e., each AU type value can appear only once in the array mai_dataset_ID[]. num_mai_indexes[i] is the number of MAI indexes for AU type mai_AU_type[i]

[0178] Indexed Strings When encoding an access unit for each genome annotation data type, the text descriptors belonging to the data encoded in the access unit are concatenated and compressed using the compressed string indexing algorithm defined in this disclosure.

[0179] The following table shows the strings that are coded in the MAI for each data type. The list of specified strings determines the value numStrings required in the following MAI description portion. numStrings is the number of text fields per genome annotation record indexed using the methods described in this invention.

[0180] [Table 21]

[0181] String Indexing The string index block is part of a master annotation index that encodes one or more strings in each record for a variable number of access units, each containing a variable number of records. The master string index can also be used to perform string pattern matching queries against the original text to retrieve it.

[0182] The list of strings encoded in the string index is hereinafter referred to as the "compressed index." The list of strings obtained by decoding the compressed index from the string index is hereinafter referred to as the "uncompressed index."

[0183] The string index provides the following functionality: 1. Count the occurrences of any substring in the list of coded strings as specified in the description below. 2. For each substring found in point 1 above, get the substring's position in the uncompressed index as specified in the description below. 3. Given a start and end position in the uncompressed index, obtain the corresponding decoded payload, which may contain any number of strings, parts of strings, or metadata associated with strings, as specified in the description below. 4. For each substring found in point 1 above, get the entire string containing that substring and the position of the entire string in the uncompressed index, as specified in the description below. 5. For each substring found in point 1 above, get the index of the access unit that contains that substring, as specified in the description below. 6. For each substring found in point 1 above, get the record index of the record that contains that substring, where record index is the 0-based index of that record within the access unit that contains it, as specified in the description below. 7. Given an access unit index, get the position in the uncompressed index of the first string in the access unit that corresponds to the access unit index, as specified in the description below. 8. Given an access unit index, a record index within the access unit corresponding to the access unit index, and a string index within the record corresponding to the record index, obtain the position of the string at the string index contained in the record of the access unit as specified in the description below.

[0184] The inputs to this process are: The variable numAUs specifies the number of access units in which the string is coded within this string index. · The variable codingMode specifies the algorithm used to encode the string index.

[0185] The number of strings coded in each record shall be the same for all records and shall correspond to the variable numStrings specified in the description below.

[0186] [Table 22] TIFF0007775215000031.tif110170 Table 4 - String index block.

[0187] The uncompressed index, encoded in Compress_index, contains a list of strings and associated optional record indexes, ordered by access unit (following the same order as the access units in Table 4) and by record for each access unit (following the same order as the records within the access unit). The total number of strings in the uncompressed index is totNumRecords*numStrings, where totNumRecords is the total number of records for all access units identified by au_id[] and numStrings is a counter of all strings compressed using the compressed indexing algorithm described above.

[0188] An uncompressed index specified as: [Table 23] TIFF0007775215000033.tif193170 The uncompressed index coded in the compress_index element of Table 5 - string_index() element.

[0189] An example of an uncompressed index specified in this disclosure, with numStrings equal to 3, is shown in FIG.

[0190] Semantics The presence of record_index[i] (rec_jdx) is signaled by setting the most significant bit in all bytes of record_index[i]. Setting the most significant bit also prevents false positive results when searching for substrings, because the most significant bit of all bytes in the string[i][j] field is not set, as specified in this disclosure for the string[i][j] element.

[0191] If record_index[i] is present and is N bytes long, it represents a non-negative integer value specified by the following formula:

number

[0192] string[i][j] is the jth coded string of the ith record. The strings must be ordered by access unit (following the same order as the access units in Table 4), and by record within each access unit (following the same order as the records within the access unit).

[0193] The string_terminator is a single byte equal to 0x0A (i.e., "\n").

[0194] Use string indexing to find the location of a substring The location in the uncompressed index of a particular substring is found in the string index as specified in the following pseudocode:

[0195] [Table 24] Table 6 - Finding the location of a substring using string indexing.

[0196] Decoding a subset of string indices String indices are decoded between specific start and end positions, as specified in the following pseudocode:

[0197] [Table 25] Table 7 - Using a string index to decode a substring at a specified position.

[0198] Searching an entire string with a string index Given a location in the uncompressed index, from the list of locations returned by SI_search_substrings() specified in this disclosure, the corresponding entire string and its starting position in the uncompressed index is decoded at the string index specified in the following pseudocode:

[0199] [Table 26] TIFF0007775215000038.tif213170TIFF0007775215000039.tif93170 Table 8 - Searching an entire string using a string index.

[0200] Use a string index to find the access unit ID and record index. Specifies the location within the uncompressed index of the bytes belonging to the string encoded in the compressed index: one location from the list of locations returned by SI_search_substrings() specified in this disclosure, the access unit ID of the access unit containing said string, the index of the record containing said string, and the index of said string within the above record, decoded at the string index as specified in the following pseudocode:

[0201] [Table 27] TIFF0007775215000041.tif208170TIFF0007775215000042.tif212170TIFF0007775215000043.tif154170 Table 9 - Using string indexes to look up access unit and record indexes.

[0202] Use a string index to find the position of the first string in an access unit The location in the uncompressed index of the first string of a particular access unit is obtained using the string index specified in the following pseudocode:

[0203] [Table 28] TIFF0007775215000045.tif64170 Table 10 - Finding the position of the first string in an access unit by string index

[0204] Use a string index to find the position of a string in a record Within the uncompressed index, the location of a string at a particular index within a record (if the record is at a particular index within a particular access unit) is obtained using the string index, as specified in the following pseudocode:

[0205] [Table 29] TIFF0007775215000047.tif213170TIFF0007775215000048.tif215170TIFF0007775215000049.tif212170 Table 11 - Using a string index to find the position of the first string in a record.

[0206] Building a String Index In accordance with the principles of the present invention, a string index is constructed from text descriptors using the following string transformation method. Separating non-indexed from indexed text descriptors for each annotation. Concatenates the indexed textual descriptors, separated by terminators and interleaved with information about the location of the genome annotation record within the access unit.

[0207] Numeric descriptors are represented as numbers, and text descriptors are represented as strings.

[0208] The result of the transformation is further transformed using compressed full-text string indexing algorithms, such as compressed suffix arrays, FM-indexes, and some categories of hash tables, to compress the resulting string index.

[0209] By interleaving information related to genome annotation with the genome annotation record location, the condensed genome annotation data can be browsed according to criteria such as the presence of a string in the record or the genome interval to which the genome record is associated. The browsing is performed by specifying a text string or substring and retrieving all genome annotation records that contain the text as part of their coded annotation.

[0210] An example implementation of this construction method is shown in Figure 5. Each record contains three textual descriptors.

[0211] The text descriptors associated with each genome annotation type described in this disclosure to construct the string index described above and in Figure 5 are selected according to input configuration encoding parameters provided by the user according to the user's requirements / needs, which are coded in the bitstream and / or transmitted from the encoder to the decoder.

[0212] Efficient interpretation of genome annotations By constructing a compressed string index as described above, we can reconstruct the genome annotation associated with a single string descriptor by following the process below.

[0213] The goal of this process is to decode all access units containing annotation data associated with a string identifier specified by a user searching for, for example, a variant name or description, a genome function name or description, or other text descriptor associated with a coded genome annotation. The function SI_search_substrings() specified above is called to search for the desired name or description. If the specified string "str" ​​is present in the compressed index, this call returns one or more positions (named "pos" in this example) as specified in the section "Searching for Substring Positions in a String Index." The access unit ID of the access unit containing the string "str," the index of the record containing the string "str," and the index of the string within the record are decoded in the string index as described in the following points.

[0214] 1. The input byte position "pos" identifies the string str containing the byte at position pos in the uncompressed index. 2. The ID of the access unit containing str is determined by comparing the pos with the value of au_offset[] specified in Table 4 and obtaining the corresponding value of au_id[] specified in Table 4. When pos < au_offset[1], the resulting access unit ID is au_id[0]. When pos >= au_offset[num_AUs - 1], the resulting access unit ID is au_id[num_AUs - 1], where num_AUs is as specified in Table 4. Otherwise, the resulting access unit ID is au_id[i], where the value of i is such that au_offset[i] <= pos < au_offset[i + 1]. 3. By repeatedly calling the function SI_decode() described in this disclosure, decode the compressed index in reverse from position pos - 1 until the entire record index recordIndex (using the record index specified in Table 5) is decoded or until the start of the compressed index is reached. When the start of the compressed index is reached, recordIndex is set to 0. While decoding in reverse, count the number of string terminator record indices (using the string terminator specified in Table 5). However, any non-printable character can be used as the string terminator. 4. Considering the number of indexed strings per record numStrings specified in this disclosure and the access unit determined in step point 2, the index within the access unit of the record containing str is equal to recordIndex + stringIndex / numStrings. 5.When given the number of indexed strings per record numStrings specified in this disclosure and the record determined in step point 4, the index of the string str within the said record is equal to stringIndex % numStrings.

[0215] Access Unit This clause extends the access unit syntax specified in ISO / IEC 23092-1 to support genome annotation data type encoding.

[0216] [Table 30]

[0217] AU header [Table 31] TIFF0007775215000052.tif227170TIFF0007775215000053.tif120170

[0218] Dynamic Attributes 1. Most genome annotation formats contain insufficiently specified fields that complement the minimum set of information defined as essential. In the case of file formats such as VCF, GFF, and GTF, these fields represent valuable information because they contain important classification clues regarding the pathogenicity of a particular variant or elements of functional annotation. Therefore, they cannot be simply discarded or treated as secondary information. In fact, some of these fields may represent the most valuable filter criteria for clinical purposes. 2. For this reason, all these fields are grouped into a set of dynamic attributes across several access unit and dataset types, which are described below. The presence of a given attribute is signaled in a specific section of the parameter set in an object of type "Attribute" specified in this disclosure. 3. Each attribute corresponds to a new descriptor. 4. The presence of a value for a particular record is signaled via a record-level bitmask, using the position of the particular attribute in the parameter set. 5. Attributes are specified in terms of: Value Type The type of array, e.g., 1 if there is a single scalar value, a fixed-size array, an array according to the number of alleles, poidy, or a combination thereof. For example, the GL field of the genotype column in the VCF file. Required array size for fixed-size arrays

[0219] This method provides a unified approach for all different annotation data types, regardless of their nature, and leaves room for future indexing / filtering tools based on the presence of specific attributes.

[0220] [Table 32]

[0221] variant Information about variants is encoded in the data structures described in this section, while information about samples (e.g., genotyping) is encoded in a separate dataset.

[0222] Variant parameters This structure within the parameter set contains the master parameters related to variant coding.

[0223] [Table 33] TIFF0007775215000056.tif219170TIFF0007775215000057.tif166170

[0224] Genome annotation record for the variant Records are sorted in ascending order by pos, and then the positions are coded differentially. Note: ref_len, ref, alt_len, alt, q_int can be coded as the "payload" of the integrated record structure, with the information as attributes.

[0225] The genome annotation record for a variant is encoded using a common genome annotation descriptor and a variant-specific genome annotation descriptor, as described in this disclosure.

[0226] Variant Descriptor Compression As described in this disclosure, information values ​​are compressed as attributes.

[0227] ref and alt information [Table 34] TIFF0007775215000059.tif208170TIFF0007775215000060.tif158170

[0228] Functional annotation (GTF, GFF) Functional annotation parameters This structure within the parameter set contains global configuration parameters related to coding functional annotation data types.

[0229] [Table 35] TIFF0007775215000062.tif139170

[0230] Genome annotation records for functional annotation The genome annotation record for a functional annotation is encoded using a common genome annotation descriptor and a functional annotation-specific genome annotation descriptor, as described in this disclosure.

[0231] Annotation Descriptor Compression [Table 36]

[0232] track Track parameters This structure within the parameter set contains global parameters related to coding the browser track.

[0233] [Table 37] TIFF0007775215000065.tif220170

[0234] Track genome annotation record The genome annotation record for a functional annotation is encoded using a common genome annotation descriptor and a functional annotation-specific genome annotation descriptor, as described in this disclosure.

[0235] Track Descriptor Compression [Table 38]

[0236] Genotype information A genotype dataset contains coded information related to the genotype information of an individual or population.

[0237] Genotype information parameters This structure within the parameter set contains global configuration parameters related to coding of genotype information.

[0238] [Table 39] TIFF0007775215000068.tif135170 format_ID identifies the format fields present in the encoded record. The semantics of each identifier are shown in Table 12. If the value 0x00 (GT) is present, it will always be first in the list.

[0239] Genotype Format Field

[0240] [Table 40] TIFF0007775215000070.tif86170 Table 12 - format_ID values ​​used in genotype_parameters() A = one value for each alternative allele R = one value for each possible allele, including the reference G = one value per genotype

[0241] Genome annotation record of genotype information Genome annotation records of genotype information are encoded using common genome annotation descriptors and genome annotation descriptors specific to the genotype information, as described in this disclosure.

[0242] Compression of genotype information As described in this disclosure, all information is compressed as attributes. Special cases such as GT and LD fields are first split into subsequences identified by subsequencelD, as described below.

[0243] [Table 41] TIFF0007775215000072.tif221170TIFF0007775215000073.tif168170

[0244] Sample information Sample Information Parameters This structure within the parameter set contains global configuration parameters related to coding information about the sample.

[0245] [Table 42]

[0246] Genome annotation record of sample information The genome annotation record for the sample information is encoded using a genome annotation descriptor that is specific to the sample information, as described in this disclosure.

[0247] Expression information This dataset encodes only the actual expression matrix: features are stored in access units of type AU_ANNOTATION, and samples are stored in access units of type AU_SAMPLE.

[0248] Expression parameters This structure within the parameter set contains global configuration parameters related to the coding of utterance information.

[0249] [Table 43] TIFF0007775215000076.tif157170

[0250] The formatID identifies the format fields present in the encoded record. The semantics of each identifier are shown in Table 12.

[0251] Genome annotation record of expression information Genome annotation records for expression information are encoded using genome annotation descriptors specific to the expression information, as described in this disclosure.

[0252] compression The compression strategy is the same as for the genotype dataset: all information is mapped to attributes and compressed, as described in the section titled "Attribute Compression." This allows each element of the matrix to contain multiple values, allowing information such as counts, tpms, and probabilities to be combined with different types and semantics into a single record. A special approach is used for sparse matrices: for each record, only non-zero values ​​are recorded, along with the corresponding position array and the total number of entries.

[0253] Contact Matrix Information A contact matrix (also known as a contact map) is generated by a Hi-C experiment and represents the spatial organization of DNA molecules within the cell nucleus. The two dimensions are genomic positions. The contact matrix value for each coordinate represents a counter of the number of times the two positions in the nucleotide sequence are measured to have an interaction.

[0254] Contact Parameters This structure in the parameter set contains global configuration parameters related to coding information about the contact matrix.

[0255] [Table 44]

[0256] The format_ID identifies the format fields present in the encoded record. The semantics of each identifier are shown in Table 12.

[0257] Genome annotation record of contact matrix information The genome annotation record of the sample information is encoded using a genome annotation descriptor that is specific to the sample information, as described in this disclosure.

[0258] compression The compression strategy is the same as for the contact information dataset.

[0259] attribute [Table 45]

[0260] Attribute Compression The attributes are compressed using as many subsequences as n_attributes+1 in the parameter set.

[0261] [Table 46]

[0262] Data Types This section describes how structured values ​​are represented in this disclosure.

[0263] Value types This is a structure used to represent numbers with sizes in bits.

[0264] [Table 47]

[0265] Type Identifier [Table 48] Table 13 - Data types and their identifiers and parameters

[0266] Sequence Identifier [Table 49] Table 14: Array types and their identifiers

[0267] Data Blocks Data blocks are structures containing compressed descriptors, encapsulated in access units. Each block contains a single type of descriptor, identified by an identifier included in the block header.

[0268] Block Syntax [Table 50]

[0269] Block Header [Table 51]

[0270] Block Payload [Table 52]

[0271] Supported query examples [Table 53] TIFF0007775215000087.tif212170TIFF0007775215000088.tif44170

[0272] Evidence of the Technical Advantage of the Invention The present invention eliminates many of the problems that exist when using state-of-the-art technology, in particular: Currently, there is no unified representation of genome annotation. Instead, a variety of formats are used. It is usually implicitly assumed that features are connected according to their physical proximity on the genome. That is, for example, variants or isoforms are associated with the genes that contain them. The unified representation of data described in this invention makes it possible to express complex relationships between different concepts beyond simple physical containment, such as "the promoter located in this interval and its methylation state (usually external to the gene) are associated with gene A, gene B, and gene C, which form an operon (i.e., a collection of genes each with a different location in the genome)." The present invention allows explicit interfacing with existing parts 1-5 of the MPEG-G standard, where sequencing reads aligned to the genome are represented. Many annotated features (e.g., functional gene models, variants, or tracks representing methylation states or protein binding) are supported by, and sometimes derived from, the presence of sequence reads at the relevant locations. Currently, it is not possible to express concepts such as "This new transcript, composed of this list of exons, is supported by this set of RNA sequence reads" or "This new variant at this position is supported by this set of DNA sequence reads." The present invention allows these concepts to be easily expressed (the latter being of great importance in clinical practice). Currently, no single format exists that can represent all the different existing sources of genome annotation. As a result, pipelines and genome browsers must use a variety of formats to load all the information they need. This invention removes the technical need to implement complex parsers for such domain-specific bioinformatics formats, which are often unclear and lack defined standards. By separating information into different types of access units, the present invention provides a mechanism for implementing efficient compression. Each information stream can be modeled as a homogeneous source with lower entropy, resulting in more efficient compression. Meanwhile, the proposed method still offers the possibility to integrate various information into a single hierarchical architecture, representing the relationships between various genome annotation concepts, genome sequences, and sequence reads. Furthermore, compressing different genome features separately allows users to selectively decompress features of interest if they are only interested in a subset of the data. By employing a family of compressed string indexing algorithms, where one algorithm can be selected at encoding time to compress text information, users can choose the balance they need between string index compression and query time speed. In particular, the use of multiple families of compressed string indexing algorithms is essential to achieve the desired optimization and essential features of the present invention, since employing a single family is insufficient for the purpose. By way of example and not limitation, we will combine two different families of compressed suffix arrays to illustrate the concept. Family [1] uses bit vectors implemented as described in Raman, Rajeev, Venkatesh Raman, and S. Srinivasa Rao. 2002 "Concise Indexable Dictionaries with Applications to k-ary Tree and Multiset Coding." Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA2002), pp. 233-242. Family [2] uses FM-Indexed Hybrid Compression of Bit Vectors. In Proc. 2014 Data Compression Conference (DCC 2014), IEEE Computer We use bit vectors implemented as described in [Society, 2014, pp. 302–311]. As shown in Figure 6, we can vary other parameters of the compressed suffix array family to obtain different compressed suffix array implementations belonging to family [1] (pink dots) and family [2] (cyan dots), exhibiting different values ​​of compression ratio and query speed. However, family [1] is inherently superior in providing higher compression ratios (and slower query speeds), while family [2] is inherently superior in providing faster query speeds (and lower compression ratios). By combining the two families, selecting the set of possible compressed suffix arrays, identified by the black squares, can provide better compression ratio options and better query speed options than would be possible using just one family of compressed suffix arrays.

[0273] Indexing Function [Table 54] TIFF0007775215000090.tif211170TIFF0007775215000091.tif70170

[0274] Genome annotation coding system FIG. 2 illustrates an encoding device according to the principles of the present invention. The encoding device receives input genome annotations, such as variants, browser tracks, functional annotations, methylation patterns and levels, sequencing coverage and statistics, feature expression matrices, contact matrices, and protein affinity for nucleic acids. The annotation data is parsed by a descriptor encoder unit 22, which separates non-indexed descriptors from text-indexed descriptors 212. Non-indexed descriptors common to all annotations are provided to a conversion unit 21. Non-indexed descriptors specific to each annotation type are provided to a conversion unit 27. Text-indexed descriptors are provided to a descriptor-to-string conversion unit 26. The outputs of the conversion units 21 and 27 are provided to different entropy coders 24 according to the specific statistical characteristics of each converted descriptor. At least one first entropy encoder (24) is used for numeric descriptors, and at least one second entropy encoder (214) is used for text descriptors not included in the subset of text descriptors (29).

[0275] The output of each entropy coder is provided to an annotation data access unit coder 23 to generate annotation data access units 25. The output of the descriptor string index conversion unit 26, an uncompressed master annotation index 210, is provided to an annotation data index coder 28 to generate master annotation index data 29. One annotation data index is associated with one or more annotation data access units. Figure 1 shows that annotation data access units (122) are coded (118) together with master annotation index data (123) and a first class of access units (119) containing compressed genome sequencing data.

[0276] The transformations applied by the descriptor transformation units 21 and 27 used in the coding device include: Run-length coding: A sequence of numbers is represented by a counter of the number of consecutive occurrences and a value for that number. Differential coding: Each number is expressed as a delta relative to the previously coded value. Byte Separation: For numbers represented by many bytes, each byte is treated separately and compressed with other bytes that have similar properties in terms of bit organization.

[0277] The transformations applied by the annotation data index coder 28 include: Burroughs-Wheeler Conversion Compressed string pattern matching Compressed suffix arrays, FM Index Hash algorithm

[0278] The advantage of applying the transformation to numeric descriptors is that it improves compression efficiency without losing information, as known to those skilled in the art. The coding of string descriptors is made more efficient by the transformation because the transformed representation is more efficiently browsable and searchable for substrings. The coding of string descriptors is made more efficient by the transformation because the transformed representation is more efficiently browsable and searchable for substrings. Once the original text is transformed, the existence of substrings can be determined without having to decompress the entire text.

[0279] Genome annotation decoding device A decoding device implemented in accordance with the principles of the present disclosure extends the functionality of an ISO / IEC 23092 compliant decoding device, as shown in FIG.

[0280] FIG. 3 illustrates a decoding apparatus according to the principles of the present disclosure. A genome annotation access unit decoder 31 receives access units 30 from a stream demultiplexer 70 and extracts the entropy-coded payload of the access units. Entropy decoders 32, 33, and 34 receive the entropy-coded extracted payloads and decode different types of genome annotation descriptors into their binary representations 35. The binary representation of the common descriptor for all genome annotations is then provided to an inverse transformation unit 36. The binary representations of the descriptors specific to each annotation data type are provided to an inverse transformation unit 314. A master annotation index 38 is provided to an indexed access unit information retrieval unit 37, which identifies the text fields belonging to each AU within the string index. Such location information 313 is then provided to an indexed information decoding unit 39, which decodes the text fields from the string index. The decoded text fields are then provided to a descriptor decoder unit 310 to reconstruct the decoded genome annotations 311.

[0281] Genome annotation text search system A text search device implemented in accordance with the principles of the present disclosure extends the functionality of an ISO / IEC 23092 compliant decoding device, as shown in FIG.

[0282] FIG. 4 illustrates a decoding apparatus according to the principles of the present disclosure. A genome annotation access unit decoder 41 receives access units 40 from a stream demultiplexer 70 and extracts the entropy-coded payloads of the access units. Entropy decoders 42, 43, and 44 receive the entropy-coded extracted payloads and decode different types of genome annotation descriptors into their binary representations 45. In some configurations of the decoding apparatus, different types or kinds of access units can be selectively extracted. The binary representation of the common descriptor for all genome annotations is then provided to an inverse transformation unit 46. The binary representations of the descriptors specific to the annotation data type are provided to an inverse transformation unit 414. A master annotation index 48 is provided to an indexed access unit information retrieval unit 47, which finds text fields matching a text query 413 within a string index. Such location information 415 is then provided to an indexed information decoding unit 49, which decodes the text fields from the string index. The decoded text fields are then fed to a descriptor decoder unit 410 to reconstruct the decoded genome annotations 411 .

[0283] FIG. 8 shows how the conceptual organization of data described in this invention prepares it for performing text queries.

[0284] Master Index Table Associate Genome interval (sequence ID + start position + end position + data class) and Access units containing compressed genome sequencing reads and associated alignment information and metadata

[0285] Annotation Index Associate A string index containing text information about features in a compressed, searchable format and · Access units, including: Compressed genome annotation and Information about the genomic interval to which they belong

[0286] A single query for the text string "APOBEC" can retrieve all associated annotations containing the text "APOBEC" and associated code sequence reads.

[0287] FIG. 9 illustrates how the conceptual organization of data described in this invention allows searches across genomic intervals to be performed.

[0288] Master Index Table Associate Genome interval (sequence ID + start position + end position + data class) and Access units containing compressed genome sequencing reads and associated alignment information and metadata

[0289] Annotation Index Associate Genome interval (sequence ID + start position + end position + data class) and Access units containing compressed genome annotations and A string index containing text information about features in a compressed, searchable format

[0290] A single query against a genomic interval N retrieves all coding sequence reads and associated annotations.

[0291] The techniques of the present invention disclosed herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, they are stored on a computer medium and executed by a hardware processing unit. The hardware processing unit may comprise one or more processors, digital signal processors, general-purpose microprocessors, application-specific integrated circuits, or other discrete logic circuitry.

[0292] The techniques of this disclosure may be implemented in a variety of devices or apparatuses, including mobile phones, desktop computers, servers, tablets, and similar devices.

Claims

1. 1. A computer-implemented method for storing or transmitting a representation of genome sequencing data in a genome file format that includes annotation data associated with the genome sequencing data, comprising: the genome sequencing data comprises nucleotide sequence reads; aligning (10) the reads to one or more reference sequences to generate aligned reads; classifying (14) the aligned reads according to classification rules based on mapping of the aligned reads to the one or more reference sequences to create (18) classes of the aligned reads; entropy coding the classified aligned reads as a block of multiple descriptors; structuring the blocks of descriptors with header information to create a first class of access units (119) containing genome sequencing data; The method further includes encoding the annotation data (12) into different access units (122) of a second class and indexing the data into a master annotation index (MAI, 123, 211), the indexed data representing a coded form of the annotation string data (212) obtained by using at least one compressed string indexing algorithm (28) on the annotation string data (212), the MAI associating the coded annotation strings with the access units of the second class; The method further includes jointly coding the first and second category access units and the MAI; The encoding of the annotation data and the indexing data comprises: encoding (22) genome annotation data (20) as genome annotation descriptors (29, 212), the genome annotation descriptors including numeric descriptors and text descriptors, the encoding comprising: - selecting a subset of textual descriptors (212) from said textual descriptors, in particular according to configuration parameters (213) provided by a user, - transforming (26) said subset (212) of text descriptors by generating a string index (210) using a first string transformation method; - converting and encoding (28) said string index (210) using a string index conversion method to generate master annotation index data (211); - transforming (21, 27) said textual descriptors (29) that are not included in said subset of numerical and textual descriptors using at least one second transformation method (21, 27) different from the first transformation method, encoding (24, 23) the transformed numerical descriptors and textual descriptors (29) not included in the subset of textual descriptors into separate access units (25) of a second class using at least one first entropy encoder (24) for the numerical descriptors and at least one second entropy encoder (214) for the textual descriptors (29) not included in the subset of textual descriptors.

2. The method described in claim 1, wherein the second classification access unit containing the annotation data further includes information data identifying genomic intervals (80), and the genomic intervals identify nucleotide sequences in one or more reference sequences such that the annotation data contained in the second classification access unit is associated with associated coded reads of the genomic sequence contained in the first classification access unit containing the genomic sequencing data.

3. The first character string conversion method (26) includes: - inserting after each of said text descriptors (51, 52, 53) a string terminator (55) character to signal the end of said text descriptors (51, 52, 53), - concatenating said textual descriptors (51, 52, 53), A method according to claim 1 or 2, comprising interleaving genome annotation record index data (54) to associate said text descriptors (51, 52, 53) with the location of genome annotation records within said second classification of access units.

4. The method of any one of claims 1 to 3, wherein the string index transformation method (28) is one of string pattern matching, suffix array, FM index, and hash table.

5. 5. The method according to claim 1, wherein the at least one second string transformation method (21, 27) is one of differential coding, run-length coding, byte separation, and entropy coders such as CABAC, Huffman coding, arithmetic coding, range transformation.

6. 6. The method of claim 1, wherein the master annotation index data includes in its header the number of AU types and the number of indexes for each AU type.

7. 7. The method of claim 1, further comprising coding the classified unaligned reads.

8. A method for identifying a genome data multiplex (710) comprising: receiving a genome data multiplex (710) including access units of a first classification, access units of a second classification, and a master annotation index (MAI, 713); Parsing the access units of the first classification to obtain aligned reads classified thereby; Parsing (713) the data in the master annotation index; selectively decoding the classified nucleotide sequence reads on one or more reference sequences to generate uncompressed nucleotide sequence reads; applying a string index inversion method to the master annotation index, thereby obtaining a string index (210); applying a first inverse string transformation method to the string index (210), thereby obtaining a subset of text descriptors (212); Decoding the second category of access units comprises: applying at least one first entropy decoder to the second class of access units, thereby obtaining transformed numerical descriptors; applying at least one second entropy decoder to the second class of access units, thereby obtaining transformed text descriptors; applying a second string inversion method different from the first string inversion method to the transformed numeric descriptors and the transformed textual descriptors, thereby obtaining numeric descriptors and textual descriptors, respectively, that are not included in the subset of textual descriptors; wherein the numerical descriptors and the text descriptors are genome annotation descriptors; 10. A computer-implemented method for sequencing and extracting nucleotides and genome annotation data encoded according to the method of claim 1, further comprising decoding the genome annotation descriptor, thereby obtaining genome annotation data.

9. 9. The method of claim 8, further comprising decoding information data associated with genomic intervals (80), the genomic intervals identifying nucleotide sequences in the one or more reference sequences such that the annotation data is associated with coded reads of the associated genomic sequence.

10. A method as described in claim 8 or 9, further comprising deinterleaving genome annotation record index data (54) to obtain information defining the association between the text descriptors (51, 52, 53) and the location of the genome annotation record within the access unit of the second classification, deconcatenating the text descriptors (51, 52, 53) based on an end-of-string character (55) located after each text descriptor (51, 52, 53), and signaling the end of each text descriptor (51, 52, 53).

11. A method described in any one of claims 8 to 10, wherein the string index inverse conversion method is one of string pattern matching, suffix array, FM index, and hash table.

12. A method according to any one of claims 8 to 11, wherein the at least one second inverse transformation method is one of differential coding, run-length coding, byte separation, and entropy coders such as CABAC, Huffman coding, arithmetic coding, and range conversion.

13. A method according to any one of claims 8 to 12, wherein the header of the Master Annotation Index (MAI) includes the number of AU types and the number of indexes for each AU type.

14. A method described in any one of claims 8 to 13, further comprising decoding the classified unaligned reads.

15. A genome encoder configured to perform a method according to any one of claims 1 to 7.

16. A method configured to perform the method of any one of claims 8 to 14. Genome decoder.

17. A computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 7.

18. A computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform a method according to any one of claims 8 to 14.

Images