Ultra-long context genomic language modeling

Abstract

Described are systems and methods for analyzing ultra-long context biological sequence data, comprising processing the biological sequence data using a machine learning model.

Description

Attorney Docket No.68993-715601 Ultra-Long Context Genomic Language Modeling Cross-Reference

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 664,292,filed June 26, 2024, which is incorporated by reference herein in its entirety. BACKGROUND

[0002] Whole-genome sequencing (WGS) and whole-exome sequencing (WES) generateextensive datasets comprising long DNA and RNA segments along with associated epigenetic signatures, such as methylation patterns. SUMMARY

[0003] The task of scaling attention mechanisms across long segments totaling millions orbillions of nucleotides may be non-trivial. Applying full attention has a computational complexity that increases quadratically with context length. This has made it difficult to scale transformer-based models to very long inputs such as genome sequences.

[0004] Recognized herein is a need for systems and methods that may utilize ultra-long (e.g.,genome length) inputs in a machine learning setting.

[0005] In an aspect, the present disclosure provides a method for analyzing biological sequencedata, comprising: obtaining biological sequence data; and processing the biological sequence data using a machine learning model comprising a linearized attention mechanism produce an output.

[0006] In some embodiments, the method comprises processing the output to generate apredicted effect of a mutation on a biological process.

[0007] In some embodiments, the biological process is a molecular function, a molecularproperty, a cellular function, a tissue function, or any combination thereof.

[0008] In some embodiments, the method comprises processing the output to design molecularcomplexes.

[0009] In some embodiments, the method comprises processing the output to predict geneinteractions.

[0010] In some embodiments, the method comprises processing the output to predict an effectof an extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.Attorney Docket No.68993-715601

[0011] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA.

[0012] In some embodiments, the biological sequence data comprises data derived fromsequencing RNA.

[0013] In some embodiments, the biological sequence data comprises data derived fromsequencing a peptide.

[0014] In some embodiments, the biological sequence data comprises data derived fromsequencing a protein.

[0015] In some embodiments, the biological sequence data comprises data derived fromepigenomic sequencing.

[0016] In some embodiments, the biological sequence data comprises data derived fromsequencing mitochondrial DNA.

[0017] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.

[0018] In some embodiments, the input data comprises textual data provided by a user.

[0019] In some embodiments, the textual data comprises written instructions that are processedby the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

[0020] In some embodiments, the machine learning model comprises an LLM.

[0021] In some embodiments, the output comprises a sequence of tokens.

[0022] In some embodiments, the output comprises at least 650 tokens.

[0023] In some embodiments, the input data comprises a sequence of tokens.

[0024] In some embodiments, the sequence of tokens are single nucleotide resolution.

[0025] In some embodiments, the sequence of tokens is at least 2 million tokens in length.

[0026] In some embodiments, the sequence of tokens is at least 3 million tokens in length.

[0027] In some embodiments, the sequence of tokens is at least 4 million tokens in length.

[0028] In some embodiments, the sequence of tokens is at least 5 million tokens in length.

[0029] In some embodiments, the sequence of tokens is at least 6 million tokens in length.

[0030] In some embodiments, the input data comprises a sequence of at most 2.5 milliontokens.

[0031] In some embodiments, the input data comprises a sequence of at most 3 million tokens.

[0032] In some embodiments, the input data comprises a sequence of at most 4 million tokens.Attorney Docket No.68993-715601

[0033] In some embodiments, the input data comprises a sequence of at most 5 million tokens.

[0034] In some embodiments, the input data comprises a sequence of at most 6 million tokens.

[0035] In some embodiments, the method comprises predicting the likelihood of associationof a genetic variation to a phenotype.

[0036] In some embodiments, the phenotype comprises at least one of an autoimmune disease,a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

[0037] In some embodiments, the phenotype comprises early onset of a disease.

[0038] In some embodiments, the phenotype comprises developmental delays.

[0039] In some embodiments, the present disclosure provides a method comprises, based atleast on the association of the genetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.

[0040] In some embodiments, determining that an elevated risk of having the phenotypecomprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

[0041] In some embodiments, determining the elevated risk of having the phenotype comprisesa specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

[0042] In some embodiments, the method comprises assaying a sample from a subject togenerate sequencing information.

[0043] The method of claim 38, wherein the sample is a buccal sample, a blood sample, asaliva sample, a urine sample, a cell sample, or a tissue sample.

[0044] In some embodiments, the method comprises extracting DNA from the bodily sample.

[0045] In some embodiments, the method comprises subjecting the DNA to a sequencingreaction.

[0046] In some embodiments, the sequencing reaction comprises a whole genome sequencingreaction, or an exome sequencing reaction, or both.

[0047] In an aspect the present disclosure provides or describes a method comprising:implementing a model comprising a set of layers comprising a linearized attention mechanism and configured to take an input, comprising a first sequence of tokens, and produce an output, comprising a second sequence of tokens; using an optimizer to improve the model’s performance by altering the set of layers thereby causing the output to be altered; receiving a set of data; processing by the model the set of data as input thereby producing the output; causing the optimizer to alter the set of layers based at least in part on the output and a measure of error of the model; and storing the model.Attorney Docket No.68993-715601

[0048] In some embodiments, at least one token of the first sequence of tokens is masked andwherein the second sequence of tokens predicts the at least one token that is masked in the first sequence of tokens.

[0049] In some embodiments, the measure of error is based at least in part on the at least onetoken that is masked in the first sequence of tokens and the prediction of the at least one token in the second sequence of tokens.

[0050] In some embodiments, the method comprises processing the output to generate apredicted effect of a mutation on a biological process.

[0051] In some embodiments, the biological process is a molecular function, a molecularproperty, a cellular function, a tissue function, or any combination thereof.

[0052] In some embodiments, the method comprises processing the output to design molecularcomplexes.

[0053] In some embodiments, the method comprises processing the output to predict geneinteractions.

[0054] In some embodiments, the method comprises processing the output to predict an effectof an extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

[0055] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA.

[0056] In some embodiments, the biological sequence data comprises data derived fromsequencing RNA.

[0057] In some embodiments, the biological sequence data comprises data derived fromsequencing a peptide.

[0058] In some embodiments, the biological sequence data comprises data derived fromsequencing a protein.

[0059] In some embodiments, the biological sequence data comprises data derived fromepigenomic sequencing.

[0060] In some embodiments, the biological sequence data comprises data derived fromsequencing mitochondrial DNA.

[0061] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.

[0062] In some embodiments, the input data comprises textual data provided by a user.Attorney Docket No.68993-715601

[0063] In some embodiments, the textual data comprises written instructions that are processedby the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

[0064] In some embodiments, the machine learning model comprises an LLM.

[0065] In some embodiments, the output comprises a sequence of tokens.

[0066] In some embodiments, the output comprises at least 650 tokens.

[0067] In some embodiments, the input data comprises a sequence of tokens.

[0068] In some embodiments, the sequence of tokens are single nucleotide resolution.

[0069] In some embodiments, the sequence of tokens is at least 2 million tokens in length.

[0070] In some embodiments, the sequence of tokens is at least 3 million tokens in length.

[0071] In some embodiments, the sequence of tokens is at least 4 million tokens in length.

[0072] In some embodiments, the sequence of tokens is at least 5 million tokens in length.

[0073] In some embodiments, the sequence of tokens is at least 6 million tokens in length.

[0074] In some embodiments, the input data comprises a sequence of at most 2.5 milliontokens.

[0075] In some embodiments, the input data comprises a sequence of at most 3 million tokens.

[0076] In some embodiments, the input data comprises a sequence of at most 4 million tokens.

[0077] In some embodiments, the input data comprises a sequence of at most 5 million tokens.

[0078] In some embodiments, the input data comprises a sequence of at most 6 million tokens.

[0079] In some embodiments, the method comprises predicted likelihood of association of agenetic variation to a phenotype.

[0080] v wherein the phenotype comprises at least one of an autoimmune disease, a cancer, aneurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

[0081] In some embodiments, the phenotype comprises early onset of a disease.

[0082] In some embodiments, the present disclosure provides a method where the phenotypecomprises developmental delays.

[0083] In some embodiments, the method comprises, based at least on the association of thegenetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.

[0084] In some embodiments, determining that an elevated risk of having the phenotypecomprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

[0085] In some embodiments, determining the elevated risk of having the phenotype comprisesa specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.Attorney Docket No.68993-715601

[0086] In some embodiments, the method comprises assaying a sample from a subject togenerate sequencing information.

[0087] In some embodiments, the sample is a buccal sample, a blood sample, a saliva sample,a urine sample, a cell sample, or a tissue sample.

[0088] In some embodiments, the method comprises extracting DNA from the bodily sample.

[0089] In some embodiments, the method comprises subjecting the DNA to a sequencingreaction.

[0090] In some embodiments, the sequencing reaction comprises a whole genome sequencingreaction, or an exome sequencing reaction, or both.

[0091] In some embodiments, the processing is autoregressive.

[0092] In some aspects the present disclosure provides or describes a method for analyzingbiological sequence data, comprising:(a) obtaining biological sequence data;(b) generating a sequence of tokens at least 2 million tokens in length based at least in part on the biological sequence data; and(c) processing the sequence of tokens using a machine learning model to produce an output.

[0093] In some embodiments, the method comprises processing the output to generate apredicted effect of a mutation on a biological process.

[0094] In some embodiments, the biological process is a molecular function, a molecularproperty, a cellular function, a tissue function, or any combination thereof.

[0095] In some embodiments, the method comprises processing the output to design molecularcomplexes.

[0096] In some embodiments, the method comprises processing the output to predict geneinteractions.

[0097] In some embodiments, the method comprises processing the output to predict an effectof an extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

[0098] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA.

[0099] In some embodiments, the biological sequence data comprises data derived fromsequencing RNA.

[0100] In some embodiments, the biological sequence data comprises data derived fromsequencing a peptide.

[0101] In some embodiments, the biological sequence data comprises data derived fromsequencing a protein.Attorney Docket No.68993-715601

[0102] In some embodiments, the biological sequence data comprises data derived fromepigenomic sequencing.

[0103] In some embodiments, the biological sequence data comprises data derived fromsequencing mitochondrial DNA.

[0104] In some embodiments, the biological sequence data comprises data derived fromsequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.

[0105] In some embodiments, the input data comprises textual data provided by a user.

[0106] In some embodiments, the textual data comprises written instructions that are processedby the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

[0107] In some embodiments, the machine learning model comprises an LLM.

[0108] In some embodiments, the output comprises a sequence of tokens.

[0109] In some embodiments, the output comprises at least 650 tokens.

[0110] In some embodiments, the input data comprises a sequence of tokens.

[0111] In some embodiments, the sequence of tokens are single nucleotide resolution.

[0112] In some embodiments, the sequence of tokens is at least 2 million tokens in length.

[0113] In some embodiments, the sequence of tokens is at least 3 million tokens in length.

[0114] In some embodiments, the sequence of tokens is at least 4 million tokens in length.

[0115] In some embodiments, the sequence of tokens is at least 5 million tokens in length.

[0116] In some embodiments, the sequence of tokens is at least 6 million tokens in length.

[0117] In some embodiments, the input data comprises a sequence of at most 2.5 milliontokens.

[0118] In some embodiments, the input data comprises a sequence of at most 3 million tokens.

[0119] In some embodiments, the input data comprises a sequence of at most 4 million tokens.

[0120] In some embodiments, the input data comprises a sequence of at most 5 million tokens.

[0121] In some embodiments, the input data comprises a sequence of at most 6 million tokens.

[0122] In some embodiments, the predicted likelihood of association of a genetic variation toa phenotype.

[0123] In some embodiments, the phenotype comprises at least one of an autoimmune disease,a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

[0124] In some embodiments, the phenotype comprises early onset of a disease.Attorney Docket No.68993-715601

[0125] In some embodiments, the phenotype comprises developmental delays.

[0126] In some embodiments, , based at least on the association of the genetic variation withthe phenotype, determining that the subject has an elevated risk of having a phenotype.

[0127] In some embodiments, determining that an elevated risk of having the phenotypecomprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

[0128] In some embodiments, determining the elevated risk of having the phenotype comprisesa specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

[0129] In some embodiments, the present disclosure provides a method comprising assaying asample from a subject to generate sequencing information.

[0130] In some embodiments, the sample is a buccal sample, a blood sample, a saliva sample,a urine sample, a cell sample, or a tissue sample.

[0131] In some embodiments, the present disclosure provides a method comprising extractingDNA from the bodily sample.

[0132] In some embodiments, the present disclosure provides a method comprising subjectingthe DNA to a sequencing reaction.

[0133] In some embodiments, the sequencing reaction comprises a whole genome sequencingreaction, or an exome sequencing reaction, or both.

[0134] Additional aspects and advantages of the present disclosure will become readilyapparent from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive. INCORPORATION BY REFERENCE

[0135] All publications, patents, and patent applications mentioned in this specification areherein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the present disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material. The publication entitled “M5: A Whole Genome Bacterial Encoder at Single Nucleotide Resolution” published July 3, 2024 at arxiv.org / abs / 2407.03392 is incorporated by reference herein in its entirety.Attorney Docket No.68993-715601 BRIEF DESCRIPTION OF THE DRAWINGS

[0136] The novel features of the present disclosure are set forth with particularity in theappended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the present disclosure are utilized, and the accompanying drawings of which:

[0137] FIG. 1 illustrates a pipeline for training and inference, demonstrating an unsupervisedpre-training phase followed by supervised fine-tuning to predict phenotypes or disease risks, using genomic segments and labeled data.

[0138] FIG. 2 shows an example pipeline for generating embedding vectors from a model orseries of models.

[0139] FIG. 3 shows an example of segmenting sequence data and generation of embeddings.

[0140] FIG. 4 shows example of data approximating on the interval [-1,2].

[0141] FIG. 5 shows example performance data on varied context lengths.

[0142] FIG. 6 shows example performance data for single nucleotide prediction models.

[0143] FIG. 7 shows example distributions for attention heads on a test set of data.

[0144] FIG. 8 shows example distributions for attention heads on a wide range of data.

[0145] FIG. 9 shows example computations to facilitating precise and computationallyefficient approximation of attention mechanisms.

[0146] FIG. 10 shows example computations used to materialize the key-specific kernelfunction as part of polynomial-approximated attention mechanisms.

[0147] FIG. 11 shows example computations for a query-specific, locus-aware polynomial-approximated kernel function, incorporating locus-dependent normalization shifts and approximation adjustments.

[0148] FIG. 12 shows example pseudocode for a forward pass of multi-head attention usinghybrid dynamic linear kernels, significantly reducing computational complexity of quadratic attention to enable scalability for large genomic datasets.

[0149] FIG. 13 shows an example of a transformer-based genomic language modelarchitecture.

[0150] FIG. 14 shows an example of input context sharding across multiple computationaldevices.

[0151] FIG. 15 shows an example of genomic segments for use as inputs in training andinference.Attorney Docket No.68993-715601

[0152] FIG. 16 shows an example of a fine-tuning pipeline for training downstream models.

[0153] FIG. 17 shows an example of a computing system.

[0154] FIG. 18 shows an example of a networked system.

[0155] FIG. 19 shows an example of a system utilizing an application (e.g., app).DETAILED DESCRIPTION

[0156] While various embodiments of the present disclosure have been shown and describedherein, such embodiments are provided by way of example only. Numerous variations, changes, or substitutions may occur without departing from the present disclosure. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed.

[0157] Throughout this application, various embodiments may be presented in a range format.It may be understood that the description in range format is merely for convenience and brevity and may not be construed as an inflexible limitation on the scope of the disclosure. Accordingly, the description of a range may be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 may be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0158] The term “about” or “approximately” means within an acceptable error range for theparticular value, which may depend in part on how the value is measured or determined, e.g., the limitations of the measurement system. For example, “about” can mean within 1 or more than 1 standard deviation. Alternatively, “about” can mean a range of up to 20%, up to 10%, up to 5%, or up to 1% of a given value. Alternatively, particularly with respect to biological systems or processes, the term can mean within an order of magnitude, preferably within 5- fold, and more preferably within 2-fold, of a value. Where particular values are described in the application and claims, unless otherwise stated the term “about” meaning within an acceptable error range for the particular value may be assumed.

[0159] As used herein, the singular forms “a,” “an,” and “the” include plural references unlessthe context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0160] As used herein, the phrases “at least one,” “one or more,” and “and / or” are open-endedexpressions that are both conjunctive and disjunctive in operation. For example, each of theAttorney Docket No.68993-715601 expressions “at least one of A, B and C,” “at least one of A, B, or C,” “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. As used herein, the phrase “at most three” can mean less than one, one, two, or three.

[0161] As used herein, the words “comprising” (and any form of comprising, such as“comprise” and “comprises”), “having” (and any form of having, such as “have” and “has”), “including” (and any form of including, such as “includes” and “include”) or “containing” (and any form of containing, such as “contains” and “contain”) are inclusive or open-ended and do not exclude additional, unrecited elements or method operations. It is contemplated that any embodiment discussed in this specification can be implemented with respect to any method or composition of the present disclosure, and vice versa. Furthermore, compositions of the present disclosure can be used to achieve methods of the present disclosure.

[0162] Reference throughout this specification to “some embodiments,” “furtherembodiments,” or “a particular embodiment,” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in some embodiments,” or “in further embodiments,” or “in a particular embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0163] The terms “determining”, “assessing”, “assaying,” and “analyzing” are often usedinterchangeably herein to refer to forms of measurement. The terms can include determining if an element is present or not (for example, detection). These terms can include quantitative, qualitative, or quantitative and qualitative determinations. Assessing can be relative or absolute. “Detecting the presence of” can include determining the amount of something present in addition to determining whether it is present or absent depending on the context.

[0164] The terms "subject," "individual," and "patient" may be used interchangeably and referto humans, as well as non-human mammals (e.g., non-human primates, canines, equines, felines, porcines, bovines, ungulates, lagomorphs, rodents, and the like). In various embodiments, the subject can be a human (e.g., adult male, adult female, adolescent male, adolescent female, male child, female child) under the care of a physician or other health worker in a hospital, as an outpatient, or other clinical context. In certain embodiments, the subject may not be under the care or prescription of a physician or other health worker. In some embodiments, the subject may be under the care of a dental professional.Attorney Docket No.68993-715601

[0165] As used herein, “treatment” or “treating” refers to an approach for obtaining beneficialor desired results with respect to a disease, disorder, or medical condition including, but not limited to, a therapeutic benefit and / or a prophylactic benefit. In certain embodiments, treatment or treating involves administering a therapeutic to a subject. A therapeutic benefit may include the eradication or amelioration of the underlying disorder being treated. Also, a therapeutic benefit may be achieved with the eradication or amelioration of one or more of the physiological symptoms associated with the underlying disorder, such as observing an improvement in the subject, notwithstanding that the subject may still be afflicted with the underlying disorder.

[0166] The term “cancer” is used herein to refer to any disease characterized by uncontrolledcell division. A cancer can be a cancer of the blood (e.g., hematological cancer), e.g., leukemia, lymphoma, or multiple myeloma, or a cancer can be neoplastic, e.g., associated with an abnormal mass of tissue in which cells grow and divide more than they should or do not die when they should. Neoplastic cancers, e.g., lung, breast or liver cancer, are associated with a solid tumor.

[0167] The term “nucleic acid” and “polynucleotide” are used interchangeably herein todescribe a polymer of any length, e.g., greater than about 2 bases, greater than about 10 bases, greater than about 100bases, greater than about 500 bases, greater than 1000 bases, greater than 10,000 bases, greater than 100,000 bases, greater than about 1,000,000, up to about 1010 or more bases composed of nucleotides, e.g., deoxyribonucleotides or ribonucleotides, and may be produced enzymatically or synthetically (e.g., PNA as described in U.S. Patent No. 5,948,902 and the references cited therein) which can hybridize with naturally occurring nucleic acids in a sequence specific manner analogous to that of two naturally occurring nucleic acids, e.g., can participate in Watson-Crick base pairing interactions. Naturally occurring nucleotides include guanine, cytosine, adenine, thymine, uracil (G, C, A, T and U respectively). DNA and RNA have a deoxyribose and ribose sugar backbone, respectively, whereas PNA's backbone is composed of repeating N-(2-aminoethyl)-glycine units linked by peptide bonds. In PNA various purine and pyrimidine bases are linked to the backbone by methylene-carbonyl bonds. A locked nucleic acid (LNA), often referred to as inaccessible RNA, is a modified RNA nucleotide. The ribose moiety of an LNA nucleotide is modified with an extra bridge connecting the 2' oxygen and 4' carbon. The bridge "locks" the ribose in the 3'-endo (North) conformation, which is often found in the A-form duplexes. LNA nucleotides can be mixed with DNA or RNA residues in the oligonucleotide whenever desired.Attorney Docket No.68993-715601

[0168] As used herein, a “sample” or a “biological sample” refers to anything capable of beinganalyzed by the methods or systems disclosed herein. A sample may be derived from a subject.

[0169] As used herein, “selective enrichment” or “selectively enriching” refers to any processwhere a concentration of a target sequence, or a target (gene) locus, is increased relative to its initial concentration. Such processes may amplify the target sequence or target locus to increase the concentration prior to sequencing. Such processes may capture the target sequence or target locus to increase the concentration prior to sequencing. Such processes may include hybrid capture methods.

[0170] As used herein, “sequencing” refers to any of a number of technologies used todetermine the sequence (e.g., the identity and order of monomer units) of a biomolecule, e.g., a nucleic acid such as DNA or RNA. Examples of sequencing methods include, but are not limited to, targeted sequencing, single molecule real-time sequencing, exon or exome sequencing, intron sequencing, electron microscopy-based sequencing, panel sequencing, transistor-mediated sequencing, direct sequencing, random shotgun sequencing, Sanger dideoxy termination sequencing, whole-genome sequencing, sequencing by hybridization, pyrosequencing, duplex sequencing, cycle sequencing, single-base extension sequencing, solid phase sequencing, high-throughput sequencing, massively parallel signature sequencing, emulsion PCR, co-amplification at lower denaturation temperature-PCR (COLD-PCR), multiplex PCR, sequencing by reversible dye terminator, paired-end sequencing, near-term sequencing, exonuclease sequencing, sequencing by ligation, short-read sequencing, single molecule sequencing, sequencing-by-synthesis, real-time sequencing, reverse-terminator sequencing, long-read sequencing, nanopore sequencing, 454 sequencing, Solexa Genome Analyzer sequencing, SOLiD sequencing, MS-PET sequencing, and a combination thereof. In some embodiments, sequencing can be performed by a gene analyzer such as, for example, gene analyzers commercially available from Illumina, Inc., Ultima Genomics, Pacific Biosciences, Inc., or Applied Biosystems / Thermo Fisher Scientific, among many others.

[0171] The term “next-generation sequencing” or “NGS”, as used herein, generally refers tosequencing technologies having increased throughput as compared to traditional Sanger- and capillary electrophoresis-based approaches, for example, with the ability to generate hundreds of thousands of relatively small sequence reads at a time. Some examples of next-generation sequencing techniques include, but are not limited to, sequencing by synthesis, sequencing by ligation, and sequencing by hybridization. In some embodiments, next-generation sequencing includes the use of instruments capable of sequencing single molecules.Attorney Docket No.68993-715601

[0172] In some embodiments, the nucleic acid molecules can be obtained or derived from abiological sample. The biological sample can be obtained or derived from a subject. The subject can comprise a human subject. The subject can be an adult. The subject can be a child. The subject can be an infant. The subject can be a newborn. In some embodiments, the subject can be a fetus. In some embodiments, the subject can be less than one year old. In some embodiments, the subject can be between seconds old and months old. In some embodiments, the subject can be between about one second old and one hour old. In some embodiments, the subject can be about a day old, a week old, a month old, or more than a month old. In some embodiments, the subject can be about 2 months old, 3 months old, 4 months old, 5 months old, 6 months old, 7 months old, 8 months old, 9 months old, 10 months old, 11 months old, 12 months old, or more than 12 months old. In some embodiments, the subject can be about 1 year old, 2 years old, 3 years old, 4 years old, 5 years old, 6 years old, 7 years old, 8 years old, 9 years old, 10 years old, 11 years old, 12 years old, 13 years old, 14 years old, 15 years old, 16 years old, 17 years old, 18 years old, or more than 18 years old. In some embodiments, the subject can be an adult. In some embodiments, the subject can be between about 18 and 100 years old.

[0173] In some embodiments, the subject may be a mammal. For example, the mammal mayinclude a mouse, a rat, a gerbil, a guinea pig, a hamster, a fox, a dog, a monkey, a sheep, a cow, a pig, or the like. The mammal may include a monkey. For example, the mammal may comprise a chimpanzee, a bonobo, an orangutan, a baboon, or the like. The subject may be a human. The subject may be an adult (e.g., at least 18 years of age). The subject may be a child (e.g., less than 18 years of age). The subject may be a male. The subject may be a female.

[0174] In some embodiments, the biological sample can comprise one or more of: a bloodsample, a plasma sample, a serum sample, a urine sample, a saliva sample, a cerebrospinal fluid sample, a stool sample, a lymph node sample, or a bone marrow sample, or any combination thereof. The biological sample may comprise a blood sample. The biological sample may comprise a plasma sample. The biological sample may comprise a serum sample. The biological sample may comprise a urine sample. The biological sample may comprise a saliva sample. The biological sample may comprise a cerebrospinal fluid sample. The biological sample may comprise a stool sample. The biological sample may comprise a lymph node sample. The biological sample may comprise a bone marrow sample.

[0175] The biological samples described herein may be obtained by any suitable method. Insome embodiments, the biological sample may be obtained by a blood draw. In some embodiments, the biological sample may be obtained by a biopsy. The biopsy may be a liquidAttorney Docket No.68993-715601 biopsy. The biopsy may comprise a fine needle aspiration (FNA). The biopsy may comprise a core needle biopsy. The biopsy may comprise a stereotactic biopsy. The biopsy may comprise an excisional biopsy, a bone marrow biopsy, an endometrial biopsy, an endoscopic biopsy, an incisional biopsy, a colposcopy-directed biopsy, a needle biopsy, a skin biopsy, a lymph node biopsy, or a combination thereof. The biological sample may be obtained by a biopsy. The biological sample may be completely obtained by a biopsy.

[0176] In some embodiments, the biological sample can be a cell-free biological sample. Thecell-free biological sample can comprise one or more of: a plasma sample, a serum sample, a urine sample, a saliva sample, a cerebrospinal fluid sample, a lymph sample, or any combination thereof. In some embodiments, the biological sample may comprise a cell-free biological sample. The cell-free biological sample may comprise a plasma sample. The cell- free biological sample may comprise a serum sample. The cell-free biological sample may comprise a urine sample. The biological sample may be at least 50% cell-free (e.g., at least 60%, at least 70%, at least 80%, at least 90%, at least 95%, at least 99%). The biological sample may be completely cell-free.

[0177] In some embodiments, at least a portion of the nucleic acid molecules can comprisedeoxyribonucleic acid (DNA), or ribonucleic acid (RNA), or both.

[0178] The DNA can comprise single-stranded DNA (ssDNA), double-stranded DNA(dsDNA), or a combination thereof.

[0179] The cell-free biological sample can comprise cell-free deoxyribonucleic acid (cfDNA),cell-free ribonucleic acid (cfRNA), or a combination thereof. The cell-free biological sample may comprise cfDNA. The cell-free biological sample may comprise cfRNA. The cell-free biological sample may comprise cfDNA and cfRNA.

[0180] Where values are disclosed as ranges, it may be understood that such disclosureincludes the disclosure of all possible sub-ranges within such ranges, as well as specific numerical values that fall within such ranges irrespective of whether a specific numerical value or specific sub-range is expressly stated.

[0181] The section headings used herein are for organizational purposes only and are not to beconstrued as limiting the subject matter disclosed.

[0182] Genomic language models (GLMs) are machine learning models, such as largelanguage models (LLMs), trained on biological sequence data (e.g., sequence data) such as DNA sequences, RNA sequences, and / or protein sequences to understand and predict patterns in genomes. They may apply techniques from natural language processing to biological sequence data, treating it as a form of language. A GLM may treat segments of the biologicalAttorney Docket No.68993-715601 sequence data as “words” in a language and learn relationships between these segments. Segments may be monomers (such as nucleotides or amino acids) or comprise k-mers (such as dimers, trimers or polymers). The length of a segment influences to vocabulary of a GLM, with monomers having small vocabularies. For example, a vocabulary of about 4 in the case of nucleotides (such as A,T, C and G) and a vocabulary of approximately 20 in amino acids (such as the canonical amino acids), though other words may be used such as words indicating start and stop, and / or variations in a sequence. Shorter vocabularies may result in with fewer parameters due to the model needing to associate fewer tokens (e.g., words) with subsequent layers of the model and / or other words. Associations within a model are often made through parameters in the model (such as weights), a model with fewer associations requires fewer computational resources (such as random access memory, vRAM, high bandwidth memory, CPU / GPU utilization, and / or long-term storage space). Machine learning models that deal with sequences (such as attention-based models, recurrent neural networks, or long short-term memory models) have a limit to the amount of a sequence the model can “see”, this limit is referred to as the model’s context window or context length. Beyond the limit of a context window the model may be unable to make associations with new sequence input. The context window of a model may be limited by computational resources on which the model is trained. The longer the context window the larger the number of parameters (such as weights) must be used to capture associations in the sequence. As the number of parameters grows the necessary computational resources grows. Models utilizing attention-based methods may suffer from difficulty scaling to long sequences due to the limitation of resources (such as random-access memory, vRAM, high bandwidth memory, and / or long-term memory like hard disk or solid state drive) on the system which they are trained and / or which is used for inference. As such, biological sequence data is often longer than the context windows of models being trained on it or being used to infer on it. Herein, methods and systems to reduce the computational resources necessary to train and / or for inference on long sequences such as, but not limited to, biological sequence data is provided as well as methods and systems to carry out training and / or inference of the models in a distributed and / or parallelized manner.

[0183] A GLM may be trained as a foundation model whose output may then be used to traindownstream models for specific tasks (such as classification) This approach may allow GLMs to learn complex relationships within biological sequence data or across various types of biological sequence data (such as between protein and DNA, RNA and protein, or epigenomic data and protein).Attorney Docket No.68993-715601

[0184] In some embodiments, methods and systems are provided that comprise obtainingbiological sequence data; and processing the biological sequence data using a machine learning model comprising a linearized attention mechanism produce an output. In some embodiments, methods and systems are provided that comprise scaling multi-head attention computations across genomic data at single nucleotide resolution, comprising: receiving genomic segments, each indexed by genomic source identifiers; encoding each genomic segment and its associated source identifier into a high-dimensional embedding configured for processing by a genomic language model, wherein the embedding incorporates position encoding capturing local genomic context; computing multi-head attention using hybrid dynamic kernel functions, wherein at least one of the kernel functions is dynamically adjusted per genomic locus and input embedding to precisely approximate exponential softmax computations, thereby achieving computational complexity linear with respect to input context length; and iteratively training the genomic language model on said embedded segments to predict disease conditions, wherein the dynamically adjusted kernel-based attention computations are repeatedly applied during optimization and inference. In some embodiments, the multi-head attention computations form part of a self-attention mechanism within the genomic language model, and wherein said genomic language model is trained using a masked training strategy. In some embodiments, the multi-head attention computations form part of a masked self-attention mechanism within the genomic language model, and wherein said genomic language model is trained using an autoregressive training strategy. In some embodiments, the disease conditions include onset of at least one of type 1 diabetes, celiac disease, depression, schizophrenia, and susceptibility to suicidal thoughts or actions. In some embodiments, the genomic segments include genomic regions selected from one or more of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome. In some embodiments, the genomic segments include mitochondrial DNA sequences from the human genome. In some embodiments, the input context is sharded across multiple computational devices, thereby enabling the input context length to scale proportionally with the number of devices while maintaining a near-constant sample processing rate. In some embodiments, the genomic data comprises RNA and DNA. In some embodiments, the genomic data further includes methylation status information for at least a subset of nucleotide positions.

[0185] In some embodiments, methods and systems are provided that comprise a linearizedattention mechanism (or linearized attention method). In some embodiments, a linearized attention method may approximate a multi-head full attention mechanism, such as those foundAttorney Docket No.68993-715601 in transformers. In some embodiments, approximations to multi-head full attention mechanisms may be made to tunable precision, such as through the use of a hybrid dynamic linear kernels. In some embodiments, the methods may scale linearly with context length.

[0186] In some embodiments, the methods and systems provided comprise a machine learningmodel. In some embodiments, the machine learning model comprises a neural network. In some embodiments, the neural network comprises a linear attention mechanism. The linear attention mechanism may extend the context length of the system (such as to multi-million tokens). In some embodiments, the context length may be at single nucleotide resolution. In some embodiments, the neural network system may comprise a transformer. In some embodiments, the neural network system may comprise a Large Language Model (LLM). In some embodiments, the model may take as input textual data from a user. In some embodiments, the textual data comprises written instructions that are processed by the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions. In some embodiments, the machine learning model may be trained. In some embodiments, the machine learning model may be pretrained. In some embodiments the pretraining of the machine learning model may comprise training on one or more genome segments (for example bacterial, viral and / or animal genome segments). In some embodiments, biological sequence data may comprise data derived from a human sample. The linear attention mechanism may approximate a full quadratic attention mechanism. The linear attention mechanism may have a simple and lightweight implementation. In some embodiments, a linearized attention mechanism may comprise polynomial approximation. The polynomial approximation may be used to represent a function using a polynomial. Polynomial approximation may simplify the calculation and / or use of more complex functions (such as a SoftMax function), to model nonlinear relationships, and / or improve efficiency of computational algorithms (such as various elements of a machine learning model like the attention layer). In some embodiments, the polynomial approximation is used to approximate an exponential function. In some embodiments, the polynomial approximation is used to approximate a SoftMax function. The polynomial approximation may comprise a Taylor series, Chebyshev polynomial, minmax approximation, locally sensitive hashing (LSH), Bernoulli sampling, and / or other polynomial approximations described herein. In some embodiments, the linearized attention method may comprise a kernel function. In some embodiments, the kernel function comprises the polynomial approximation. The kernel function may be used alongside or in place of traditional attention in a neural network. The kernel function may transform the Query (Q) and Key (K) vectors. The kernel function may approximate the directAttorney Docket No.68993-715601 calculation of the query and key vectors similarity using a transformed representation of Q and K. Attention weights may be calculated through the implicit mapping of the query and key vectors into a higher-dimensional feature space using the kernel function, where their similarity may be calculated in that space. The linearized attention method may be applied to biological sequence data (e.g., DNA, RNA, and / or protein data). The linearized attention method may be dynamically adjusted for genomic loci. In some embodiments, the linearized attention method may comprise a dynamic, locus-aware approximation-dependent kernel function (e.g., hybrid dynamic linear kernels).

[0187] In some embodiments, a machine learning model may comprise an attention headand / or a plurality of attention heads. In some embodiments, an attention head may comprise parameters related to a key (e.g., key parameters), query (e.g., query parameters), and / or value (e.g., value parameters). The parameters related to key, query and value may be a weight or set of weights which may be arranged as a tensor of varying dimensionalities. The parameters related to the key, query and value may be used in the generation of embeddings related to the key, query and value. The parameters related to the key, query, and value may be computed simultaneously or individually.

[0188] In some embodiments, a transformer may be used in a larger system to embed genomesegments for a second machine learning model. In some embodiments, the machine learning model may share weights across portions of the machine learning model such as across layers and / or across attention heads (such as the weights of a transformer) or a portion thereof. In some embodiments, weights are trained individually. In some embodiments, the second machine learning model comprises a neural network. In some embodiments, the second machine learning model comprises a transformer. In some embodiments, the second machine learning model comprises an LLM. In some embodiments the second machine learning model accepts one or more outputs from the first machine learning model. In some embodiments, the one or more outputs from the first machine learning model comprise a token. For example, by subdividing a large genome into smaller possibly overlapping segments and a transformer, such as those utilizing methods described herein, to generate learned (trainable) or static embeddings for each segment the second transformer incorporates an even larger genome context and may be used to incorporate whole genomes into an LLM.

[0189] Biological Sequence Data

[0190] In some embodiments, the methods and systems provided herein may comprisebiological sequence data. In some embodiments, biological sequence data may comprise data derived from sequencing deoxyribonucleic acid (DNA). In some embodiments, biologicalAttorney Docket No.68993-715601 sequence data may comprise data derived from sequencing ribonucleic acid (RNA). In some embodiments, biological sequence data may comprise data derived from sequencing mitochondrial DNA. In some embodiments, biological sequence data may comprise data derived from sequencing a peptide. In some embodiments, biological sequence data may comprise data derived from sequencing a protein. In some embodiments, biological sequence data may comprise data derived from epigenetic sequencing. In some embodiments, the biological sequence data comprises data derived from sequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 651.3 of the human genome.

[0191] In some embodiments, biological sequence data is derived from a sample. In someembodiments, the sample comprises a buccal sample, a blood sample, a saliva sample, a urine sample, a cell sample, and / or a tissue sample. DNA is extracted from a bodily sample. In some embodiments. In some embodiments, extracted DNA may be subjected to a sequencing reaction to produce the biological sequence data. In some embodiments the sequencing reaction may comprise whole genome sequencing, whole exome sequencing, whole genome bisulfite sequencings, targeted sequencing, or any combination thereof. In some embodiments proteins and / or peptides may be extracted from a sample. In some embodiments, extracted proteins may be assayed to produce biological sequence data. In some embodiments, protein sequence data may be assayed using mass spectrometry. In some embodiments, cDNA or mRNA sequences may be used to generate protein or peptide sequence data.

[0192] Model architecture

[0193] In some embodiments, a machine learning model may be a neural network. A neuralnetwork may comprise artificial neurons (neurons), units that carryout operations in both training and inference of the model utilizing the parameters of the model (such as weights and biases), and activation functions. The artificial neurons may be arranged into layers such that neurons in a layer operate on a set of input to that layer and pass their output to neurons in connected layers. Some layers may connect to themselves and operate on data oriented sequentially (such as biological sequence data). In some embodiments, the neural network may comprise a plurality of layers. The layers may comprise one or more types of layers (such as fully connected, convolutional, attention, multi-attention, transformer, recurrent, bi-directional, and / or pooling). In some embodiments, the layers may be connected residually (e.g., a skip connection. A residual connection may be a connection between two layers that allows the network to bypass one or more layers and directly connect an earlier layer to a later one, for example connection 608 in FIG.13 illustrates a residual connection.Attorney Docket No.68993-715601

[0194] In some embodiments, a machine learning model outputs an embedding. Theembedding may be output from the last layer of the model or a middle layer of the model. In some embodiments, a machine learning model may output a classification. In some embodiments, a machine learning model may output more than one output (such as both an embedding and a classification). Model Training

[0195] In some embodiments, a machine learning model may be trained. During training amodel may be initialized with a set of parameters (weights and biases) that are trainable, the parameters may be initialized using various methods such as random initialization or initialized with weights learned in another round of training. A round of training may comprise training a model over one or more epochs where each epoch is a complete pass through a training dataset. During a round of training a model may be optimized wherein the parameters of the model are iteratively changed over the epochs of training to improve a measure of the model (such as an error / loss calculation). A machine learning model may undergo more than one round of training. An example of a training workflow is depicted in FIG.1 where training data (10) is processed by a data pipeline (11) that processes the training data into a format that may be used by the model (such as tokens) , the machine learning model may then be trained (12) and aspects of the model (such as performance, distribution of information in the models latent layers, and / or attention maps) may be evaluated (13), model artifacts (14), such as trained weights, the model file and / or performance metrics (as illustrated at 15), may be collected and possibly stored for later use and / or inspection. In some embodiments, a model may be trained to predict a masked token.

[0196] In some embodiments, a pretrained model, or a portion of the pretrained model (suchas a subset of the pretrained model’s parameters (e.g., weights) may be trained and / or further trained concurrent with training a second machine learning model. In some embodiments, the embedding vectors of the pretrained machine learning model may be learned or further learned while training a second machine learning model. In some embodiments the parameters of the pretrained model may be static (not trainable) when the second machine learning model is being trained.). In some embodiments, a pretrained model may be fine-tuned. The fine tuning may comprise a pretrained model undergoing a second round of training on data tailored to train the model on a set of data tailored to a specific task. For example, a model may be trained in a first round of training on a set of data that comprises a wide range of samples (such as a wide range of DNA sequences) and subsequently trained in a second, fine-tuning round on aAttorney Docket No.68993-715601 smaller set of samples (which may or may not have been present in the first round of training) that are focused on a subset of the type of data the model was trained on in the first round.

[0197] In some embodiments, a model may be trained in a first round of training (pretrained)and later trained in a transfer learning round of training. Transfer learning may be a second round of training that focuses on data similar to that in the first round of training but not represented in the original training data.

[0198] The data in the second round of training may be related to the data in the first roundsuch that the information learned in the first round may be applicable to the data of the second round and may result in learning the second round being learned on less data or faster than the first. This may be the result of the model having learned information in the first round that is applicable to the data in the second round of training. This may be true in both fine-tuning and / or transfer learning.

[0199] In some embodiments, downstream models (e.g., a second machine learning model)may be trained based at least in part on the output of a first machine learning model (such as a pretrained model). FIG. 2 shows an example workflow for using a first machine learning model, such as a pretrained model (21). In the example of FIG. 2 an artifact obtained from process (14) in FIG. 1 may be used to generate one or more embedding vectors (22) (for example the embedding vector may be the output of the pretrained model). For example, pre- aligned long genome segments may be input to the pretrained model (21) which generated embedding vectors (22) that may be used as input to the second machine learning model (23) which may be trained using the training pipeline (24).

[0200] In some embodiments, the second machine learning model (such as a downstreammodel) may be trained to determine the probability of a genetic condition in humans. In some embodiments, a model’s (such as the pretrained model or the second machine learning model) output may comprise the probability of a condition (such as a disease) in a subject. In some embodiments, the condition may be a genetic condition. The genetic condition may be a polygenic condition or a monogenic condition. In some embodiments, a model’s output may assess the genetic component of psychotic disorders, depression and suicidal thoughts or tendencies.

[0201] In some embodiments, a model may assess one or more of the geometric and molecularproperties of the artifacts generated by the DNA or RNA through translation and transcription to proteins or RNA molecules in cells.Attorney Docket No.68993-715601

[0202] In some embodiments, a model may be trained in a first round of training using anembedding as an output and trained in later rounds of training with an additional or different output such as classification.

[0203] Model Input

[0204] In some embodiments, the model may take an input. In some embodiments, the inputmay comprise biological sequence data. In some embodiments, the input may comprise segments of a sequence. In some embodiments the segments may be aligned to a genome (such as a reference genome). In some embodiments, the segments are genomic segments. In some embodiments, the segments may be over 100,000 tokens long, over 200,000 tokens long over 300,000 tokens long, over 400,000 tokens long. over 500,000 tokens long. over 600,000 tokens long. over 700,000 tokens long. over 1,000,000 tokens long, over 2,000,000 tokens long each, over 2,500,000 tokens long , over 3,000,000 tokens long, over 4,000,000 tokens long, over 5,000,000 tokens long, over 6,000,000 tokens long, over 7,000,000 tokens long, over 8,000,000 tokens long, over 9,000,000 tokens long, over 10,000,000 tokens long, over 20,000,000 tokens long, over 30,000,000 tokens long, over 40,000,000 tokens long, over 50,000,000 tokens long, over 100,000,000 tokens long, over 200,000,000 tokens long, over 300,000,000 tokens long, over 400,000,000 tokens long, over 500,000,000 tokens long, over 600,000,000 tokens long, over 700,000,000 tokens long, over 800,000,000 tokens long, over 1,000,000,000 tokens long, over 2,000,000,000 tokens long, over 3,000,000,000 tokens long, over 4,000,000,000 tokens long, over 5,000,000,000 tokens long, over 6,000,000,000 tokens long, over 7,000,000,000 tokens long, over 8,000,000,000 tokens long, over 9,000,000,000 tokens long, over 10,000,000,000 tokens long, over 20,000,000,000 tokens long, over 30,000,000,000 tokens long, over 40,000,000,000 tokens long, over 50,000,000,000 tokens long, over 100,000,000,000 tokens long, over 200,000,000,000 tokens long, over 300,000,000,000 tokens, or any combination thereof. In some embodiments, the segments may be up to 1,000,000 tokens long, up to 2,000,000 tokens long up to 2,500,000 tokens long , up to 3,000,000 tokens long, up to 4,000,000 tokens long, up to 5,000,000 tokens long, up to 6,000,000 tokens long, up to 7,000,000 tokens long, up to 8,000,000 tokens long, up to 9,000,000 tokens long, up to 10,000,000 tokens long, up to 20,000,000 tokens long, up to 30,000,000 tokens long, up to 40,000,000 tokens long, up to 50,000,000 tokens long, up to 100,000,000 tokens long, up to 200,000,000 tokens long, up to 300,000,000 tokens long, up to 400,000,000 tokens long, up to 500,000,000 tokens long, up to 600,000,000 tokens long, up to 700,000,000 tokens long, up to 800,000,000 tokens long, up to 1,000,000,000 tokens long, up to 2,000,000,000 tokens long, up to 3,000,000,000 tokens long, up to 4,000,000,000 tokensAttorney Docket No.68993-715601 long, up to 5,000,000,000 tokens long, up to 6,000,000,000 tokens long, up to 7,000,000,000 tokens long, up to 8,000,000,000 tokens long, up to 9,000,000,000 tokens long, up to 10,000,000,000 tokens long, up to 20,000,000,000 tokens long, up to 30,000,000,000 tokens long, up to 40,000,000,000 tokens long, up to 50,000,000,000 tokens long, up to 100,000,000,000 tokens long, up to 200,000,000,000 tokens long, up to 300,000,000,000 tokens.

[0205] In some embodiments, the segments may be inputs to a machine learning model. Insome embodiments, the input may be at most 3 million tokens. In some embodiments, the input may be at most 4 million tokens. In some embodiments, the input may be at most 5 million tokens. In some embodiments, the input may be at most 6 million tokens. In some embodiments, the input may be over 100,000 tokens long, over 200,000 tokens long over 300,000 tokens long, over 400,000 tokens long. over 500,000 tokens long. over 600,000 tokens long. over 700,000 tokens long. over 1,000,000 tokens long, over 2,000,000 tokens long each, over 2,500,000 tokens long , over 3,000,000 tokens long, over 4,000,000 tokens long, over 5,000,000 tokens long, over 6,000,000 tokens long, over 7,000,000 tokens long, over 8,000,000 tokens long, over 9,000,000 tokens long, over 10,000,000 tokens long, over 20,000,000 tokens long, over 30,000,000 tokens long, over 40,000,000 tokens long, over 50,000,000 tokens long, over 100,000,000 tokens long, over 200,000,000 tokens long, over 300,000,000 tokens long, over 400,000,000 tokens long, over 500,000,000 tokens long, over 600,000,000 tokens long, over 700,000,000 tokens long, over 800,000,000 tokens long, over 1,000,000,000 tokens long, over 2,000,000,000 tokens long, over 3,000,000,000 tokens long, over 4,000,000,000 tokens long, over 5,000,000,000 tokens long, over 6,000,000,000 tokens long, over 7,000,000,000 tokens long, over 8,000,000,000 tokens long, over 9,000,000,000 tokens long, over 10,000,000,000 tokens long, over 20,000,000,000 tokens long, over 30,000,000,000 tokens long, over 40,000,000,000 tokens long, over 50,000,000,000 tokens long, over 100,000,000,000 tokens long, over 200,000,000,000 tokens long, over 300,000,000,000 tokens, or any combination thereof. In some embodiments, the input may be up to 1,000,000 tokens long, up to 2,000,000 tokens long up to 2,500,000 tokens long , up to 3,000,000 tokens long, up to 4,000,000 tokens long, up to 5,000,000 tokens long, up to 6,000,000 tokens long, up to 7,000,000 tokens long, up to 8,000,000 tokens long, up to 9,000,000 tokens long, up to 10,000,000 tokens long, up to 20,000,000 tokens long, up to 30,000,000 tokens long, up to 40,000,000 tokens long, up to 50,000,000 tokens long, up to 100,000,000 tokens long, up to 200,000,000 tokens long, up to 300,000,000 tokens long, up to 400,000,000 tokens long, up to 500,000,000 tokens long, up to 600,000,000 tokens long, up to 700,000,000 tokens long, up toAttorney Docket No.68993-715601 800,000,000 tokens long, up to 1,000,000,000 tokens long, up to 2,000,000,000 tokens long, up to 3,000,000,000 tokens long, up to 4,000,000,000 tokens long, up to 5,000,000,000 tokens long, up to 6,000,000,000 tokens long, up to 7,000,000,000 tokens long, up to 8,000,000,000 tokens long, up to 9,000,000,000 tokens long, up to 10,000,000,000 tokens long, up to 20,000,000,000 tokens long, up to 30,000,000,000 tokens long, up to 40,000,000,000 tokens long, up to 50,000,000,000 tokens long, up to 100,000,000,000 tokens long, up to 200,000,000,000 tokens long, up to 300,000,000,000 tokens.

[0206] FIG. 3 shows an example of dividing DNA or RNA using genome loci that determinethe mapping between the segments and the corresponding embedding system. In the example shown in FIG.3 models, called M5, (31 to 38) are assigned individually to different start and stop positions in the genome, each model may be the same pretrained model or may be fine- tuned the segment across multiple genomes using the training process from 24 of FIG.2. After evaluating each model on each segment the resulting embeddings, shown in this example as e1 to e8, may be stacked up into a matrix before being processed further by model.

[0207] In some embodiments, training data may comprise labels. Labels may be phenotypicdata.

[0208] In some embodiments, model training may comprise weight decay. Weight decay mayenhance generalization and to maintain numerical stability for the range based polynomial approximations.

[0209] Tokenization / Data pipeline

[0210] In some embodiments, the methods and systems provided herein may comprise the useof tokens. In some embodiments, tokens may encode the vocabulary of the model. In some, embodiments, tokens may be embeddings of segments. In some embodiments, tokens may encode biological sequence data. In some embodiments, tokens may encode biological sequences at a monomer resolution (such as a single nucleotide resolution or single animo-acid resolution). In some embodiments, tokens may encode biological sequences at a resolution greater than a monomer (such as a k-mer with a length of 2 or greater). The use of monomers and shorter k-mers produces a smaller vocabulary of tokens (such as in the single digits for nucleotides and double digits for amino acids). As k-mers grow in length so do the number of possible k-mers. For example, a vocabulary of tokens based on monomers of canonical DNA nucleotides is 4 (A,T,C,G), a vocabulary of tokens based on k-mers of 2 is 10 (AA, AT, AC, AG, TT, TC, TG, CC, CG, GG). As the k-mer length grows so does the length of the vocabulary of tokens leading to increased complexity. Different lengths of k-mers may be better suited for different types of analysis; longer k-mers may cause variations at the monomer level to becomeAttorney Docket No.68993-715601 less significant while shorter k-mers or monomer resolution may allow variations at the monomer level to be relied on more heavily. The use of monomer level resolution does not preclude the use of information present in longer k-mer vocabularies as they may be arrived at in a model itself where successive layers may learn more abstract representations of the data (for example, through hierarchical feature learning). Thus, the use of monomer level resolution may provide an advantage through a smaller vocabulary which requires fewer parameters to process resulting in a lower burden on computational resources.

[0211] In some embodiments, the methods and systems may comprise token IDs. In someembodiments token IDs may represent nucleotides (A, C, G, T), RNA, amino acids, unknown bases, spacing tokens, or any combination thereof. In some embodiments, training and / or inference of the machine learning model may comprise at least a subset of the token IDs. In some embodiments, token IDS may be masked IDs. In some embodiment, the methods and systems provided herein may comprise of segment IDs, per locus indicating inheritance source, chromosome IDs, and / or other genomic alignment markers. In some embodiments, the token IDs may be arranged as an input tensor or set of input tensors. In some embodiments, the token IDs and / or segment IDs, may be one-hot-encoded as part of the input pipeline.

[0212] In some embodiments, the token IDs may be transformed into vector embeddings. Thevector embeddings may match the model dimensionality required by the transformer block (603). In some embodiments, the methods and systems provided herein may comprise positional embeddings. In some embodiments, the positional vectors may be generated using learned convolutional kernels capturing the local sequence context 9 such as genomic or animo acid context). In some embodiments, the segment IDs may be embedded and combined with positional embeddings.

[0213] Training

[0214] In some embodiments, a model may be trained in an unsupervised manner. In someembodiments, model training may comprise token masking. In some embodiments, model training may be autoregressive. Autoregressive training may comprise next token prediction.

[0215] In some embodiments, model training may comprise optimization of the model. Insome embodiments, optimization may comprise minibatch stochastic optimization.

[0216] Model Artifacts

[0217] A model may produce one or more artifacts. Artifacts may be files or statisticsassociated with the model such as weight files, the model architecture file, saved outputs, or performance statistics.Attorney Docket No.68993-715601

[0218] Non-limiting examples of performance metrics include accuracy, sensitivity,specificity, f-measure, AUCROC, precision, recall mean absolute error, mean squared error or any suitable performance metric. In some embodiments, the methods and systems provided herein have a measured sensitivity of at least about 50%, 60%, 70%, 80%, or 90% for a given task. In some embodiments, the methods and systems provided herein have a measured specificity of at least about 50%, 60%, 70%, 80%, or 90% for a given task. In some embodiments, the methods and systems provided herein have a measured accuracy of at least about 50%, 60%, 70%, 80%, or 90% for a given task. In some embodiments, the methods and systems provided herein have a measured AUCROC of at least about 50%, 60%, 70%, 80%, or 90% for a given task. In some embodiments, the methods and systems provided herein have a measured f-measure of at least about 50%, 60%, 70%, 80%, or 90% for a given task.

[0219] Model tasks

[0220] In some embodiments, a model may output a prediction. In some embodiments, a modelmay predict a likelihood of association of a genetic variation to a phenotype. In some embodiments, the phenotype comprises at least one of an autoimmune disease, a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof. In some embodiments, the phenotype comprises early onset of a disease. In some embodiments, the phenotype comprises developmental delays. In some embodiments, the model prediction comprises, based at least on the association of the genetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.

[0221] In some embodiments, a model may output a prediction of a masked token or portionof a sequence. In some embodiments, a model may output a prediction of a disease. In some embodiments, a model may output a prediction of risk of disease. In some embodiments, a model may output a prediction of a phenotype.

[0222] In some embodiments, the method may comprises processing the output to generate apredicted effect of a mutation on a biological process. In some embodiments, a model may generate a predicted effect of a mutation on a biological process. In some embodiments, a model’s output may be processed to generate a predicted effect of a mutation on a biological process. In some embodiments, the biological process is a molecular function, a molecular property, a cellular function, a tissue function, or any combination thereof.

[0223] In some embodiments, a model may output a prediction of gene interactions. In someembodiments, a model’s output may be processed to predict gene interactions.Attorney Docket No.68993-715601

[0224] In some embodiments, the output of a model may be processed to predict an effect ofan extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

[0225] In some embodiments, a model may output a classification.

[0226] In some embodiments, a model’s output comprises a generate biological sequence. Insome embodiments, a model’s output comprises generating sequencing information.

[0227] In some embodiments, methods and systems provided herein may comprise processingthe output to design molecular complexes.

[0228] In some embodiments, methods and systems provided herein may a method thatcomprises processing the output to predict gene interactions.

[0229] In some embodiments, the model may output a sequence of tokens. In someembodiments, the sequence of tokens may be at least 650 tokens in length. In some embodiments the sequence of tokens is t a single nucleotide resolution. In some embodiments, the sequence of tokens is at least 2 million tokens in length. In some embodiments, the sequence of tokens is at least 3 million tokens in length. In some embodiments, the sequence of tokens is at least 4 million tokens in length. In some embodiments, the sequence of tokens is at least 5 million tokens in length. In some embodiments, the sequence of tokens is at least 6 million tokens in length.

[0230] Parallelization

[0231] In some embodiments, the methods and systems provided herein may comprise amethod parallelizing the training of a machine learning model. In some embodiments, the methods and systems provided herein may comprise inference of a machine learning model in utilizing parallelization. In some embodiments, parallelization of training and / or inference of amachine learning model may comprise sharding of the input context onto multiple devices(e.g., GPUs / TPUs) using a memory efficient all-reduce mechanism. In some embodiments, the methods and systems provided herein, may be used to carry out downstream tasks including classification of disease risk in humans.

[0232] In some embodiments, weights may be sharded. In some embodiments, weights maybe sharded based at least on head dimension.

[0233] In some embodiments, the methods and systems provided herein may comprise trainingand / or inference of a model across multiple devices. Such embodiments may facilitate scaling of the input context length during training and inference with the number of devices available, such as GPUs or TPUs, achieving a near constant throughput (iterations per second) even at extreme context lengths.Attorney Docket No.68993-715601

[0234] In some embodiment GPUS may communicate via NVlink. NVLink is a high-speed,direct GPU-to-GPU interconnect which allows for fast data transfer and communication between GPUs and between GPUs and CPUs and efficient scaling in multi-GPU systems, which is especially beneficial for workloads involving artificial intelligence, deep learning, and scientific computing. NVLink allows GPUs to share memory, which may reduce bottlenecks and improve overall system performance. By facilitating faster, more coherent memory access, NVLink may facilitate creation of a unified memory space, allowing applications to process larger datasets with greater speed and efficiency.

[0235] In some embodiments, TPUS may communicate via Inter-chip interconnect. (ICI).Inter-Chip Interconnect (ICI) is a high-speed communication link for Tensor Processing Units (TPUs). Designed to allow fast, scalable data exchange between TPU chips, ICI provides high bandwidth and low latency. This allows multiple TPUs to work together in clusters, known as TPU pods, which may be used for training and running large-scale machine learning models.

[0236] In some embodiments, parallelization may comprise the all-reduce algorithm.Methods for training a machine learning model

[0237] In some cases, models herein (e.g., the spectrum allocation model, locationoptimization model, channel optimization model, or modem optimization model) can include machine (ML) models. Many ML methods implemented as algorithms are suitable as approaches to perform the methods described herein. Such methods include but are not limited to supervised learning approaches, unsupervised learning approaches, semi-supervised approaches, or any combination thereof.

[0238] Machine learning algorithms may include without limitation neural networks (e.g.,artificial neural networks (ANN), multi-layer perceptrons (MLP), long short-term memory (LSTM)), support vector machines, k-nearest neighbors, Gaussian mixture model, Gaussian process, naïve Bayes, decision trees, random forest, or gradient boosting trees. Linear machine learning algorithms may include without limitation linear regression with or without regularization, logistic regression, naïve Bayes classifier, perceptron, or support vector machines (SVMs). Other machine learning algorithms for use with methods according to the present disclosure may include without limitation quadratic classifiers, k-nearest neighbor, boosting, decision trees, random forests, neural networks, pattern recognition, Bayesian networks, or Hidden Markov models. Other machine learning algorithms, including improvements or combinations of any of these, commonly used for machine learning, can also be suitable for use with the methods described herein. Any use of a machine learning algorithm in a workflow can also be suitable for use with the methods described herein. The workflowAttorney Docket No.68993-715601 can include, for example, cross-validation, nested-cross-validation, feature selection, row compression, data transformation, binning, normalization, standardization, and algorithm selection.

[0239] A machine learning algorithm can generally be trained by the following methodologyto build a machine learning model. In some cases, generated models may determine or predict spectrum to allocate to users from spectrum availability, frequency optimization, location optimization, time optimization, and the like. Input data can include, for example, spectrum parameters, regulatory parameters, compliance parameters, interference parameters, fair access parameters, license parameters, or commercial parameters. Output data can include, for example, spectrum to allocate to users.

[0240] 1. Gather a dataset for “training” and “testing” the machine learning algorithm. Thedataset can include many features, for example, features associated with channels (or frequencies), times or time periods, and locations. The training dataset is used to “train” the machine learning algorithm. The testing dataset is used to “test” the machine learning algorithm.

[0241] 2. Determine “features” for the machine learning algorithm to use for training andtesting. The accuracy of the machine learning algorithm may depend on how the features are represented. For example, feature values may be transformed using one-hot encoding, binning, standardization, or normalization. Also, not all features in the dataset may be used to train and test the machine learning algorithm. Selection of features may depend on, for example, available computing resources and time or importance of features discovered during iterative testing and training. For example, it may be discovered that features associated with spectrum parameters and regulatory parameters are predictive for dynamically allocating spectrum to users.

[0242] 3. Choose an appropriate machine learning algorithm. For example, a machine learningalgorithm described elsewhere herein may be chosen. The chosen machine learning algorithm may depend on, for example, available computing resources and time or whether the prediction is continuous or categorical in nature. The machine learning algorithm is used to build the machine learning model.

[0243] 4. Build the machine learning model. The machine learning algorithm is run on thegathered training dataset. Parameters of the machine learning algorithm may be adjusted by optimizing performance on the training dataset or via cross-validation datasets. After parameter adjustment and learning, the performance of the machine learning algorithm may be validated on a dataset of naive samples that are separate from the training dataset and testing dataset. TheAttorney Docket No.68993-715601 built machine learning model can involve feature coefficients, importance measures, or weightings assigned to individual features.

[0244] Once the machine learning model is determined as described above (“trained”), it canbe used to generate predictions for dynamically allocating spectrum to users. Computing systems

[0245] In another aspect, disclosed herein is a computer program product for implementingmethods of the present disclosure.

[0246] Referring to FIG. 17, a block diagram is shown depicting an exemplary machine thatincludes a computer system 1700 (e.g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG.17 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments.

[0247] Computer system 1700 may include one or more processors 1701, a memory 1703, anda storage 1708 that communicate with each other, and with other components, via a bus 1740. The bus 1740 may also link a display 1732, one or more input devices 1733 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 1734, one or more storage devices 1735, and various tangible storage media 1736. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 1740. For instance, the various tangible storage media 1736 can interface with the bus 1740 via storage medium interface 1726. Computer system 1700 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0248] Computer system 1700 includes one or more processor(s) 1701 (e.g., central processingunits (CPUs) or general-purpose graphics processing units (GPGPUs)) that carry out functions. Processor(s) 1701 optionally contains a cache memory unit 1702 for temporary local storage of instructions, data, or computer addresses. Processor(s) 1701 are configured to assist in execution of computer readable instructions. Computer system 1700 may provide functionality for the components depicted in FIG. 17 as a result of the processor(s) 1701 executing non- transitory, processor-executable instructions embodied in one or more tangible computer- readable storage media, such as memory 1703, storage 1708, storage devices 1735, and / or storage medium 1736. The computer-readable media may store software that implementsAttorney Docket No.68993-715601 particular embodiments, and processor(s) 1701 may execute the software. Memory 1703 may read the software from one or more other computer-readable media (such as mass storage device(s) 1735, 1736) or from one or more other sources through a suitable interface, such as network interface 1720. The software may cause processor(s) 1701 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 1703 and modifying the data structures as directed by the software.

[0249] The memory 1703 may include various components (e.g., machine readable media)including, but not limited to, a random access memory component (e.g., RAM 1704) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase-change random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 1705), and any combinations thereof. ROM 1705 may act to communicate data and instructions unidirectionally to processor(s) 1701, and RAM 1704 may act to communicate data and instructions bidirectionally with processor(s) 1701. ROM 1705 and RAM 1704 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 1706 (BIOS), including basic routines that help to transfer information between elements within computer system 1700, such as during start-up, may be stored in the memory 1703.

[0250] Fixed storage 1708 is connected bidirectionally to processor(s) 1701, optionallythrough storage control unit 1707. Fixed storage 1708 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 1708 may be used to store operating system 1709, executable(s) 1710, data 1711, applications 1712 (application programs), and the like. Storage 1708 can also include an optical disk drive, a solid-state memory device (e.g., flash-based systems), or a combination of any of the above. Information in storage 1708 may, in appropriate cases, be incorporated as virtual memory in memory 1703.

[0251] In one example, storage device(s) 1735 may be removably interfaced with computersystem 1700 (e.g., via an external port connector (not shown)) via a storage device interface 1725. Particularly, storage device(s) 1735 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1700. In one example, software may reside, completely or partially, within a machine-readable medium on storage device(s) 1735. In another example, software may reside, completely or partially, within processor(s) 1701.Attorney Docket No.68993-715601

[0252] Bus 1740 connects a wide variety of subsystems. Herein, reference to a bus mayencompass one or more digital signal lines serving a common function, where appropriate. Bus 1740 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0253] Computer system 1700 may also include an input device 1733. In one example, a userof computer system 1700 may enter commands and / or other information into computer system 1700 via input device(s) 1733. Examples of an input device(s) 1733 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some embodiments, the input device is a Kinect®, Leap Motion®, or the like. Input device(s) 1733 may be interfaced to bus 1740 via any of a variety of input interfaces 1723 (e.g., input interface 1723) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0254] In particular embodiments, when computer system 1700 is connected to network 1730,computer system 1700 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 1730. Communications to and from computer system 1700 may be sent through network interface 1720. For example, network interface 1720 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1730, and computer system 1700 may store the incoming communications in memory 1703 for processing. Computer system 1700 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 1703 and communicated to network 1730 from network interface 1720. Processor(s) 1701 may access these communication packets stored in memory 1703 for processing.Attorney Docket No.68993-715601

[0255] Examples of the network interface 1720 include, but are not limited to, a networkinterface card, a modem, and any combination thereof. Examples of a network 1730 or network segment 1730 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 1730, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0256] Information and data can be displayed through a display 1732. Examples of a display1732 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 1732 can interface to the processor(s) 1701, memory 1703, and fixed storage 1708, as well as other devices, such as input device(s) 1733, via the bus 1740. The display 1732 is linked to the bus 1740 via a video interface 1722, and transport of data between the display 1732 and the bus 1740 can be controlled via the graphics control 1721. In some embodiments, the display is a video projector. In some embodiments, the display is a head-mounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive®, Oculus Rift®, Samsung Gear VR®, Microsoft HoloLens®, Razer OSVR®, FOVE VR®, Zeiss VR One®, Avegant Glyph®, Freefly VR®headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0257] In addition to a display 1732, computer system 1700 may include one or more otherperipheral output devices 1734 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 1740 via an output interface 1724. Examples of an output interface 1724 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0258] In addition, or as an alternative, computer system 1700 may provide functionality as aresult of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this present disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to aAttorney Docket No.68993-715601 computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0259] Various illustrative logical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0260] The various illustrative logical blocks, modules, and circuits described in connectionwith the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0261] The steps of a method or algorithm described in connection with the embodimentsdisclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0262] In accordance with the description herein, suitable computing devices include, by wayof non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles.Attorney Docket No.68993-715601 Select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations.

[0263] In some embodiments, the computing device includes an operating system configuredto perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. Suitable server operating systems include, by way of non-limiting examples, FreeBSD®, OpenBSD®, NetBSD®, Linux®, Apple®Mac OS X Server®, Oracle Solaris®, Windows Server®, and Novell NetWare®. Suitable personal computer operating systems include, by way of non-limiting examples, Microsoft Windows®, Apple Mac®OS X, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia Symbian®OS, Apple®iOS, Research In Motion BlackBerry®OS, Google®Android®, Microsoft®Windows Phone®OS, Microsoft®Windows Mobile OS, Linux®, and Palm®WebOS. Suitable media streaming device operating systems include, by way of non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung®HomeSync®. Suitable video game console operating systems include, by way of non-limiting examples, Sony®PS3®, Sony®PS4®, Microsoft®Xbox 360®, Microsoft Xbox One®, Nintendo Wii®, Nintendo Wii U®, and Ouya®. Suitable virtual reality headset systems include, by way of non-limiting example, Meta Oculus®. Non-transitory computer readable storage mediums

[0264] In some embodiments, the platforms, systems, media, and methods disclosed hereininclude one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of an optionally networked computing device. In further embodiments, a computer readable storage medium is a tangible component of a computing device. In still further embodiments, a computer readable storage medium is optionally removable from a computing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media. Computer programsAttorney Docket No.68993-715601

[0265] In some embodiments, the platforms, systems, media, and methods disclosed hereininclude at least one computer program, or use of the same. A computer program includes a sequence of instructions, executable by one or more processor(s) of the computing device’s CPU, written to perform a specified task. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the present disclosure provided herein, a computer program may be written in various versions of various languages.

[0266] The functionality of the computer readable instructions may be combined or distributedas desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof. Web applications

[0267] In some embodiments, a computer program includes a web application. In light of thepresent disclosure provided herein, a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, a web application is created upon a software framework such as Microsoft®.NET or Ruby on Rails®(RoR). In some embodiments, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, and XML database systems. In further embodiments, suitable relational database systems include, by way of non-limiting examples, Microsoft®structured query language (SQL) Server, mySQL™, and Oracle®. A web application, in various embodiments, is written in one or more versions of one or more languages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or eXtensible Markup Language (XML). In some embodiments, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). InAttorney Docket No.68993-715601 some embodiments, a web application is written to some extent in a client-side scripting language such as Asynchronous Javascript and XML®(AJAX), Flash Actionscript, Javascript®, or Silverlight®. In some embodiments, a web application is written to some extent in a server-side coding language such as Active Server Pages®(ASP), ColdFusion®, Perl®, Java®, JavaServer Pages®(JSP), Hypertext Preprocessor®(PHP), Python®, Ruby®, Tcl®, Smalltalk®, WebDNA®, or Groovy®. In some embodiments, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application integrates enterprise server products such as IBM Lotus Domino®. In some embodiments, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe®Flash®, HTML 5, Apple®QuickTime®, Microsoft Silverlight®, Java®, and Unity®.

[0268] Referring to FIG. 18, in a particular embodiment, an application provision systemcomprises one or more databases 1800 accessed by a database management system (DBMS) 1810. Suitable DBMSs include Firebird®, MySQL®, NoSQL®, PostgreSQL®, SQLite®, Oracle Database®, Microsoft SQL Server®, IBM DB2®, IBM Informix®, SAP Sybase®, SAP Sybase®, Teradata®, PostGIS®, Apache®Hive, Apache®Impala, time-series databases, graph databases, key-value storage, and the like. In this embodiment, the application provision system further comprises one or more application severs 1820 (such as Java®servers, .NET®servers, PHP®servers, and the like) and one or more web servers 1830 (such as Apache®, IIS®, GWS®and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 1840. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces. In some cases, a DBMS may be a relational DBMS.

[0269] Referring to FIG. 19, in a particular embodiment, an application provision systemalternatively has a distributed, cloud-based architecture 1900 and comprises elastically load balanced, auto-scaling web server resources 1910 and application server resources 1920 as well synchronously replicated databases 1930. Mobile applications

[0270] In some embodiments, a computer program includes a mobile application provided toa mobile computing device. In some embodiments, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.Attorney Docket No.68993-715601

[0271] In view of the present disclosure provided herein, a mobile application is created bytechniques using hardware, languages, and development environments. Mobile applications are written in several languages. Suitable programming languages include, by way of non- limiting examples, C, C++, C#, Objective-C, Java®, Javascript®, Pascal®, Object Pascal®, Python™, Ruby®, VB.NET®, WML®, and XHTML / HTML with or without CSS, or combinations thereof.

[0272] Suitable mobile application development environments are available from severalsources. Commercially available development environments include, by way of non-limiting examples, AirplaySDK®, alcheMo®, Appcelerator®, Celsius®, Bedrock®, Flash Lite®, .NET Compact Framework®, Rhomobile®, and WorkLight Mobile Platform®. Other development environments are available without cost including, by way of non-limiting examples, Lazarus®, MobiFlex®, MoSync®, and Phonegap®. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone®and iPad®(iOS) SDK, Android®SDK, BlackBerry®SDK, BREW SDK, Palm®OS SDK, Symbian®SDK, webOS®SDK, and Windows®Mobile SDK.

[0273] Several commercial sources are available for distribution of mobile applicationsincluding, by way of non-limiting examples, Apple®App Store, Google®Play, Chrome®WebStore, BlackBerry®App World, App Store®for Palm devices, App Catalog®for webOS, Windows®Marketplace for Mobile, Ovi Store for Nokia®devices, Samsung®Apps, and Nintendo®DSi Shop. Standalone applications

[0274] In some embodiments, a computer program includes a standalone application, which isa program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. Standalone applications are often compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C®, COBOL®, Delphi®, Eiffel®, Java®, Lisp®, Python®, Visual Basic®, and VB .NET®, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, a computer program includes one or more executable compiled applications. Additionally, microservices related to Python®and JavaScript®may be used. Web browser plug-ins

[0275] In some embodiments, the computer program includes a web browser plug-in (e.g., webextension, etc.). In computing, a plug-in is one or more software components that add specificAttorney Docket No.68993-715601 functionality to a larger software application. Makers of software applications support plug-ins to enable third-party developers to create abilities which extend an application, to support easily adding new features, and to reduce the size of an application. When supported, plug-ins enable customizing the functionality of a software application. For example, plug-ins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display particular file types. Several web browser plug-ins may include Adobe Flash Player®, Microsoft Silverlight®, and Apple QuickTime®. In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0276] In view of the present disclosure provided herein, several plug-in frameworks areavailable that enable development of plug-ins in various programming languages, including, by way of non-limiting examples, C++, Delphi®, Java®, PHP®, Python®, and VB .NET®, or combinations thereof.

[0277] Web browsers (also called Internet browsers) are software applications, designed foruse with network-connected computing devices, for retrieving, presenting, and traversing information resources on the World Wide Web. Suitable web browsers include, by way of non- limiting examples, Microsoft Internet Explorer®, Mozilla Firefox®, Google Chrome®, Apple Safari®, Opera Software Opera®, and KDE Konqueror®. In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, mini- browsers, and wireless browsers) are designed for use on mobile computing devices including, by way of non-limiting examples, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, by way of non-limiting examples, Google Android®browser, RIM BlackBerry®Browser, Apple Safari®, Palm Blazer®, Palm WebOS®Browser, Mozilla Firefox®for mobile, Microsoft Internet Explorer Mobile®, Amazon Kindle Basic Web®, Nokia Browser®, Opera Software Opera Mobile®, and Sony PSP®browser. Software modules

[0278] In some embodiments, the platforms, systems, media, and methods disclosed hereininclude software, server, and / or database modules, or use of the same. In view of the present disclosure provided herein, software modules are created by techniques using machines, software, and languages. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or combinations thereof. In furtherAttorney Docket No.68993-715601 various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location. Databases

[0279] In some embodiments, the platforms, systems, media, and methods disclosed hereininclude one or more databases (DB), or use of the same. In view of the present disclosure provided herein, many databases are suitable for storage and retrieval data. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity- relationship model databases, associative databases, XML databases, time-series databases, graph databases, and the like. Further non-limiting examples include SQL, PostgreSQL®, MySQL®, Oracle®, DB2®, and Sybase. In some embodiments, a database is internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.

[0280] While preferred embodiments of the present disclosure have been shown and describedherein, such embodiments are provided by way of example only. It is not intended that the present disclosure be limited by the specific examples provided within the specification. While the present disclosure has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions may occur without departing from the present disclosure. Furthermore, it shall be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may beAttorney Docket No.68993-715601 employed in practicing the present disclosure. It is therefore contemplated that the present disclosure shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the present disclosure and that systems, methods and structures within the scope of these claims and their equivalents be covered thereby. Examples

[0281] While various examples of the present disclosure have been shown and describedherein, such examples are provided by way of example only. Numerous variations, changes, or substitutions may occur without departing from the present disclosure. It should be understood that various alternatives to the examples described herein may be employed.

[0282] Example 1 – M5: A WGS bacterial genome encoder at a single nucleotide tokenresolution

[0283] A linear attention mechanism is described to extend the context length of an encoderonly transformer, called M5 in this report, to a multimillion single nucleotide resolution foundation model pretrained on bacterial whole genomes. The linear attention mechanism used approximates a full quadratic attention mechanism tightly and has a simple and lightweight implementation for the use case when the key-query embedding dimensionality is low. The M5-small model is entirely trained and tested on one A100 GPU with 40gb of memory up to 196K nucleotides during training and 2M nucleotides during testing. We test the performance of the M5-small model and record notable improvements in performance as whole genome bacterial sequence lengths are increased as well as demonstrating the stability of the full multi- head attention approximation used as sequence length is increased.

[0284] Introduction

[0285] LLMs that incorporate DNA or RNA sequences need to successfully combine twofactors: the extreme length of DNA sequences and the simplicity of the 4 main tokens involved. The most common way to combine these two factors is to use tokens representing k-mers to combine multiple nucleotides into a single token. Alternative approaches use single nucleotide resolution, i.e., k=1, as is the case with M5. The approach reported on here includes adjusting the architecture of the LL model to facilitate long sequences, directly taking advantages of the small token set and letting the model itself learn and optimize position encodings.

[0286] DNA and RNA segments, phased DNA and DNA in whole genomes represent achallenge for LLMs due to LLMs limitations on input context length and due to sub-optimal recall observed in many non-quadratic attention approaches, see [7]. Recently, benchmarks have been introduced for comparing the performance of the various LLMs and models appliedAttorney Docket No.68993-715601 to DNA and RNA sequences, see

[0010] , [8], reflecting and providing an overview of some of the current approaches of measuring the performance of LLMs when specialized to DNA / RNA and applied to ultra long sequences.

[0287] For bacteria the genome segments are relatively short, ranging up to ap-proximately 14million nucleotides with most bacterial genomes below 5 million nucleotides, see [6] and

[0011] . M5 is an LLM designed to work with genomes in this size range at a single nucleotide resolution. It explores using a high number of attention heads, small key-query dimensionality, a family of approximations to the exponential function to linearize the attention mechanism and learned neighborhood-based position embeddings. We hypothesize that this is an efficient way to work with DNA and RNA sequences at a single nucleotide resolution.

[0288] For testing we use M5-small, a linear encoder, progressively pretrained using maskingof WGS segments of up to 196,608 nucleotides sampled from bacterial genomes. We report performance of testing the M5-small network on input segments of up to 2 million nucleotides using a single A100-40gb GPU both for the training and testing.

[0289] Background

[0290] Multiple papers reference using polynomials and kernel methods to linearize thesoftmax operators used in multi-head attention, BASED [2] uses a 2nd order Taylor series to approximate the exponential function; [9] by Angelos Katharopoulos et al. discusses polynomial linearization as well as using other kernel functions; Performers [4] approximate attention using a method called “Fast Attention Via positive Orthogonal Random features approach” obtaining linear space and time complexity; symmetric and asymmetric kernels are discussed in

[0016] by Yao Hung Hubert Tsai et al. The ReZero paper [3] uses zero initialized gated residual networks to remove reliance on layer normalization and improve training of extremely deep neural networks. The

[0019] paper by Michael Zhang et al. discusses using shallow trained feed-forward networks to match softmax attention weights resulting in linear attention networks that preserve the spikiness and monotonic properties of the softmax operator.

[0291] In the paper [1] Josh Alman and Zhao Song provide theoretical arguments and proofs,including using the prerequisite of connecting the key-query dimension dk to the input length N, i.e., number of nucleotides. Inspired by the paper, and as a future direction of research, we consider the relationship dk ∼ log(N) in our full M5 report.

[0292] Long context length models in biology that use single nucleotide resolution includeEVO

[0013] at 131Kb at the time of the publication and HyenaDNA at 1 million nucleotides,

[0012] . HyenaDNA swaps out attention for the ”Hyena operator” and EVO uses theAttorney Docket No.68993-715601 ”StripedHyena” architecture. Both of the models are pre-trained using the next token prediction strategy, i.e., autoregressively.

[0293] Methods

[0294] Linear attention used by M5

[0295] We outline the linear attention mechanism used by the M5 transformer encoder.Starting with a given key-query embedding dimension d = dk we approximate the value exp(q◦ k + m), where ◦ denotes the dot product of two vectors q, k ∈ Rdk and m is a number possiblydepending on the q and k vector domains, using nonlinear transformations θm, ϕ such that

[0296] exp(q ◦ k + m) ≈ θm (q) ◦ ϕ(k).

[0297] This approximation of exp(q ◦ k + m) can be made as exact as needed at the expense ofadditional compute on a given bounded interval using the fact that polynomials may be used to uniformly approximate the exponential function on a given bounded interval. The motivation of including the value m in the formula is to direct the theoretical input q ◦ k + m into a bounded from below interval, with almost all of the values falling into a given bounded region where the approximation holds well.

[0298] First let’s start with any given polynomial approximation,

[0299] exp(^^) ≈ ∑^^^^^^=0^^^^^^

[0300] If x = q◦k+m then rewriting this polynomial approximation is straightforward using thebinomial formula ^^ ^^^^

[0302] Using ^^^^^^^^^^^^) we obtain the formulation^^

[0303] where we are using dot product to combine the direct sums in (2). Combining (1) and(2) we obtain the expression: ^^^^^^ ^^

[0311] andAttorney Docket No.68993-715601

[0312] ^^(^^) = 1 ⊕^^1 ^^^^1 ⊕^^1,^^2 ^^^^1^^^^2 ⊕ ⋯⊕^^1,…,^^^^ ^^^^1 ⋯ ^^^^^^ (7)

[0313] Instead of starting with a polynomial approximation of exp(x) we could equivalentlystart with an approximation of exp(^^ / √^^^^) since the softmax is normally scaled using√^^^^e.g., in the groundbreaking “Attention Is All You Need'” paper.

[0314] So we assume, in the below, that we are using a polynomial approximation:

[0315] exp(^^ / √^^^^) ≈ ∑ ^^^=^^^0 ^^^^^^(8)

[0316] The difficulty with θm and ϕ is that these map Rd to R1+d+d2+···+dn, so the dimensionalitygrows fast with n and d and the maps quickly become infeasible in any practical setting for large values of n or d. On the one hand, dk can be reduced by mapping linearly the query and key projections to a smaller dimensional space, as in [2]. On the other hand, we can try to let n be small, e.g., 3, and the key-query dimension d = dk be small also, say less than 10 and use a relatively high number of “heads” in a multi-head attention encoder. For a very small vocabulary, such as DNA at nucleotide resolution, we evaluate if this is a reasonable approach or not.

[0317] 3.1.1 Example

[0318] Empirically, if we are using n = 3, dk = 4 and

[0319] (a0, a1, a2, a3) = (1.0017636, 0.49488056, 0.12190779, 0.02954964)

[0320] we obtain an excellent approximation to exp(^^ / √^^^^) on the interval [−1, 2], see FIG.4. The values of (a0, a1, a2, a3) are chosen so that the integral of the squared difference between the polynomial and the scaled exponential function are minimized on the interval [0, 2], giving a total area under the squared difference of 9.5E-71over the interval.

[0321] Assuming we have a triad composed of query, key and value projections^^ ∈ ^^^^×^^^^,^^ ∈ ^^^^×^^^^ and ^^ ∈ ^^^^×^^^^we can, using the above, approximate a single row, vi′ ,in theattention operator from

[0017] given as:

[0322] Attention(^^,^^, ^^) = softmax^^^^^^

[0323] Using the formula^^^^

[0326] here we are using the vector to matrix convention Rn = to translate from dotproducts to matrix operations. Note that approximation (10) can be made arbitrarily precise onAttorney Docket No.68993-715601 a given closed interval. The softmax function is invariant under constant shifts in the inputlogits, ^^ ^^ ^^ ^^^^^^ .

[0327] Since the softmax normalizes each row in the hypothetical attention matrix then wecan pick m = mi to be given by different values for each input position i in (10).

[0328] Here we let

[0330] this formula guarantees that mi ≥ qi o kj for all j E {1, ..., N} and hence this allows usto assume that x = q o k + m in approximation (8) is always non-negative. Note that we never calculate qi o kj directly for all i and j since that would require N2compute. We can however, use the inequality −mi ≤ qi o kj ≤ mi and monitor mi during training to map out if our assumptions, regarding the interval for where approximation (8) holds, are well-founded. To summarize, we have defined a family of transformations ^^^^^^^^^^+^^−^^

[0331] ^^^^^^ :^^ ^^ → ^^ ^^−^^ ^^^^^^ ^^ ∈ {1, … ,^^} and d= dk (13)

[0332] where N is the encoder context length such that attention mechanism (9) may belinearized up to arbitrary precision using

[0334] and where approximation (14), for each i, is no worse than ^i ≥ 0 with^^

[0336] Compute required by the M5 attention mechanism

[0337] In this section we approximate the exponential function with a polynomial of degree nand evaluate the compute required by the attention mechanism (10) over context length N. Calculating (10) for all i = 1,... , N requires compute on the order of

[0338] ∼ ^^^^^^^^^^^^ (16)

[0339] while computing quadratic attention requires compute on the order of

[0340] (dk + dv)N2.

[0341] Assuming that dk << dv we therefore obtain a speedup ratio of

[0342] [(^^ )2 ^^ + ^^^^ ^^(17)

[0343] If we let dk = dmodel / h where h is the number of heads, then the compute speedup orslowdown translates toAttorney Docket No.68993-715601

[0344] ∼ℎ^^^^ ^^ ^^ model(18)

[0345] Accordingly, we focus on models with a large number of heads, moderate modeldimension and small values for dk such as 2, 3 or 4. Note that the relationship dk = dmodel / h is not a requirement for attention to work and we may deviate from it and rely on (17) instead.

[0346] Position embeddings

[0347] Training is performed on continuous slices of bacterial genomes with a repeated fixedtoken between individual chromosomes or plasmids. A learned position embedding is added to each nucleotide representing the genome segment, numbered from 1 to 4, the idea being to teach the network that the bacterial genome is made up of individual separated chromosomes and plasmids. We use a multi-layer learned convolutional network with max pool activations to map the neighborhood of each nucleotide to a position embedding that is then added to the nucleotide embedding (A, C, G, T, unknown, separator or mask). This is because we think of a position in the genome as being almost uniquely determined by its neighboring DNA sequence, hence we train a network to learn the position of a center nucleotide from its neighborhood. We set the neighborhood length to be 1024 nucleotides. The artifacts (RNA and proteins) produced by the bacterial genome during transcription and translation interact in the cell environment so interaction between elements should to a large degree be determined by the localized DNA sequence of each element instead of the absolute or relative position of tokens within the sequence.

[0348] For a future addition to this report, we include the ability to switch on rotary positionalembedding in the M5 model setting, see

[0015] for a discussion of the relative positional rotary encoding. The rotary positional embeddings are for the purpose of the current first version of this report switched off.

[0349] Network training

[0350] M5 is a transformer encoder only. It is trained using masking, see [5], 12% of knowntokens are masked during training, 3% are included as-is into the prediction. In addition to this a large continuous segment of random length up to a maximum of 4096 nucleotides, capped at 15% of the context length, is hidden during training by replacing it with the mask token. The context length is increased gradually starting with context length of only 1024 and then doubled between sessions. Each session initializes the Adam learning rate, adjusts it to a new batch size, when needed, and uses a cosine learning schedule with warm-up. Weight-decay is used only for the key-query transformation weight matrices and the gradients are clipped individually at 0.05. The key-query attention matrices (weights) are regulated using a small l2 weight decayAttorney Docket No.68993-715601 factor in some of the training runs in order to guarantee that approximation (8) holds sufficiently well.

[0351] M5-small training data

[0352] We collected bacterial whole genome data from GTDB (Genome TaxonomyDataBase), see

[0014] , release 09-RS220 from April 24th, 2024. After processing the genomes downloaded and after scanning for keywords like “complete genome” and “complete sequence” in a description field we obtained approximately 7,000 whole genomes that we randomly divided into 80%-10%-10% train, eval and test split. The total number of genomes downloaded from GTDB is much larger than the ∼ 7,000 satisfying our criteria. For the purpose of evaluating the M5-small model architecture and evaluating its performance we used only the complete genomes as indicated by the aforementioned keywords inserted by the submitters of the data.

[0353] Network architecture

[0354] The original encoder found in [5] and

[0017] is our starting point. As explained in section3.1 the attention mechanism is linearized by approximating the exponential function with a family of asymmetric kernel function pairs - one function per input position / token. The position embeddings come from a simultaneously learned CNN network and the top classification layer is similarly a multi-layer CNN network that is allowed to see a range of neighboring tokens before making a prediction. Layer normalization is applied after positional encodings have been added and layer normalization is used as the last step of each multi-head attention repeat. Additional skip connections are used and all skip connections are initialized so that the residual signal starts out as zero, this is achieved by initializing one sub-layer within each residual block with zeros only and is inspired by the logic found here [3] used to reduce the reliance on layer normalization in both cases.

[0355] FIG. 9 shows an example of computations involving a polynomial approximation(degree 3 shown in this example) of the normalized exponential function. In this example, the approximation is dynamically adjusted for each genomic locus, this is done to approximation of the softmax function, as shows using formulas (9) through (14).

[0356] Given a selected key-query dimension dk and a polynomial degree, the normalizedexponential function is approximated according to formula (8). For example, a dimension dk = 4 and polynomial degree 3 yield coefficients a = 1.0017636, b = 0.49488056, c = 0.12190779 and d = 0.02954964, corresponding respectively to polynomial terms a0, a1, a2and a3.

[0357] These coefficients, or others depending on specific implementations, are employed ina computational function labeled (202). This function takes an input tensor mbhs, labeled asAttorney Docket No.68993-715601 (201), indexed by batch (b), attention head (h) and sequence position or locus (s). It computes and returns four tensors (am, bm, cm, dm) via element-wise operations and tensor broadcasting.

[0358] Variations in polynomial degree, approximation coefficients, and normalizationconstants, as discussed in reference, can be accommodated, leading to corresponding adjustments in the computations of function (202).

[0359] FIG. 10 illustrates computations used to implement the polynomial-dependent kernelfunction detailed in formula (7), building upon the polynomial approximations described in FIG.9.

[0360] The computational function (302) takes as input a tensor of key projections labeled(301), explicitly indexed by sample (b, for batch), attention head (h), sequence position or locus (s) and key-query embedding dimensionality dkwith indices i, j, and k explicitly shown to index the key-query dimension. The output from function (302) is a concatenation of four tensors: a constant tensor filled with ones at each position (303), the original key projections themselves (304, same as 301), and two tensors (305 and 306) computed here via Einstein summation notation, representing quadratic and cubic interactions among the key embeddings. This combined output structure supports the hybrid kernel-based computations implementing the polynomial-approximated attention methods described in this example.

[0361] FIG. 11 illustrates computations used to implement intermediate steps involved in thedynamic, locus-aware approximation-dependent kernel function described by formula (6). The illustrated computations explicitly incorporate the locus-aware logic detailed by formula (12) and the shift-variable method described in this example.

[0362] The input to the function, labeled (402), comprises query projections along with themaximum 2-norm over the embedding dimension across all loci of the key projections. This second input tensor is thus indexed exclusively by batch (b) and attention head (h). The pseudocode begins by computing intermediate variables (403), (404) and (405). Step (403) calculates the 2-norm over the embedding axis (last dimension) of the query projections. Step (404) multiplies this result elementwise by the max 2-norm over the key embeddings, using Einstein summation notation, to implement formula (12). Maintaining small magnitudes in these computed tensors, achieved by weight decay during training, preserves numerical stability and accuracy of the polynomial approximation, as depicted in FIG.7 in the paper.

[0363] Step (405) employs the “coeff” function illustrated on FIG. 9 to calculate tensors am,bm, cm and dm shifted by a small value (e.g., -1). Finally, the four tensors concatenated into the output of function (402) are explicitly computed in steps (406) through (409), exemplifying aAttorney Docket No.68993-715601 polynomial of degree 3 used to approximate the normalized exponential function following the methodology previously outlined in FIG.9.

[0364] FIG. 12 illustrates pseudocode for the forward pass computation of multi-headattention using hybrid dynamic linear kernels, specifically exemplified with a polynomial approximation of degree 3. The steps as well as the steps shown on FIG.9, FIG.10 and FIG. 11 can be executed in various order and transformed further using mathematical principles as well as further approximations, evident by the lack of functional dependencies between certain steps. These and other methods presented here may also require further adjustments to ensure numerical stability depending on variable types used.

[0365] The input (501) into the multi-head attention module is a tensor of embeddings for eachsample across loci, shown having dimensionality indexed by batch ^^, locus position s, and embedding coordinate index e. The model also contains multiple sets of weights, the 4 weight variables used in the attention module would include k_param, q_param, v_param and z_param as indicated by (502). The weight tensors may be organized in many different ways but for clarity we can assume that the shape or dimensionality of the k_param and q_param weight tensors is (nr-of-heads, model-dimensionality, key-query-embedding dimensionality), the shape of v_param is (nr-of-heads, model-dimensionality, value-embedding dimensionality) and the shape of the z_param tensor is (nr-of-heads, value-embedding dimensionality, model- dimensionality). These weight tensors are used to create the key, query and value embeddings or projections, for example the key embeddings (k) in step (504) would be created as k = einsum("bse,hed->bhsd", input, k_param) using Einstein summation notation and similarly for the query and value embeddings. Alternatively, the weights can be sharded using the head dimension depending on memory requirements and memory pressure.

[0366] The MHA (multi-head-attention) computations shown as (503) is used to compute theattention formula (10) illustrated in steps (504) to (512). Each step is detailed using pseudocode. Step (504) indicates how key, query, and value embeddings can be computed simultaneously or individually. Step (505) indicates that one must compute the 2-norm over the last dimension of the key embeddings and then take the max of all such values over the loci (also called sequence or context) axis. This kind of computation can be extended across multiple devices using a lightweight all-reduce algorithm as shown by FIG.14 if access to all the loci axis is not on the same device. Step (506) references the computations detailed on FIG. 10 and step (507) references the computations detailed on FIG.11. Step (508) requires us to sum the computations from (506) over the loci axis. Again, if we do not have access to all the loci on the device an all-reduce algorithm is used to aggregate the values across all the requiredAttorney Docket No.68993-715601 devices as explained on FIG. 14. Calculations (509) and (510) are illustrated using Einstein summation notation, in addition a small epsilon value, such as 1e-8, is introduced to avoid division by zero when computing the normalization factor in (510). Step (511) shows the required normalization of the computations from step (509), i.e., the computations in step (509) are divided elementwise by the “N” tensor computed in (510) in step (511). In some cases, the shape of tensor “N” computed in step (510) may require explicit broadcasting or reshaping before division. Finally, in step (512) the multi-head attention is aggregated and transformed, using Einstein summation notation, to input-compatible model embeddings using the learnable z_param tensor and returned as a materialization of the computations presented as formula (10), adjusted for the multi-head setting. This way we obtain a strict approximation of classical full quadratic multi-head attention, sidestepping the quadratic compute complexity of full attention.

[0367] Further modifications, such as additional activation functions or layers, may be applied.

[0368] FIG. 13 illustrates a transformer-based Genomic Language Model (GLM) setup. Thefigure identifies the main elements of an end-to-end GLM network embodiment in which the multi-head attention, described using FIG.12, is used as part of a genomic language model.

[0369] Input (601) comprises token IDs, representing nucleotides (A, C, G, T), RNA, aminoacids, unknown bases, and spacing tokens. During training, a subset of the token IDs may be mask IDs. The input also comprises segment IDs, per locus indicating inheritance source, chromosome IDs, or other genomic alignment markers. The input tensors are then processed by step (602) in the figure. The token IDs and segment IDs, in some embodiments, are one- hot-encoded as part of the input pipeline.

[0370] At step (602), token IDs (one-hot-encoded if not already) are transformed into vectorembeddings matching the model dimensionality required by the transformer block (603). Positional embeddings, typical for transformer architectures, may be generated using learned convolutional kernels capturing the local genomic context. Segment IDs are similarly embedded and combined with positional embeddings. Layer normalization may subsequently be applied to these aggregated embeddings.

[0371] Processed embeddings serve as input to transformer block (603), which may beencoder-only or decoder-only. Within each transformer block, the efficient multi-head attention algorithm detailed on FIG.9, FIG.10, FIG.11 and FIG.12 and shown as (606) is applied. The signal may then be added to the transformer block input signal again, forming a residual connection (608) and in some embodiments normalized again before processing by a fully connected feed forward network (607) and a second residual connection (609) in additionAttorney Docket No.68993-715601 to further normalization. The output signal is then processed again by a transformer block multiple times as indicated by (605). After being repeatedly processed by the transformer block, a classification layer (604) processes some of the loci, after considering the output from neighboring loci in some embodiments, and during pretraining is used to predict the masked or randomized input tokens and during fine-tuning the classification head may be altered or further structured and aggregated, see (610), in order to predict disease risk or phenotypes associated with the network’s input.

[0372] The transformer, convolutional and fully connected elements of the network shown onFIG. 13 may in some embodiments apply further normalization, dropout, activations, memorization layers and multiple other methods that are commonly associated with large language models and genomic language models.

[0373] The model may implement look-ahead or other masking strategies with attention (606),typically utilizing autoregressive next-token prediction during pre-training.

[0374] FIG. 14 illustrates input context (loci) sharding onto multiple computational devices,such as GPUs or TPUs. The task may require the definition of one or more mesh axes used to shard tensor data onto available or designated devices. For the multi-head attention mechanism described on FIG.9, FIG.10, FIG.11 and FIG.12 most of the steps are specific to a given locus and can therefore be performed on the device where the locus (sequence position) was placed, when data is sharded using the input sequence position. For example, on FIG.10 step (305) involves the computations “einsum("bhsi,bhsj->bhsij", k, k)” and since “s” is the sequence locus (3rddimension), and is unchanged between input and output, this computation can be performed on each device individually, not requiring any cross-device communication. On the other hand, there are 3 computational steps illustrated on FIG.12 that require additional cross-device communication when the input context is sharded on multiple devices. These are steps (505), (508) and (509). In each case the computations are by default performed on only individual devices initially. Therefore, before the computations can be used in the following steps the computations have to be synchronized and aggregated across devices. The all-reduce algorithm may be used to sync and aggregate the computations, in the case of step (505) a “pmax” all-reduce operation is added to find and synchronize the maximum value across all the devices, in steps (508) and (509) a simple “psum” all-reduce operation is added after the local computations, in each case, to sum the results across all devices before the computations can be carried to the next steps.

[0375] The all-reduce computations are illustrated explicitly in FIG. 14. (701) illustrates thesharding of the input context onto multiple devices, using two devices to exemplify theAttorney Docket No.68993-715601 sharding (704) and (705), first a local computation operation such as sum, mean or max is performed across each of the shards associated with each device, for example half of the loci in this exemplified case. Next the results of the local computations are communicated (702) to the next device, usually in a ring like fashion, and combined with that device’s local result using the appropriate operation. Once the all-reduce process ends, each device, (706) representing (704) in a new state and (707) representing (705) in a new state, holds the same result aggregated (e.g., by sum, max or mean) across the complete context axis and computations can continue populating the sharded axis with new or updated results (703).

[0376] Other methods may be applied to implement the sharding when working with theconvolutions, classification heads and the fully connected networks shown as part of the transformer network on FIG. 13 in addition to applying the all-reduce methods explained above. Sharding the input context, for example genomic position, may not prevent the use of other axis as well for sharding such as sharding by the data dimension (into multiple smaller batches) or the model dimension (sharding the weight parameters) or using a combination of the axes for sharding when training the model or applying inference using multiple devices.

[0377] FIG. 15 illustrates how the genomic language model (GLM, FIG. 13) receives inputas concatenated genomic DNA or RNA segments, along with corresponding segment index arrays. This is illustrated with a small example on FIG.15. Referencing the figure, (801) and (802) exemplify individual maternal and paternal homolog DNA segments from a human proband WGS or WES sample. The segments shown are assumed to be long and may represent millions of nucleotides each. Segments (801) and (802) may be encoded into numerical arrays using numbers such as 0,1,2 and 3 representing the A,C,G and T nucleotides respectively and concatenated into an array shown as (805). Additional tokens, such as '4', represent spacing ('S') between segments. In this case, the first sequence is assigned the index id “1” as indicated by the segment index id array shown as (804) and the second sequence is assigned the index “2” as indicated by the array values also. Spacing receives the index id 0 in the figure. Depending on embodiments, the same index can be used for the segments representing the start positions (803) instead of the distinct indices 1 and 2 for the individual homologs at the start position. To assign meaning to the indexes, it may be useful in some embodiments to synchronize the indexes across samples, during training and inference, by, for example, reusing index ids representing segments that start at the same position and cover similar genomic regions as shown in the figure by the genomic start position (803).

[0378] After batching, encoded DNA arrays (805) and segment index arrays (804) may be one-hot encoded and used as input to the GLM detailed in FIG.13. During fine-tuning of the modelAttorney Docket No.68993-715601 each sample may be associated with a label representing, for example, the onset of a disease or condition or be a control sample and the model trained to classify the outcome based on the genomic segments input into the model.

[0379] FIG. 16 illustrates a fine-tuning pipeline for genomic language models using specificsets of genomic segments. The training data used (901) may comprise one or more maternal and / or paternal homolog genome segments from the 1q21.1, 1q21.2 regions on chromosome 1, segments from the HLA region on chromosome 651.3 as well as extended regions and multiple other regions of interest associated with conditions including autoimmune diseases and neurodevelopmental disorders. The genome segments may be indexed according to the schema described in the text associated with FIG. 15. Segments may also include mitochondrial DNA for each sample. The aggregated length of the segments considered for each sample may be a considerable fraction of the probands genome, up to hundreds of megabases (MB) of nucleotides long, spanning both coding and non-coding regions of the genome. During training or fine-tuning (904) of the model the model is initialized with a set of weight parameters (903) either using pseudo-random initialization and / or pretrained weights. During training (904) the model learns to predict conditional risks, such as early onset of diseases, of one or more of the conditions associated with the input genome segments. These conditions may include type 1 diabetes, celiac disease, or developmental delays in newborns. Performance statistics are collected (906), and trained parameters are stored for validation, further training, or deployment into production.

[0380] Results

[0381] Small M5 model

[0382] The M5-small encoder is trained starting with a context length of 1,024 and then thecontext length is increased (normally doubled) between sessions until it reaches 196,608 nucleotides during training. For the initial small setup, we use a key-query dimension of 4, dk = 4 and linearize the attention mechanism with a polynomial of degree 3, see 3.1 for the details. We train and test the model using only one A10040gb GPU. For testing we use bacterial whole genome sequence segments of length up to 2,000,000 nucleotides. Everything, both testing and training is done at a single nucleotide resolution using only the 4 tokens A,C,G,T plus 3 meta- tokens (unknown, masking and a token representing both the separation of chromosomes and plasmids). Batch size is reduced from 16 to 1 due to memory restrictions as context length is increased and the learning rate schedule is modified as batch size is decreased or increased.

[0383] Improved predictions as context length increasesAttorney Docket No.68993-715601

[0384] FIG. 5 demonstrates, using cross-entropy measure, that there is a clear benefit oftraining on long sequences when it comes to inferencing about the bacterial genome. The figure shows the mean sample cross-entropy, on the y-axis, obtained using models trained on various sequence lengths, as detailed below, using the 10% holdout test set. Cross-entropy is computed using the masked nucleotides, and measures how confidently the model on average is able to reconstruct the masked nucleotides, with zero representing complete confidence. As explained in section 3.412% of the nucleotides are masked, 3% are unchanged and included in the cross- entropy test compute. Additionally, during training a uniform random length continuous segment of up to a maximum of 15% of the context length, capped at 4,096 nucleotides, is completely cleared of all nucleotide information. Plot A shows results from multiple models trained progressively on whole genome bacterial segments ranging from 1,024 to 196,608 nucleotides and then tested on segments equaling the longest training segment. Cross-entropy is reported for the model trained up to the given testing context input length shown on the x- axis. It is apparent from plot A how the performance of our models improves steadily when trained on longer segments. Plot B shows the cross-entropy of a single model that is trained on nucleotide segments of length 1,024 only and then applied to longer DNA segments during the testing shown. Interestingly, when the context length is increased during inference the performance of this model does not appear to improve much if any. Plot C shows the sample cross-entropy measured using a single model trained on, up to, bacterial whole genome sequence lengths of 196,608 nucleotides and then tested on various shorter and longer input context lengths shown on the x-axis. Even when this model is tested on the smallest input context length of 1,024 it by far outperforms the model trained explicitly only on that short sequence length. Combined, plots A, B and C show that training a model on long bacterial nucleotide sequences has clear benefits for predicting, using the M5-small model setup, when inferencing about shorter bacterial sequence segments as well as being beneficial for inferencing on longer sequences.

[0385] FIG. 6 demonstrates, using prediction accuracy of masked nucleotides and shortregions, the benefit of training on long sequences. The figure shows the accuracy of randomly masked (12% masked + 3% original) nucleotide predictions using the weights from models trained on bacterial segments of up to 196,608 bases. The accuracy is measured on the 10% holdout set of the bacterial genomes collected as before. Plot A shows the single nucleotide model prediction accuracy achieved by models trained progressively up to and tested at the shown input context length (x-axis), i.e., 1,024 - 196,608 nucleotides. Plot B shows the accuracy (SNPAcc) for different input context length used during testing using a single modelAttorney Docket No.68993-715601 trained on sequences of up to length 196,608 nucleotides. Plot C shows the accuracy (SNPAcc) sampled on a randomly masked region of uniform random size between 2 and 15 nucleotides using the same encoder trained on sequences of up to length 196,608 nucleotides and tested for various sequence lengths as shown.

[0386] Validity of the linear approximation to the softmax

[0387] Here we look at the distribution of “m” values that are observed by the M5-smallmodels. As demonstrated by formula (15), for our estimate of the approximation of the exponential function used to linearize the softmax in the M5 transformer encoder, we would like our approximation (8) to hold in the interval [0, 2mi]. Inspired by eyeballing the approximations used in practice if we are using polynomials of odd degree such as the one seen in FIG.4 for dk = 4 we can shift the mi values by some small constant δ, say -1 to increase the range where we have a solid approximation.

[0388] FIG. 7 shows the distribution of the values mi − 1 computed for all the attention heads,input tokens, and repeats for the 10% test set and the M5-small encoder trained and tested using context length of 16,384 using formula (12). For clarity we show the exponential function and the approximation used, demonstrating the tight approximation (8) used to approximate the softmax on all applicable values in the test set. The distribution is impacted by the l2 weight decay factor used to control the growth of the key and query weights (weight matrices), in this case it is set to 10E-5 for the key-query transformation matrices only. Empirically, we observe that dist(m − δ) is a one-sided long tail distribution. The top of the distribution (40%) is concentrated in the first histogram bucket (-1.00 to -0.90). Notably, the distribution implies a spiky dot product attention with 0.12% of the mi entries larger than 2 representing over 7 standard deviations from the mean, with mean for mi at 0.2223, and standard deviation of the distribution at 0.2558.

[0389] FIG.8 shows two distributions of the values mi − 1 computed for all the attention heads,input tokens, and repeats for the model trained on sequences up to 196,608 nucleotides. One of the distributions is for the testing set sampled for segments of length N=1,024 and the other one is for the testing set sampled for segments of length N=2,000,000. Apparently, these two extremes in the context length only shift the distributions in very minor ways, even though the computations of the mi values have to factor in the complete context input length as is evident by the use of the “max” function over all indices in the input context from formula (12).

[0390] Example 2 – Linear attention mechanism for extending context length to multi-million DNA or RNA tokens at single nucleotide resolutionAttorney Docket No.68993-715601

[0391] A neural network system is described, utilizing a linear attention mechanism to extendthe context length of the system to multi-million DNA or RNA tokens at single nucleotide resolution. The neural network system is also classified in the contemporary industry and science literature as a transformer or an LLM (Large Language Model). To distinguish it from other technologies of the same nature we refer to it herein for short as M5. It may be pretrained on one or more bacterial, viral or human genome segments. The linear attention mechanism used approximates a full quadratic attention mechanism tightly and has a simple and lightweight implementation.

[0392] In some embodiments, the M5 transformer is used as nodes in a larger system to embedgenome segments for a second transformer model. In some embodiments the M5 nodes share the same weights and in other embodiments the weights are not shared and are trained or finetuned individually for each node. The second transformer is an LLM that accepts as tokens one or more outputs from each of the M5 nodes. By subdividing a large genome into smaller possibly overlapping segments and applying the M5 transformer to generate learned (trainable) or static embeddings for each segment the second transformer incorporates an even larger genome context and may be used to incorporate whole genomes into an LLM.

[0393] FIG. 1 exemplifies the training of the M5 network. It demonstrates a training pipelinewhere the genome segments (10) and in some embodiments phenotypical data (labels) are used to train the model using a data pipeline (11) that in some embodiments follows a token masking and / or a next token autoregressive training technique. The model is trained in step (12) using minibatch stochastic optimization and after each training epoch or session the model is evaluated (13) and artifacts (14) set aside, including model weights (15), based on performance metrics collected (15).

[0394] FIG. 2 shows how a pretrained or randomly initialized model (21) artifact obtainedfrom process (14) may be used to generate one or more embedding vectors (22) for pre-aligned long genome segments. In some embodiments the pre-aligned genome segments are over 100,000 nucleotides long and in other embodiments the pre-aligned genome segments are over 2,000,000 nucleotides long each. The embedding vectors may be learned or static depending on whether the training pipeline (24) is configured to further modify the weights of one or more of the M5 (21) models concurrently with modifying the weights of the second system (23). The artifacts resulting from training process (24) are in some embodiments used to input embedded, by (21), whole genome segments into a system (23) to determine the probability of a genetic condition in humans. In some embodiments this includes assessing the genetic component of psychotic disorders, depression and suicidal thoughts or tendencies. In other embodimentsAttorney Docket No.68993-715601 system (23) is used to assess one or more of the geometric and molecular properties of the artifacts generated by the DNA or RNA through translation and transcription to proteins or RNA molecules in cells.

[0395] FIG.3 further demonstrates how the DNA or RNA segments are divided into segmentsusing genome loci that determine the mapping between the segments and the corresponding embedding system M5. The figure exemplifies how multiple M5 models (31 to 38) are assigned individually to different start and stop positions in the genome. In some embodiments each M5 model is then fine-tuned specifically to that segment across multiple genomes using training process (24) and in other embodiments the M5 models are shared. After evaluating each model on each segment the resulting embeddings, exemplified as e1 to e8 are stacked up into a matrix before being processed further by model (23) exemplified on FIG.2.

Claims

Attorney Docket No.68993-715601 CLAIMS WHAT IS CLAIMED IS:

1. A method for analyzing biological sequence data, comprising:(a) obtaining biological sequence data; and (b) processing the biological sequence data using a machine learning model comprising a linearized attention mechanism to produce an output.

2. The method of claim 1, further comprising processing the output to generate a predictedeffect of a mutation on a biological process.

3. The method of claim 2, wherein the biological process is a molecular function, amolecular property, a cellular function, a tissue function, or any combination thereof.

4. The method of claim 1, further comprising processing the output to design molecularcomplexes.

5. The method of claim 1, further comprising processing the output to predict geneinteractions.

6. The method of claim 1, further comprising processing the output to predict an effect ofan extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

7. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing DNA.

8. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing RNA.

9. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing a peptide.

10. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing a protein.

11. The method of claim 1, wherein the biological sequence data comprises data derivedfrom epigenomic sequencing.

12. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing mitochondrial DNA.

13. The method of claim 1, wherein the biological sequence data comprises data derivedfrom sequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.Attorney Docket No.68993-71560114. The method of claim 1, wherein the input data comprises textual data provided by auser.

15. The method of claim 14, wherein the textual data comprises written instructions that areprocessed by the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

16. The method of claim 1, wherein the machine learning model comprises an LLM.

17. The method of claim 1, wherein the output comprises a sequence of tokens.

18. The method of claim 17, wherein the output comprises at least 650 tokens.

19. The method of claim 1, wherein the input data comprises a sequence of tokens.

20. The method of claim 19, wherein the sequence of tokens are single nucleotideresolution.

21. The method of claim 19, wherein the sequence of tokens is at least 2 million tokens inlength.

22. The method of claim 19, wherein the sequence of tokens is at least 3 million tokens inlength.

23. The method of claim 19, wherein the sequence of tokens is at least 4 million tokens inlength.

24. The method of claim 19, wherein the sequence of tokens is at least 5 million tokens inlength.

25. The method of claim 19, wherein the sequence of tokens is at least 6 million tokens inlength.

26. The method of claim 19, wherein the input data comprises a sequence of at most 2.5million tokens.

27. The method of claim 19, wherein the input data comprises a sequence of at most 3million tokens.

28. The method of claim 19, wherein the input data comprises a sequence of at most 4million tokens.

29. The method of claim 19, wherein the input data comprises a sequence of at most 5million tokens.

30. The method of claim 19, wherein the input data comprises a sequence of at most 6million tokens.

31. The method of claim 1, further comprising predicted likelihood of association of agenetic variation to a phenotype.Attorney Docket No.68993-71560132. The method of claim 31, wherein the phenotype comprises at least one of anautoimmune disease, a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

33. The method of claim 31, wherein the phenotype comprises early onset of a disease.

34. The method of claim 31, where the phenotype comprises developmental delays.

35. The method of claim 31, further comprising, based at least on the association of thegenetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.

36. The method of claim 35, wherein determining that an elevated risk of having thephenotype comprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

37. The method of claim 35, wherein determining the elevated risk of having the phenotypecomprises a specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

38. The method of claim 1, further comprising assaying a sample from a subject to generatesequencing information.

39. The method of claim 38, wherein the sample is a buccal sample, a blood sample, asaliva sample, a urine sample, a cell sample, or a tissue sample.

40. The method of claim 38, further comprising extracting DNA from the bodily sample.

41. The method of claim 40, further comprising subjecting the DNA to a sequencingreaction.

42. The method of claim 41, wherein the sequencing reaction comprises a whole genomesequencing reaction, or an exome sequencing reaction, or both.

43. A method comprising:implementing a model comprising a set of layers comprising a linearized attention mechanism and configured to take an input, comprising a first sequence of tokens, and produce an output, comprising a second sequence of tokens; using an optimizer to improve the model’s performance by altering the set of layers thereby causing the output to be altered; receiving a set of data; processing by the model the set of data as input thereby producing the output; causing the optimizer to alter the set of layers based at least in part on the output and a measure of error of the model; andAttorney Docket No.68993-715601 storing the model.

44. The method of claim 43, wherein at least one token of the first sequence of tokens ismasked and wherein the second sequence of tokens predicts the at least one token that is masked in the first sequence of tokens.

45. The method of claim 44, wherein the measure of error is based at least in part on the atleast one token that is masked in the first sequence of tokens and the prediction of the at least one token in the second sequence of tokens.

46. The method of claim 43, further comprising processing the output to generate apredicted effect of a mutation on a biological process.

47. The method of claim 44, wherein the biological process is a molecular function, amolecular property, a cellular function, a tissue function, or any combination thereof.

48. The method of claim 43, further comprising processing the output to design molecularcomplexes.

49. The method of claim 43, further comprising processing the output to predict geneinteractions.

50. The method of claim 43, further comprising processing the output to predict an effect ofan extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

51. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing DNA.

52. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing RNA.

53. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing a peptide.

54. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing a protein.

55. The method of claim 43, wherein the biological sequence data comprises data derivedfrom epigenomic sequencing.

56. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing mitochondrial DNA.

57. The method of claim 43, wherein the biological sequence data comprises data derivedfrom sequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.Attorney Docket No.68993-71560158. The method of claim 43, wherein the input data comprises textual data provided by auser.

59. The method of claim 58, wherein the textual data comprises written instructions that areprocessed by the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

60. The method of claim 43, wherein the machine learning model comprises an LLM.

61. The method of claim 43, wherein the output comprises a sequence of tokens.

62. The method of claim 61, wherein the output comprises at least 650 tokens.

63. The method of claim 43, wherein the input data comprises a sequence of tokens.

64. The method of claim 63, wherein the sequence of tokens are single nucleotideresolution.

65. The method of claim 63, wherein the sequence of tokens is at least 2 million tokens inlength.

66. The method of claim 63, wherein the sequence of tokens is at least 3 million tokens inlength.

67. The method of claim 63, wherein the sequence of tokens is at least 4 million tokens inlength.

68. The method of claim 63, wherein the sequence of tokens is at least 5 million tokens inlength.

69. The method of claim 63, wherein the sequence of tokens is at least 6 million tokens inlength.

70. The method of claim 63, wherein the input data comprises a sequence of at most 2.5million tokens.

71. The method of claim 63, wherein the input data comprises a sequence of at most 3million tokens.

72. The method of claim 63, wherein the input data comprises a sequence of at most 4million tokens.

73. The method of claim 63, wherein the input data comprises a sequence of at most 5million tokens.

74. The method of claim 63, wherein the input data comprises a sequence of at most 6million tokens.

75. The method of claim 43, further comprising predicted likelihood of association of agenetic variation to a phenotype.Attorney Docket No.68993-71560176. The method of claim 31, wherein the phenotype comprises at least one of anautoimmune disease, a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

77. The method of claim 75, wherein the phenotype comprises early onset of a disease.

78. The method of claim 75, where the phenotype comprises developmental delays.

79. The method of claim 75, further comprising, based at least on the association of thegenetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.

80. The method of claim 79, wherein determining that an elevated risk of having thephenotype comprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

81. The method of claim 79, wherein determining the elevated risk of having the phenotypecomprises a specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

82. The method of claim 43, further comprising assaying a sample from a subject togenerate sequencing information.

83. The method of claim 82, wherein the sample is a buccal sample, a blood sample, asaliva sample, a urine sample, a cell sample, or a tissue sample.

84. The method of claim 82, further comprising extracting DNA from the bodily sample.

85. The method of claim 84, further comprising subjecting the DNA to a sequencingreaction.

86. The method of claim 41, wherein the sequencing reaction comprises a whole genomesequencing reaction, or an exome sequencing reaction, or both.

87. The method of claim 43, wherein the processing is autoregressive.

88. A method for analyzing biological sequence data, comprising:(a) obtaining biological sequence data; (b) generating a sequence of tokens at least 2 million tokens in length based at least in part on the biological sequence data; and (c) processing the sequence of tokens using a machine learning model to produce an output.

89. The method of claim 88, further comprising processing the output to generate apredicted effect of a mutation on a biological process.Attorney Docket No.68993-71560190. The method of claim 89, wherein the biological process is a molecular function, amolecular property, a cellular function, a tissue function, or any combination thereof.

91. The method of claim 1, further comprising processing the output to design molecularcomplexes.

92. The method of claim 88, further comprising processing the output to predict geneinteractions.

93. The method of claim 88, further comprising processing the output to predict an effect ofan extracellular molecule on a subject based on input data derived from at least a portion of DNA of the subject.

94. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing DNA.

95. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing RNA.

96. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing a peptide.

97. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing a protein.

98. The method of claim 88, wherein the biological sequence data comprises data derivedfrom epigenomic sequencing.

99. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing mitochondrial DNA.

100. The method of claim 88, wherein the biological sequence data comprises data derivedfrom sequencing DNA comprising at least one of the 1q21.1 region, the 1q21.2 region on chromosome 1, and the Human Leukocyte Antigen (HLA) region at chromosome 6p21.3 of the human genome.

101. The method of claim 88, wherein the input data comprises textual data provided by auser.

102. The method of claim 101, wherein the textual data comprises written instructions thatare processed by the machine learning model to alter the processing of the input, such that the output of the machine learning model satisfies the written instructions.

103. The method of claim 88, wherein the machine learning model comprises an LLM.

104. The method of claim 88, wherein the output comprises a sequence of tokens.

105. The method of claim104, wherein the output comprises at least 650 tokens.

106. The method of claim 88, wherein the input data comprises a sequence of tokens.Attorney Docket No.68993-715601107. The method of claim 106, wherein the sequence of tokens are single nucleotideresolution.

108. The method of claim 106, wherein the sequence of tokens is at least 2 million tokens inlength.

109. The method of claim 106, wherein the sequence of tokens is at least 3 million tokens inlength.

110. The method of claim 106, wherein the sequence of tokens is at least 4 million tokens inlength.

111. The method of claim 106, wherein the sequence of tokens is at least 5 million tokens inlength.

112. The method of claim 106, wherein the sequence of tokens is at least 6 million tokens inlength.

113. The method of claim 106, wherein the input data comprises a sequence of at most 2.5million tokens.

114. The method of claim 106, wherein the input data comprises a sequence of at most 3million tokens.

115. The method of claim 106, wherein the input data comprises a sequence of at most 4million tokens.

116. The method of claim 106, wherein the input data comprises a sequence of at most 5million tokens.

117. The method of claim 106, wherein the input data comprises a sequence of at most 6million tokens.

118. The method of claim 88, further comprising predicted likelihood of association of agenetic variation to a phenotype.

119. The method of claim 118, wherein the phenotype comprises at least one of anautoimmune disease, a cancer, a neurodevelopmental disorder, type 1 diabetes, celiac disease, depression, schizophrenia, susceptibility to suicidal thoughts or action, or any combination thereof.

120. The method of claim 118, wherein the phenotype comprises early onset of a disease.

121. The method of claim 118, where the phenotype comprises developmental delays.

122. The method of claim 118, further comprising, based at least on the association of thegenetic variation with the phenotype, determining that the subject has an elevated risk of having a phenotype.Attorney Docket No.68993-715601123. The method of claim 122, wherein determining that an elevated risk of having thephenotype comprises a sensitivity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

124. The method of claim 122, wherein determining the elevated risk of having thephenotype comprises a specificity of at least about 50%, about 60%, about 70%, about 80%, or about 90%.

125. The method of claim 88, further comprising assaying a sample from a subject togenerate sequencing information.

126. The method of claim 125, wherein the sample is a buccal sample, a blood sample, asaliva sample, a urine sample, a cell sample, or a tissue sample.

127. The method of claim 125, further comprising extracting DNA from the bodily sample.

128. The method of claim 127, further comprising subjecting the DNA to a sequencingreaction.

129. The method of claim 128, wherein the sequencing reaction comprises a whole genomesequencing reaction, or an exome sequencing reaction, or both.

Images