Generating a sample with a trained Bayesian flow network for development of an amino acid product

Abstract

Computer method for generating, with a Bayesian Flown Network (BFN), an amino acid sequence comprising the identity of an amino acid token at a plurality of sequence positions. The method includes a

Description

Technical Field The present invention relates to a computer-implemented method that generates a sample with a trained Bayesian Flow Network for development of an amino acid product. Also provided is a program and an information processing apparatus. Background Proteins drive nearly every biological process in biological systems, playing a central role in the operations of both healthy and pathological processes. However, despite their critical role, only a tiny fraction of the possible proteome - characterised by the vast combinatorial complexity of possible amino acid sequences - has been explored. This represents not only a gap in our fundamental knowledge of biological processes, but also potentially untapped opportunities for developing novel functional and therapeutic proteins. In recent years, machine learning has brought improvements to computational biology. Techniques initially developed for natural language processing have proven particularly impactful. Drawing on parallels between modelling sequences of words and amino acids (the ‘language of proteins’), these self-supervised models learn to generate novel sequences from corpora of unlabelled data. Proteins are large biomolecules, or macromolecules, which are made of sequences of amino acids linked together by peptide bonds. In some cases, it may be desirable to change some of the amino acids in a protein to modify the characteristics of the protein. For example, when developing new drugs, such as a treatment for Influenza, it may be desirable to design a protein that has greater binding affinity with Influenza virus HA protein than the human receptor protein. The underlying biological and chemical processes which govern the sequence of amino acids in a protein and their relative three-dimensional structure make evaluating and modifying proteins a labour-intensive task. Given that proteins are usually composed of a number N of amino acids ranging from 50 to 2,000, there may be many positions in a protein sequence that could in theory be modified in an attempt to change a given characteristic of the protein. Accordingly, improvements that can analyse the protein structure and can generate amino acid sequences that are similar to those that occur in nature are desirable. Summary According to a first aspect of the present invention, there is provided a computer-implemented method that generates a sample with a trained Bayesian Flow Network for development of an amino acid product, the sample being an amino acid sequence comprising the identity of an amino acid token at each of a plurality of sequence positions, the method comprising for a plurality of iterations: sampling a vector, y, from a first distribution that depends on a prediction, an accuracy schedule that increases accuracy as the iterations progress, and isotropic noise; generating a second distribution around the sampled value, y, and inputting the parameters of the second distribution to the trained Bayesian Flow Network to obtain an updated prediction; wherein after the plurality of iterations, the predictions identify the generated sample amino acid sequence. In some embodiments, the amino acid product may be a drug. The method may further comprise manufacturing the drug. In other embodiments, the amino acid product may be a genetically engineered organism. In such embodiments, the method may comprise manufacturing the genetically engineered organism. The method may be a method that identifies a sample amino acid sequence that in-paints a seed sequence that defines target amino acid tokens for a subset of the amino acid positions in the sequence. The method may comprise: sampling a plurality of vectors, yj, from the first distribution to generate a set of particles; generating a plurality of second distributions, a respective second distribution being formed around each sampled particle, yj; applying the neural network to parameters of each second distribution determined from the particles to generate a first set of predictions; performing particle filtering on the first set of predictions; and obtaining an updated prediction by generating a third distribution from the filtered set of predictions and sampling the third distribution to generate the updated set of predictions. The parameters input to the Bayesian Flow Network may include one or more entropy values representing an entropy of the input distribution. The method may comprise: repeating the method to identify a plurality of candidate amino acid sequences, inputting each candidate amino acid sequence to the Bayesian Flow Network to determine a likelihood of the candidate amino acid sequence; dividing an inverse of the determined likelihood of each candidate amino acid sequence by the candidate amino acid sequence length to generate a perplexity score; and identifying one or more candidate amino acid sequences based on the generated perplexity scores. In some implementations, the method further comprises using the Bayesian Flow Network to generate a sample amino acid sequence that is subject to one or more constraints. The constraint may be enforced by sampling conditionally from the output distribution of the Bayesian Flow Network. In some implementations, the constraint comprises one or more of: a geometrical parameter; sequence length; isoelectric point; an instability index; secondary structure fraction; charge at ph; molecular weight; paired sequence alignment (both protein and rna); paired structure alignment (could be modelled and / or experimental); germline genes; germline gene identities; light chain type (kappa / lambda); complementary-determining regions (cdr) lengths; cdr canonical forms; hydrophobicity; electrostatic patches (type, area, size etc.); charge distribution; antigen sequence; antigen structure (possibly collecting all structure under a single complex structure mode); species; expression; stability (e.g. thermostability from melting curves); and affinity. The Bayesian Flow Network may comprise at least one of an attention-based network, a transformer-based language model, a convolutional neural network, and a graph neural network. The first distribution may be given by y <- -1) + K / 3 (t)z, where P(Z) is the accuracy schedule, Ai is a number of types of available amino acid token, 0 is the prediction, and z is the isotropic noise. The accuracy schedule may determine the accuracy based on a time parameter, t, that identifies the position of the current iteration within the plurality of iterations. In some implementations the accuracy schedule is given by ^(t) <- ^t2, where is a free parameter. In some implementations the second distribution is a normal distribution, and a parameter of the normal distribution is determined based on the accuracy schedule. The method may further comprise training the Bayesian Flow Network, wherein during training the method comprises: setting initial input distribution parameters that define an input distribution for each category; and for a plurality of iterations: determining a sender distribution that defines a distribution for each category based on a data sample and an accuracy parameter determined for the current iteration; generating a sample from the sender distribution; using Bayesian inference to generate a new input distribution based on the sample and an existing input distribution; inputting parameters of the new input distribution into the Bayesian Flow Network to generate output parameters that define an output distribution for each category; generating a receiver distribution based on the output distribution parameters and the accuracy parameter for the next iteration; determining the value of a loss function based on a difference between the sender distribution and the receiver distribution; and updating parameters of the Bayesian Flow Network based on the output of the loss function. The accuracy parameter may be parameterised in an accuracy schedule that defines the accuracy parameter at each time. A parameter of the accuracy schedule may be included in the loss function, and the parameter may be learned during the training of the Bayesian Flow Network. In some implementations a separate accuracy schedule is used for each of the plurality of sequence positions, wherein a parameter of the accuracy schedule for each sequence position is learned during the training of the Bayesian Flow Network. According to a second aspect of the invention there is provided a program that, when executed by an information processing apparatus, causes the information processing apparatus to perform a method according to the first aspect of the invention. According to a third aspect of the invention there is provided an information processing apparatus comprising a processor and a storage, wherein the storage comprises instructions that when executed causes the information processing apparatus to perform a method that generates a sample with a trained Bayesian Flow Network for development of an amino acid product, the sample being an amino acid sequence comprising the identity of an amino acid token at each of a plurality of sequence positions, the method comprising for a plurality of iterations: sampling a vector, y, from a first distribution that depends on a prediction, an accuracy schedule that increases accuracy as the iterations progress, and isotropic noise; generating a second distribution around the sampled value, y; and inputting the parameters of the second distribution to the trained Bayesian Flow Network to obtain an updated prediction; wherein after the plurality of iterations, the predictions identify the generated sample amino acid sequence. Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only, which is made with reference to the accompanying drawings. Brief Description of the Drawings Figure 1 is a schematic diagram showing a step during training of a Bayesian Flow Network; Figure 2 shows an algorithm for generating a sample from a trained Bayesian Flow Network; Figure 3 shows an algorithm for generating an inpainted sample in which some amino acid tokens are known using a Bayesian Flow network; Figure 4 is a chart showing a comparison between the density of different amino acid tokens between samples generated by ProtBFN and those occurring in the UniProtCC dataset; Figure 5 is a chart showing frequency of occurrence of oligomers in both samples generated by ProtBFN and those occurring in the UniProtCC dataset; Figure 6 is a graph showing a comparison of the distribution of lengths in various datasets; Figure 7 is a graph showing mean predicted local distance difference test scores (pLDDT) for various datasets; Figure 8 is a graph showing a density of the number of samples generated by ProtBFN having a given sequence similarity to a sequence occurring in the UniProtCC dataset; Figure 9 is a chart showing occurrence of Coil, Beta, and Helix structures by normalized position within the amino acid sequence for both ProtBFN generated samples and samples occurring in the UniProtCC dataset; Figure 10 is a table showing Amino Acid Recovery (AAR) rates on observed antibody space (OAS) test data for AbBFN in comparison to other antibody-specific language models; Figure 11 is a table showing Amino Acid Recovery (AAR) rates for various models on the 10-fold SAbDab benchmark; and Figure 12 is a schematic diagram showing elements of an information processing apparatus. Detailed Description When developing new proteins, for example, to create new drugs or genetically modified organisms, a known protein may be selected as a starting point. Modifications are made to the known protein to change the characteristics of the protein according to one or more criteria. Proteins generally have an amino acid sequence length of between 50 and 2,000 amino acids, where any one of 20 possible amino acid tokens may be present at each of the 50 to 2,000 positions in the amino acid sequence. Given the large number of positions in an amino acid sequence and the number of possible amino acids, the total number of variations of a protein which could be produced are often too high to reasonably compute and test individually. In particular, for a given sequence with (pointwise) M modifications, there are 19M possible variations, without taking into account additions or deletions. Methods described herein may be used to generate amino acid sequences. The methods may also be used to evaluate an amino acid sequence to provide information which can be used to identify one or more amino acids in the sequence which are promising starting points for modification when attempting to develop a new amino acid sequence, such as a protein, according to one or more criteria. The methods described herein may also be used to inpaint between known portions of an amino acid sequence in order to generate new plausible amino acid sequences.. The methods described can provide information which is representative of underlying characteristics of an amino acid sequence, and which can be used when developing new amino acid sequences. In certain examples, the methods described herein include determining probability values which provide estimations of the physiologically plausibility of given amino acid sequences. A representation of an amino acid sequence comprises a sequence of two or 5 more letters. Each letter of the sequence of letters corresponds to a respective amino acid of a set of possible amino acid tokens and a position of each letter in the sequence of letters represents a respective position of the respective amino acid in the sequence. The following table shows a set of possible amino acids which may be present at each position in the amino acid sequence. AMINO ACID ASSOCIATED LETTER Alanine A Arginine R Asparagine N Aspartic acid D Cysteine C Glutamine Q Glutamic acid E Glycine G Histidine H Isoleucine I Leucine L Lysine K Methionine M Phenylalanine F Proline P Serine S Threonine T Tryptophan W Tyrosine Y Valine V Introduction to Bayesian Flow Networks 15 Bayesian Flow Networks are described in the paper “Bayesian Flow Networks” A. Graves et al, arXiv preprint arXiv:2308.07037 (2023). The following description gives a brief overview of some aspects of this paper to assist with understanding the present application. However, for a detailed description the reader is referred to the original paper, which is incorporated in its entirety herein by reference for all purposes. Figure 1 is a schematic diagram illustrating a step in a sequence of steps for training a Bayesian Flow network (BFN) 10 using the methods described in the Bayesian Flow Networks paper. Note that the BFN 10, which will be described in greater detail below, is shown twice in Figure 1 (i.e. it is the same network) because training the neural network is repetitive process performed across a training data set and only a single sample from a single training protein sequence is illustrated. The illustrated method involves a plurality of variables, such as amino acid positions, each of which may take one of three categorical values, A, B and C, such as amino acid token values. The use of three categorical values is for ease of illustration and is not limiting. As explained above, the number of different amino acid tokens is typically twenty and the described methods are applicable regardless of the number of categorical values and the number of variables (positions in the amino acid sequence). The processing for variable 1 is shown in detail in Figure 1, but the same process may be performed for each variable (i.e. each amino acid position). Details for the other variables are omitted for ease of illustration. In terms of terminology, the BFN 10 receives parameters, 0, of an input distribution 12 and outputs parameters, y, of an output distribution 13. The neural network 10 may also receive a process time, t, or as described further below a value of entropy. For example, as illustrated, the neural network may take parameters, 0(d). that consist of probabilities of the categorical distribution (i.e. the probability of each of A, B, and C) in the input distribution 12. To train the BFN 10, a sender distribution 14 is generated from training data 15. The training data may have reference values, such as a string of known amino acid sequence token values from a naturally occurring sequence. Accordingly, in the example shown, the value B is known from the training data for variable 1. To generate the Sender distribution 14, noise is added to the data. Accordingly, for each category, a distribution of values is generated as shown in Figure 1. The centers of the distributions are shifted. For example, for present example, the distributions for values A and C will be centered on zero because these data values are not present in the training data value. The distribution for B will be shifted to represent the higher probability that the value is B. The amount of noise added at each step of training the BFN 10 on a single training data value is determined by an accuracy parameter, a. The way the accuracy varies (often increases) is referred to as the accuracy schedule P(t) which is determined as follows: d a(t) = — / ?(t) The iterations in the training on a single training data value are parameterized by the value, t, which represents process time. As noted above, the training is iterative and the process takes N steps where N is typically a larger number, which can be generalized to a continuous time, t. A sample, y, is taken from the Sender distribution 14 and a previous input distribution for the neural network is updated to generate a new input distribution. The previous input distribution 15 is the input distribution that was input to the BFN 10 at the previous iteration of the method. The categorical probabilities from the previous input distribution 15 are updated using Bayesian inference to determine the probabilities for the new input distribution 12 given knowledge of the new sample, y, from the sender distribution. The goal of the training of the BFN 10 is for the neural network to generate output parameters corresponding to the probabilities of the categories for the variable (e.g. probabilities of amino acid token values at a particular position) given the input probabilities for each variable (the input distribution 12). Accordingly, once trained, the BFN 10 can be used to generate a measure of the probabilities for each amino acid token in an amino acid sequence by inputting an input distribution corresponding to the amino acid sequence. To determine a loss and allow back propagation, a Receiver distribution 16 is generated from the output distribution 13 using the same noise that was used to generate the sender distribution 15 (i.e. according to the accuracy schedule). A Kullback-Leibler (KL) divergence is calculated between the Receiver distribution 16 and the Sender distribution 14. The KL divergence is a measure of how much one probability distribution is different from the second probability distribution. The loss function is defined in two parts. The first part, referred to as discretetime loss, relates to the number of nats (natural units of information) required to transmit the sender samples generated so far (yi ... yn at step n). The second part, referred to as reconstruction loss, is the expected number of nats required to then transmit the sample x under the Output distribution. The total loss is defined as: L(x) = Ln{x} + L'(x) where L is the overall loss function, Ln is the n-step discrete time loss and U is the reconstruction loss . In the method above, performed for n steps for each single training data value, the noisiness of the input distribution (generally) decreases over time as it is obtained by performing Bayesian inference from the previous input distribution, taking into account the noisy observation. This process generally reduces the noise in the input distribution as time goes on, regardless of how noisy the observation is, noting that a noisier observation will generally lead to a smaller reduction in noise in the input distribution. The above process is repeated for a number of steps, n, for each training data value and repeated across multiple sender distributions corresponding to different training data values from known amino acid sequences in a data set. The loss L is used to train the neural network, such as by using an Adam optimizer. Other techniques for training neural networks are known in the art and may be used as appropriate. After training the BFN 10, during inference / sample generation, sample protein sequences may be generated as follows. A set of initial values of the output distribution are generated. As no information about the values of the category for each variable (e.g. amino acid token for each location) is known, the initial parameter values, 6, may be set to 1 / K, where K is the number of categories (e.g. the number of types of amino acid token). A uniform prior distribution is used for mathematical simplicity. In other implementations, initially fitted probability distributions could be used. For a series of iterations, N, the following steps are performed where t is defined at the ith iteration as: i - 1 A noisy observation of the amino acid value at a variable (i.e. location in the amino acid sequence), x, is generated as follows. The output distribution is generated as a softmax distribution based on the output parameters generated by the network 10 (in the first step using the prior output distribution discussed above). A sample vector, k, is taken from the output distribution. A sample, y, is taken from a Gaussian distribution centered on k with a standard deviation determined based on the accuracy parameter / sampling step. The sample is taken from the distribution defined in the following equation: PR(yl^;^) = (y\a{Ke(k} - l^aKl) where the categorical probabilities in the output distribution are xi and accuracy parameter is a. J\f is a Gaussian distribution, 1 is a vector of ones, K is the size of the vocabulary (i.e. the number of categories - in this example the number of types of amino acid code) and I is the identity matrix. e(k) is a one-hot encoding that takes a value 1 if xi is equal to the category value and 0 otherwise. Using Bayesian inference a new input distribution, Pi, for each category, Fj, of the output distribution, xi, given the sampley is determined as follows: Pi(x: = y>’ = ----7 = Pi i i \ l j u >j n f ^k=lUk where Pi-ij is the previous input distribution’s parameters and is the new set of input distribution parameters. The new input distribution is a ‘best guess’, on a per category basis, of the true value of the category (amino acid token). The input distribution is then input into the trained Bayesian Flow Network 10 to generate parameters for a further output distribution. This process effectively improves upon the guess by taking into account the updated input parameters for each category at each position (variable) along the amino acid sequence. Accordingly, the whole context of the amino acid sequence is considered. This process is repeated with reducing uncertainty according to the accuracy schedule. In this way, the inference / sample generation process ‘hallucinates’ a new protein sequence. Model details, training data, and procedural details Two models specific example models will now be described. The reader will appreciate that the techniques described herein can be applied to many different models and data sets and that the examples given are for the purposes of illustration only and are not limiting. ProtBFN is a 650 million parameter model trained for de novo generation of protein sequences. ProtBFN was then further refined to produce a second antibody specific model, referred to as AbBFN. ProtBFN uses a network that is based on the 650 million parameter architecture used in ‘Lin et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction, bioRxiv, 2022' which is a BERT-style encoder-only transformer (see Devlin et al. ‘Bert: Pre-training of deep bidirectional transformers for language understanding’ arXiv preprint arXiv: 1810.04805, 2018) with rotary positional embeddings (see Su et al: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024). The architecture consists of 33 layers, each consisting of 20-head multi-head self-attention (see Vaswani et al. ‘Attention is all you need.’ Advances in neural information processing systems, 30, 2017). in 1280-dimensional space followed by a single-layer MLP with GeLU activation (see Hendrycks et al. ‘Gaussian error linear units (gelus)’. arXiv preprint arXiv: 1606.08415, 2016) and a hidden dimension of 5120. A significant difference among several differences between ProtBFN’s architecture and that used in Lin et al, ‘Language models of protein sequences at the scale of evolution enable accurate structure prediction’, bioRxiv, 2022' is that the initial token embedding is replaced with a linear projection. This change is because Bayesian Flow Network techniques are being applied and the BFN’s input 6 is a distribution over possible token values as explained above. ProtBFN is trained on data obtained from the January 2024 release of UniProtKB. The data is filtered according to the ‘Protein Existence’ (PE) property, including only those proteins which are inferred from homology (PE = 3), has evidence at the transcript level (PE = 2) or evidence at the protein level (PE = 1). By removing those proteins which are known to be hypothetical (PE = 4) or are of unknown existence (PE = 5), ProtBFN was restricted to model the distribution of proteins that are very likely to exist, meaning that greater confidence can be placed in the sequences it generates. Additionally, it was substantially faster to train a model when removing hypothetical proteins, indicating that they introduce substantial amounts of additional entropy which may imply their generally lower quality. Additionally, ProtBFN was trained only on those sequences with length Z <512, with the final token used to encode an end-of-sequence (EOS) token. After filtering by PE and length, the final training set contained 71 million sequences. In examples below where clusters are used to reweight and debias the data, the clusters are obtained from the UniRef50 dataset. Each sequence is represented by its amino acids followed by an end-of-sequence (EOS) token. All other tokens after EOS are PAD tokens which are treated as normal tokens for the purposes of noisy observations, predictions, and the loss. As this dataset is a (C)leaned and (C)lustered subset of UniProt, we will refer to it for the rest of the description as UniProtCC. ProtBFN is first pre-trained for 250,000 training steps with a batch size of 8,192. It was found that a large batch size was appropriate to obtain stable gradient estimates. Adam optimiser is used with ^1 = 0.9 and / ?2 = 0.98. The learning rate is initialised to 0 and linearly increased to 10 4 at step 10,000, after which it is held constant. Throughout training, the norm of the gradient is clipped to 500. A copy of the network’s parameters is maintained with an exponential moving average of the weights with decay rate 0.999. During this first phase of training, samples are drawn uniformly at random from all training data. Next, the model is trained for a further 250,000 steps with clustered data. Specifically, each cluster is constructed by taking all samples within the corresponding UniRef50 cluster which pass through both the Protein Existence and the length filters as discussed above. During this training phase, each cluster is sampled with probability proportional to the square root of its size, so that ProtBFN is debiased away from those proteins most heavily studied by humans, but not overly focused on very data-sparse 1-member clusters as would happen when uniformly sampling clusters or training only on the UniRef50 cluster centers. Once a cluster has been sampled, any sequence contained within it is chosen uniformly at random. During this second phase of training, the Adam optimiser is completely reset and again, the learning rate is linearly increased from 0 to 105 over the first 10,000 steps. While detailed training data and processes have been described above, these are provided by way of example and the reader will appreciate that the techniques applied and datasets used can be varied, cleaned and adapted according to context, as is customary in the art. As noted above, a second BFN, AbBFN, was generated from ProtBFN. The training data for AbBFN uses the unpaired OAS dataset discussed in Olsen et al ‘Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences’, Protein Sci., 31 (1): 141 — 146, January 2022 as was available on 21st February 2024. The data was filtered as follows: 1. Filter out the studies: "Bonsignori et al., 2016", "Halliley et al., 2015", "Thomqvist et al., 2018". 2. Filter out studies originating from immature B cells (BType is "Immature-B-Cells" or "Pre-B-Cells"). 3. Filter out studies originating from B cell cancers (Disease is "Light Chain Amyloidosis" or "CLL"). Then for each remaining study filter the sequences as follows: 1. Filter if: sequence contains a stop codon; sequence is non-productive; V and J regions are out of frame; Framework region 2 is missing; Framework region 3 is missing; CDR3 is longer than 37 amino acids; J region identity is less than 50%; CDR edge is missing an amino acid; locus does not match sequence type. 2. Remove sequences if they only appear once in a study, then make unique. 3. Filter if the conserved cysteine residue is not present or is misnumbered in the ANARCI numbering. 4. Create the near-full-length sequence IMGT positions 21 through 128 (127 for light sequences and for heavy sequences from rabbits and camels). Filter if framework 1 region is <21 amino acids. 5. Remove duplicate near-full-length sequences and tally up the counts, filter out sequences which only appeared once. These sequences are dropped on the grounds of insufficient evidence that they are a genuine biological sequence as opposed to a sequencing error. 6. Filter out sequences which contain any amino acids which are not in the standard twenty identified above. 7. Use the sequence from the full ANARCI numbering, not the near-full-length sequence. This filter method corresponds to all of the filters / preprocessing described in more detail in Bachas et al. ‘Antibody optimization enabled by artificial intelligence predictions of binding affinity and naturalness’ BioRxiv, pages 2022-08, 2022. AbBFN uses the full ANARCI sequence instead of the near-full-length sequence. An additional filter is applied which removes any sequences which had an ANARCI numbering with an empty region. To create a SAbDab test set, SAbDab (The Structural Antibody Database) data downloaded on 29th February 2024 was used, specifically the summary table to select only the paired sequences (paired sequences ensures no camelid sequences - most unpaired heavy sequences are camelid and come from different genetic lineages which are not present in the training data). Single-sequence fragment variable (scFv) antibodies were also removed because ANARCI has trouble parsing these correctly. The SEQRES attributes are then parsed from the original SAbDab PDB files, and ANARCI (Dunbar et al. ‘ANARCI: antigen receptor numbering and receptor classification’ Bioinformatics, 32(2):298-300, 2016) is run on these to obtain the final sequences. To ensure that the training data is dis-similar to the testing data the filtered OAS data is split into heavy and light sequences, similarly SAbDab is split into heavy and light. Uniformly at random 20,000 heavy sequences were selected from the OAS data to create a fasta file which combines the 20,000 sample of OAS data and the heavy SAbDab sequences. An MMSeqs2 search (Steinegger et al. ‘Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets’ Nature biotechnology, 35(11):1026^1028, 2017) was run from the OAS data without the 20,000 sampled sequences against the 20,000 heavy OAS sequence sample and heavy SAbDab with default sensitivity 5.7, minimum sequence identity 0.95, coverage 0.8 and coverage mode 0. Any hits from the training data were removed. The test data that belonged to SAbDab was found to be more out-of-distribution to OAS and filtering out the similar sequences still retained 99% of the data, whereas filtering out sequences which are similar to the 20,000 uniform test set retained 79% of the OAS data. Filtering out all the similar sequences gives 195 million training examples, from 248 million before sequence similarity filtering. AbBFN was fine-tuned from the ProtBFN model on the filtered OAS data. For computational efficiency, the maximum sequence length was reduced to 256. AbBFN was trained for 100,000 steps with a batch size of 8,192. Adam optimization was used with / ?i = 0.9 and fh = 0.98. The learning rate was initialised to 0 and linearly increased to 10 5 at step 10,000, after which it was held constant. Throughout training, the norm of the gradient was clipped to 500. A copy of the network’s parameters is maintained with an exponential moving average of the weights with decay rate 0.999. To move the output of the AbBFN model closer to the distribution of SAbDab heavy sequences, the model was further fine-tuned on the 9 train folds for each test fold within the 10-fold cross validation methodology of Kong et al ‘Conditional antibody design as 3d equivariant graph translation’ arXivpreprint arXiv.2208.06073, 2022. The AbBFN model was fine-tuned for 1,000 training-steps with a batch size of 512. Adam optimizer is used with / ?i = 0.9 and ^2 = 0.98 and the learning rate is linearly increased from 0 to 10 5 over the 1,000 training steps. Throughout training, the norm of the gradient is clipped to 500. A copy of the network’s parameters is maintained with an exponential moving average of the weights with decay rate 0.995. Again, while detailed training data and processes have been described, these are provided by way of example and the reader will appreciate that the techniques applied and datasets used can be varied, cleaned and adapted according to context, as is customary in the art. ODE sampling technique The sample generation method for Bayesian Flow networks in general was described above. However, the sample generation technique used for ProtBFN and AbBFN was improved compared to that described in the original BFN paper. The updated method, which is expected to work regardless of the details of the dataset, is illustrated in Figure 2. A variety of alternative sampling methods were investigated that reduced the overall temperature of sample generation from ProtBFN. In the conventional discretedata sampling method. The discrete sample generation process moves from yt at time t toys at time s according to the equation: ys = yt + (^^(0) - 1) + yjat^sKz where 6t = softmax(jzt), z ~ N(0, Id / x) is isotropic noise, and at^s = ~ is the change in accuracy schedule, / ?. K is the number of classes (i.e. the number of different amino acid tokens). By the additivity of normally distributed variables and with yo = 0, the distribution of ytn given preceding steps 0, ti . . . ytn^ is: n-1 ytn = / 3(ty -1) + (=0 Sampling would yield an Ordinary Differential Equation. However, it was found that by replacing the summation of previous predictions with the most recent prediction, e.g., ytn = P(.tn) (^£(^-1)) -1)+ substantially reduced the perplexity of generated samples. Effectively, this method is equivalent to taking the most recent prediction 0tn_1 and supposing that it had been predicted at every step 0, ti . . , / n-r The direction of the noise remains stationary throughout the sampling process, and it was found that this ‘Ordinary Differential Equation-like’ behaviour was a more stable approach. The overall sampling method is illustrated in Figure 2. In steps 1 and 2 it is noted that the number of iterations, N, and the trained BFN, ¢, are inputs. The output is the generated sample sequence, x. In step 3 isotropic noise with a normal distribution for D amino acid positions and K amino acid token types is prepared. In step 4, predictions, 0 , are initialized at 0. In steps 6 to 7, the time, I, is defined in relation to the number of iterations, N and an accuracy schedule P is defined with reference to a parameter Pi, which is a free hyperparameter that determines the final accuracy. In step 8, a valuer is sampled using the equation described above. At step 9, a softmax distribution is formed about the sampled value, y, and parameters of the distribution, 9, are determined. In step 10, the parameters of the distribution and the entropy (P(t)) are entered into the BFN, ¢, to generate a prediction, 0. The use of entropy (P(t)) will be discussed in more detail further below. The steps are repeated for N steps and the resulting prediction, 0, that will converge on a sampled sequence, x, due to the accuracy schedule. Conditional and unconditional generation of sequences Performing inference / sample generation starting from, for example, a uniform prior distribution as described above in connection with BFNs using the ProtBFN or AbBFN models will generate an unconditional sequence in which the amino acid token at each location in the sequence is hallucinated. However, for some tasks it may be desired to set the value of the amino acid tokens at one or more locations. This is referred to herein as conditional generation of sequences or ‘inpainting’ because the values of one or more tokens at particular locations is set as a condition in advance. That is to say that the method identifies a candidate amino acid sequence that in-paints a seed sequence that defines target amino acid tokens for a subset of the amino acid positions in the sequence. The amino acid tokens for positions that aren’t in the seed sequence are hallucinated. Consider the task of inpainting some sequence x according to a binary mask me [0,1]where m = 0 indicates that the ith element of x should be used as conditioning information (i.e. the amino acid token is determined in advance), and mi = 1 indicates that the ith element of x should be inpainted (i.e. hallucinated by the trained BFN). The conditional generation of the inpainted regions can be viewed as a sequential Monte Carlo problem which can be solved with particle filtering and specifically, through the use of residual resampling of particles (Doucet et al. ‘A tutorial on particle filtering and smoothing: Fifteen years later’ Handbook of nonlinearfilter / ng, 12(656-704):3, 2009). Given a prediction 0(t) = ¢(0,, / ?( / )) at time t, the log-likelihood of the regions being predicted are indicated by the one-hot encoding ex under the receiver distribution as: 1*1 logp(e(x)|0(t),m) = ) - fi(tf\\2 i=l It is noted that mi is zero for locations containing conditioning information. Given p particles at time t with predictions 0i(t) . . . fip(t), each particle is resampled with probability proportional to p(e(x) | fit, m). Combining Sequential Monte-Carlo with the sampling method provides the method shown in Figure 3 in which the variables are: Ni number of iterations of the inference, ¢: operation of the BFN, p: resampling probability, x: sequence of amino acid tokens (provided for masked positions, output for all positions), and m: the mask. Steps 3 and 4 in Figure 3 initialize variables. Similar to before, isotropic noise, zi, with a normal distribution for D amino acid positions and K amino acid token types is prepared for each of p particles. The input also includes x, which includes the seed values of the sequence that are to be inpainted around. Step 4 sets initial predictions, 0, to 0. In steps 5 to 7 the accuracy schedule, P, is set and time, / , is defined in connection with the number of steps N. As before, parameter, Pi, is a hyperparameter. In steps 8 to 12, p samples, yj, are taken from the distribution defined in the equation shown in line 9. In step 10 a softmax distribution is formed around each sample, yj. Accordingly, the method generates multiple samples to generate a set of particles, yj. In step 11, the parameters of each softmax distribution are input to the trained BFN 10 to generate a set of predictions <pj. In step 12, particles are resampled with a probability p to generate a set of resampled particles, Zj. In step 14, an output distribution, L, is formed from the resampled particles, Zj. In step 16 to 18, predictions 0j are sampled from the distribution, L. The predictions are then subject to one-hot encoding for the locations in the sequence to be inpainted (i.e. mk= 1). After this process has been repeated for N+l steps, in step 22 the predicted sequence, including the inpainted values, can be obtained because the accuracy schedule causes the predictions, 0, to converge on a particular inpainted token sequence, x. While a particle filtering technique has been described above, more generally sampling from the output distribution from the BFN may be performed conditionally to enforce the constraint (in this case, the seed amino acid token values). For example, a gradient-based approach, such as those used for diffusion networks, may be applied. Entropy During the sample generation using the techniques described above, the entropy of the input distribution was found to be noticeably higher at a given time t in comparison to that observed during training. It is hypothesized that this phenomenon occurs as the input distribution 9 contains additional entropy from uncertainty in the output distribution. When time t is presented as an additional input to the BFN as described in the section providing a summary of BFNs, this mismatch can be out of distribution for the network ¢, hampering performance. To resolve this, the conventional encoding of time t may be replaced with a encoding of the entropy of each variable, appended to its corresponding input distribution before being passed into the network. Perplexity filtering To generate the 10,000 samples used for ProtBFN de novo generation results described further below, the sampling algorithm described in Figure 2 is repeated 10,000 times. The method of Figure 2 may be used with N = 10,000 e.g. 10,000 sampling steps to generate each sample. When generating samples, it is occasionally possible to generate sequences which are quite unlikely under the model. This is analogous to, when sampling from a Gaussian distribution, occasionally generating scalar values that are quite far from the mean. As the combinatorial space of sequences explodes with sequence length, it is not sufficient to only look at the likelihood of a given sequence under the model, as shorter sequences will necessarily have higher likelihoods. Instead, it is appropriate to look at the amino acid perplexity of a given sequence, which is the (inverse) likelihood of that sequence normalised by its length (measured in amino acids). It was found that amino acid perplexity is strongly correlated with the 'naturalness’ of a given protein sequence (e.g. plDDT score) and that a meaningful shift in the quality of the generated samples was generated by rejecting those sequences with the highest amino acid perplexity. The perplexity of an amino acid sequence may be found by inputting the candidate amino acid sequence to the trained BFN 10 to determine a likelihood of the candidate amino acid sequence and dividing an inverse of the likelihood of each candidate amino acid sequence by the candidate amino acid sequence length to generate a perplexity score. In some implementations, any sequence within a predetermined portion, such as the top 30th percentile, with respect to their perplexity may be discarded. These two processes together rejected approximately 44% of generated samples in the results discussed below. Multimodal modelling In the section above on conditional and unconditional generation of sequences, the generated amino acid sequences were subject to a constraint (the values of the amino acid tokens at the seed positions). In other implementations further or different constraints may be enforced on the amino acid sequences. For example, the trained BFN may be used to generate a candidate amino acid sequence that is subject to one or more constraints, wherein the constraint is enforced using a sequential Monte Carlo algorithm and particle filtering using the trained Bayesian Flow Network. As noted above, the constraint may be enforced by other techniques such as a gradient-based approach. The constraint may be one or more of: a geometrical parameter; sequence length; isoelectric point; an instability index; secondary structure fraction; charge at ph; molecular weight; paired sequence alignment (both protein and rna); paired structure alignment (could be modelled and / or experimental); germline genes; germline gene identities; light chain type (kappa / lambda); complementary-determining regions (cdr) lengths; cdr canonical forms; hydrophobicity; electrostatic patches (type, area, size etc.); charge distribution; antigen sequence; antigen structure (possibly collecting all structure under a single complex structure mode); species; expression; stability (e.g. thermostability from melting curves); and affinity. These constraints may be enforced by applying appropriate particle filters to select particles that enforce the relevant constraint. Learnable schedules Training of the BFN 10 has been described above. As noted, there is a monotonically increasing accuracy schedule, P(t) where: ^(t): [0,1] IR+ d a(t) = — ^(t) In some embodiments an accuracy schedule may be used in which the entropy of the input distribution decreases in roughly linear fashion with: ^(t) = ^t2 where Pi is a parameter of the accuracy schedule that determines the final accuracy. In some embodiments, Pi may be included in the loss function, and the parameter is learned during the training of the Bayesian Flow Network. In some implementations, the variable may be different for each variable (i.e. amino acid position) and the inclusion of the variable in the loss function allows the BFN 10 to optimize the learning schedule on a variable-by-variable basis. In further embodiments, the accuracy schedule may be generalized to a polynomial spline, or more generally to other functions. Parameters of the spline or other function may be learned by during training by the Bayesian Flow Network. Results As described above, ProtBFN was trained to learn the distribution of natural proteins. Although no single metric can definitively assess the ‘naturalness’ of generated protein samples, various statistical and biophysical properties of generated samples can be computed and compared against the expected natural distributions. To this end, Figures 4 to 10 present a selection of such experiments that suggest that ProtBFN not only matches the natural distribution on which it was trained but does so more faithfully than the autoregressive and discrete diffusion baseline models. A direct test of the plausibility of ProtBFN’s generated set of protein sequences is to examine the frequency with which amino acids and oligomers occur as these are strongly related to the robustness of the genetic code. As shown in Figure 4, the frequencies of amino acid token occurrence in samples generated by ProtBFN are well aligned with the frequencies of occurrence in the UniProtCC dataset. Figure 5 is a chart showing that the frequency of oligomer occurrence is also similar between the samples generated by ProtBFN and the UniProtCC dataset. However, whilst these frequency metrics are indicative, they do not alone confirm the coherence of individual samples, therefore per-sample metrics were also computed. Figure 6 is a graph showing that ProtBFN’s generated sequences appear to have a natural-like distribution of lengths when compared to the uniProtCC dataset and compared to samples generated by previous models ProtGPT2 and EvoDiff. This is another straightforward indicator of ProtBFN’s ability to model the underlying distribution. ProtGPT2 is seen to significantly diverge from the UniRef50 distribution under this metric while it is believed that EvoDiff artificially matches the target distribution because sequence length is pre-selected before generation. To review structural features, NetSurfP-3.0 (Hoie et al ‘NetSurfP-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning. Nucleic Acids Research, 50(W 1): W510-W515, 06 2022) was used to annotate each residue in the sequences with structural information. As can be seen from Figure 9, ProtBFN exhibits equivalent annotations to the natural distribution, even when considering how these properties vary as a function of position in the sequence. To then evaluate the overall structural properties, the mean predicted local distance difference test (pLDDT) of ESMFold (Lin al. ‘Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-1130, 2023) was used as a measure of confidence in the predicted structure. The results are shown in Figure 7. As interactions between residues far apart within in a protein sequence are critical in determining overall structure, higher pLDDT scores indicate that ESMFold can identify similar interactions within the generated samples as were seen in the training data. ProtBFN was found to consistently produce sequences which obtain high pLDDT scores, closely matching those of naturally occurring proteins. The sensitivity of this metric to the global coherence of a protein is evidenced by ProtGPT2 and EvoDiff s divergence from their training distribution; even when accounting for the greater number of lower scored proteins in UniRef compared to UniProtCC. To confirm ProtBFN is generating novel proteins rather than memorizing training data, a search was performed for the nearest match for each generated sequence within the UniProtCC training data. Results showed that generated sequences are highly likely to be novel, with 9489 (8851 and 4444) of the generated samples having sequence identity to the nearest match being less than 95% (80% and 50%), respectively. The results are illustrated in Figure 8. Broad coverage of the proteome ensures that a model has learned the diversity of protein sequences, and thus holds the potential for developing a wide array of functional proteins. Having confirmed the naturalness and novelty of the sequences generated by ProtBFN, the proteome coverage provided by these sequences was assessed. To do so, Mmseqs2 (Steinegger et al. ‘Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets.’ Nature biotechnology, 35(11): 1026-1028, 2017) was used to align generated sequences with UniRef50 (Table below), as UniRef50 represents a non-redundant set of protein clusters spanning the known protein diversity. Model Cluster hits Coverage score ProtBFN 69.7% 0.544 ProtGPT2 15.7% 0.095 EvoDiff 2.6% 0.034 69.7% of ProtBFN’s of sequences are found to align (>50% sequence identity) with a known UniRef50 cluster. This is a significantly higher proportion than ProtGPT2 and EvoDiff despite that fact that these models are trained uniformly on the UniRef50 clusters, whereas ProtBFN uses a different training distribution. As the number of sampled sequences (10,000) is far smaller than the number of UniRef50 clusters (~65 million), coverage was measured as the ratio of observed to expected unique clusters hit if drawing 10,000 samples from a model’s training distribution. Empirically it was found that ProtBFN provides substantially better coverage of the protein space than the baseline methods. These results reinforce the idea that ProtBFN creates novel sequences that still cover the functional protein space present in UniProtKB. To visualise the distribution coverage, the mean embeddings from the final layer of ESM-2 were analysed, projecting them into two dimensions using the UMAP algorithm. It was found that ProtBFN closely mirrors the distribution of UniProtCC, in contrast to ProtGPT2, which more closely aligns with UniRef50. Protein function is closely tied to structure. Therefore, to characterise BFN generated protein sequences, which may differ substantially from those observed in nature, the protein sequences were analysed for correspondence with empirically determined protein conformations. The CATH S40 database contains approximately 30,000 non-redundant, experimentally solved, protein domains that provide a broad coverage of known structural diversity. The structural similarity of 2,000 sequences sampled from each model were compared against CATH S40 domains using template modelling (TM) scores. The TM1- and TM2-scores were normalised against the length of the generated protein and the length of the CATH S40 domain, respectively, with a TM score above 0.5 generally recognised as indicative of the same fold. To avoid matching only fragments of CATH domains to generated proteins, or vice versa, a positive match requires both TM1 and TM2 scores to exceed 0.5. ProtBFN achieves a hit-rate of 65.7%, surpassing ProtGPT2 (25.3%) and EvoDiff (12.0%). Furthermore, it was found that ProtBFN generates longer sequences that fold into known domains more frequently. However, despite high structural correspondence, ProtBFN’s sequences exhibit low sequence similarity to their CATH S40 targets. This capability to produce recognisable globular folds with novel sequences is a prerequisite for the use of ProtBFN for rational protein design and suggests a meaningful capability to go beyond training data and explore the dark proteome. ProtBFN was also found to be capable of generating a structurally diverse array of functionally complex protein domains. The model spans the breadth of classes catalogued in CATH, including alpha helical, beta sheet, alpha-beta, and irregular domains. ProtBFN also effectively models various functional types, such as transmembrane proteins, including porins and transporters, along with enzymes. Additionally, the generated globular proteins span small and large structural domains, irrespective of CATH class. Merizo (Lau et al. ‘A rapid and accurate protein domain segmentation method using invariant point attention.’ Nature Communications, 14(1):8445, 2023), a tool for domain segmentation, was used to understand the domain distributions of ProtBFN samples. Over half of the tested samples (50.2%) have two domains, followed by single domain sequences (33.6%), and three or more domains with (14.4%). The multi domain constructs typically exhibit meaningful interactions between domains. This highlights that ProtBFN can generate proteins with globally coherent interactions, even when the interacting residues are separate in sequence space and belong to different folds. More broadly, this analysis demonstrates the structural and functional depth of ProtBFN’s learned distribution, highlighting the potential applicability of BFN generation across a diverse range of biological and biotechnological applications. In connection with AbBFN, the design of antibodies, a class of proteins that identifies and neutralises pathogens, is central to protein engineering applications such as monoclonal antibody-based cancer therapies. To adapt ProtBFN for specialised domains, we developed AbBFN, an antibody-specific model, by finetuning on variable heavy (VH) sequences from the Observed Antibody Space (OAS) database. Two validation sets are considered: 20,000 VH samples uniformly drawn from OAS and heavy sequences from the SAbDab benchmark. To ensure no data leakage, the training set is filtered of sequences similar to those in either validation set, leaving 195 million training examples. Analogous to the results of investigating ProtBFN, it was found that AbBFN accurately captures the natural distribution of VH sequences. Antibody VH sequences, generated by inpainting using AbBFN 10, were segmented into three complimentary determining regions (CDRs) - which primarily determine the binding specificity - and four framework (FR) regions - which provide a scaffold for the CDRs as well as providing important contributions to functions such as immunogenicity. A common task for antibody sequence models is to conditionally generate subsets of these regions based on a given partial sequence (‘inpainting’), with performance measured in terms of the amino acid recovery (AAR) rate. Recalling that the BFN 10 can sample from the conditional distribution of any subset of variables (i.e. amino acids) despite being trained only for the unconditional generation - antibody inpainting represents a natural task with which to assess these zero-shot conditional generation capabilities. AbBFN is compared to leading antibody-specific language models AntiBERTy (Ruffolo et al. ‘Deciphering antibody affinity maturation with language models and weakly supervised learning.’ arXiv preprint arXiv:2112.07782, 2021) and AbLang2 (Olsen et al. ‘Addressing the antibody germline bias and its effect on language models for improved antibody design’ bioRxiv, pages 2024-02, 2024), which were trained for BERT-style conditional generation. As both of these methods were trained on the entire OAS, they have likely been exposed to both our validation datasets, potentially causing data leakage. Despite this, AbBFN is able to recover individual FR and CDR regions as well as these specialist models on the OAS validation set (see Figure 10), whilst demonstrating AAR rates consistent with the known increased variability of CDR, and in particular CDR-H3, regions. Notably, AbBFN significantly outperforms the baseline models at predicting all FR regions simultaneously. This may be attributable to larger masked region being out-of-distribution for BERT-style methods, which does not afflict the more flexible generation of a BFN. On the SAbDab benchmark (see Figure 11 for results), AbBFN retains strong performance - significantly outperforming baseline methods which have not trained on this data, including those which can condition the generation on additional structural data. The increased gap between AbBFN and the language model methods may be attributed to the removal of similar sequences to the SAbDab benchmark from our training data. To address this, the performance of AbBFN+ has been included, which AbBFN+ model underwent rapid fine-tuning (1,000 adaptation steps) on the nine training folds for each test fold. AbBFN+ outperforms all methods on CDR-H1 and CDR-H2 and significantly closes the gap on CDR-H3. The inventors expect similar success when BFNs are applied to RNA and DNA datasets. Uses By analysing amino acid sequences using a BFN of the type described above, it is possible to use embedded knowledge to identify amino acids in the amino acid sequence being analysed which might be good candidates for modification. In other words, by inputting the amino acid token identities associated with a sequence of a known amino acid sequence to the trained BFN, a measure of the likelihood or natural fit of the amino acid may be obtained based on the output distribution. In this way, regions of the amino acid sequence that are less naturally viable could be identified and varied. Further, by inpainting regions of a known amino acid sequence, such as a sequence belonging to a pathogen, variants of the pathogen that may be likely to occur in nature may be identified. For example, using the techniques described, portions of the amino acid sequence may be regenerated to meet a particular constraint. For example, the amino acid sequence may be selected to optimize for at least one of: solubility, weight, length, charge, and hydrophobicity. The optimization will depend upon the context in which the amino acid sequence is being generated. In an example, a researcher may develop a new protein comprising a plurality of amino acids which has been designed to bond to a target further protein using techniques known in the art, but not described in detail herein. By evaluating the protein developed by the researcher using the techniques described herein, it may be possible to determine whether the protein is physiologically viable. For example, by applying a BFN which is trained on real life proteins which are stable and naturally occurring, it may be possible to identify whether any of the amino acids in the protein developed by the researchers may cause the protein to be unstable and / or should be modified or reevaluated. The techniques above may be used in a wide range of applications in proteomics such as, but not limited to, amino acid sequence design, epitope design, de novo amino acid sequence generation, nanoparticle design, protein engineering, affinity enhancement, and assessment of reactivity. As explained further above, sequences can be generated with affinity or reactivity criteria using the techniques described above in connection with multimodal modelling. Another use case may involve evaluating a protein-protein interactions, such as evaluating a protein to determine which of the amino acids in the protein can be modified to develop a drug which has a greater binding affinity with another protein. In this case, the method may provide one or more probability values which can indicate which of the amino acids in the protein sequence are promising candidates for optimization whilst still allowing the synthesized protein to be physiologically viable. In some examples, variants of a disease associated pathogen may be generated by inpainting portions of the protein sequence as described herein. Protein-protein interactions may also be important in the development of therapeutics for treatment of cancer. The methods described may be applied to the development of therapeutics for treatment of cancer. An immunogen refers to a molecule or biological entity capable of eliciting an immune response by an organism’s immune system. In an embodiment, the immunogen is a disease associated antigen which gets presented on the organism’s antigen presenting cells in association with MHC molecules. In an embodiment, the immunogen is a pathogen or immunogenic material from a pathogen. A variant immunogen herein may also refer to an entire disease-causing pathogen, such as a variant strain or SARS-CoV-2, or an immunogenic sequence from such pathogen such as an antigen or epitope that is capable of being presented on the organism’s antigen presenting cells in association with MHC molecules. In one example, the immunogen is an immunogenic protein from the pathogen, such as the coronavirus spike protein. The immunogens identified by the methods herein by inpainting may have several applications in disease surveillance, diagnosis, prevention and therapy. If the immunogens are from pathogens such as influenza, Ebola or coronaviruses then generation of variants to identify possible variants of concerns can have huge utility in surveillance of variants. Hence, the invention also relates to methods of surveillance utilising variants identified or characterised as variants of concern. Immunogenic compositions, such as vaccines, comprising such immunogens, methods of making such compositions and methods of prevention or treatment using such compositions all form further aspects of the present invention. Each immunogen in the composition may be a peptide sequence. The immunogen(s) in such an immunogenic composition may be nucleic acid sequences such as mRNA sequences. The nucleic acids can be delivered complexed to cationic compounds, such as cationic lipids. The composition may include one, two or several (for example 3 to 10) immunogens selected or identified according to the methods disclosed herein. The immunogen(s) in the immunogenic compositions can also be included in viral vectorbased vaccine platforms, such as vaccinia, fowlpox, self-replicating alphavirus, marabavirus, adenovirus (See, e.g., Tatsis et al, “Adenoviruses”, 2004). The effective amount and method of administration of a particular composition can vary based on the individual patient and other factors evident to one skilled in the art. In some embodiments, the immunogenic composition can be formulated or administered with one or more pharmaceutically acceptable excipients such as stabilising agents, encapsulating agents and buffers. In some embodiments, the immunogenic composition may contain an amount of an adjuvant such as alum and MPL. In some embodiments, the immunogen is formulated with liposomes or microparticles. A variety of methods are available for preparing liposomes, as described in, e.g., Szoka et al, “Comparative properties and methods of preparation of lipid vesicles (liposomes)”, 1980. The immunogens generated by any elements within any aspect of the invention can also be used in developing diagnostics and therapeutics. Therefore, these methods can be methods for selection or identification of immunogens for the purpose of developing therapeutics, such as antibodies, or for the purpose of preparing diagnostic assays instead of for the purpose of preparing an immunogenic composition. The techniques described herein may also have application in enzyme optimization, such as to develop better yeast or other organisms that produce desirable compounds. Further, the techniques may find application in the development of genetically modified organisms (GMO) such a higher yielding or more hardy crops. Computer hardware The BFNs, training and sample generation methods described above may implemented using an information processing apparatus. The information processing apparatus, in some cases, may be implemented as a cloud service or as one or more separate standalone server. Figure 12 is a schematic diagram of components of an example information processing apparatus suitable for use in the methods described above. The diagram is illustrative and different hardware configurations for information processing apparatus are possible as is well known in the art. The information processing apparatus includes an I / O interface 110, such a USB port, Thunderbolt port, etc. to which an additional device, such as a storage device, could be connected. The information processing apparatus comprises a processor 111 (such as a CPU, GPU or NPU), a storage in the form of memory 112, a network module 113, a display 114, and a user interface 115. The network module may allow the information processing apparatus to communicate over a network such as a Wi-Fi network, a mobile telecommunications network, a local area network etc. The user interface may include components such as a keyboard, mouse, camera, etc. The components of the information processing apparatus may communicate with each other over a bus 116. Further components may be provided but are not shown or described. Any of the steps of the methods described above may be performed by computer-readable instructions of one or more programs stored in a storage and executed by a processor on one or more information processing apparatuses. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

1. A computer-implemented method that generates a sample with a trained Bayesian Flow Network for development of an amino acid product, the sample being an amino acid sequence comprising the identity of an amino acid token at each of a plurality of sequence positions, the method comprising for a plurality of iterations:sampling a vector, y, from a first distribution that depends on a prediction, an accuracy schedule that increases accuracy as the iterations progress, and isotropic noise;generating a second distribution around the sampled value, y; andinputting the parameters of the second distribution to the trained Bayesian Flow Network to obtain an updated prediction;wherein after the plurality of iterations, the predictions identify the generated sample amino acid sequence.

2. A method according to claim 1 that identifies a sample amino acid sequence that in-paints a seed sequence that defines target amino acid tokens for a subset of the amino acid positions in the sequence, wherein the method comprises:sampling a plurality of vectors, yj, from the first distribution to generate a set of particles;generating a plurality of second distributions, a respective second distribution being formed around each sampled particle, yj;applying the neural network to parameters of each second distribution determined from the particles to generate a first set of predictions;performing particle filtering on the first set of predictions; andobtaining an updated prediction by generating a third distribution from the filtered set of predictions and sampling the third distribution to generate the updated set of predictions.

3. A method according to claim 1 or claim 2, wherein the parameters input to the Bayesian Flow Network include one or more entropy values representing an entropy of the input distribution.

4. A method according to any preceding claim, comprising:repeating the method to identify a plurality of candidate amino acid sequences, inputting each candidate amino acid sequence to the Bayesian Flow Network to determine a likelihood of the candidate amino acid sequence;dividing an inverse of the determined likelihood of each candidate amino acid sequence by the candidate amino acid sequence length to generate a perplexity score; andidentifying one or more candidate amino acid sequences based on the generated perplexity scores.

5. A method according to any preceding claim further comprising using the Bayesian FlowNetwork to generate a sample amino acid sequence that is subject to one or more constraints, wherein the constraint is enforced by sampling conditionally from the output distribution of the Bayesian Flow Network , and the constraint comprises one or more of:a geometrical parameter;sequence length;isoelectric point;an instability index;secondary structure fraction;charge at ph;molecular weight;paired sequence alignment (both protein and rna);paired structure alignment (could be modelled and / or experimental);germline genes;germline gene identities;light chain type (kappa / lambda);complementary-determining regions (cdr) lengths;cdr canonical forms;hydrophobicity;electrostatic patches (type, area, size etc.);charge distribution;antigen sequence;antigen structure (possibly collecting all structure under a single complex structure mode);species;expression;stability (e.g. thermostability from melting curves); and affinity.

6. A method according to any preceding claim, wherein the Bayesian Flow Network is comprises at least one of an attention-based network, a transformer-based language model, a convolutional neural network, and a graph neural network.7 . A method according to any preceding claim wherein the first distribution is given by y <- -1)+ yfKp(t)z, where 0( / ) is the accuracy schedule, A’ is anumber of types of available amino acid token, 0 is the prediction, and z is the isotropic noise.

8. A method according to any preceding claim, wherein the accuracy schedule determines the accuracy based on a time parameter, t, that identifies the position of the current iteration within the plurality of iterations.

9. A method according to claim 8, wherein the accuracy schedule is given by ^(t) «- ^t2, where is a free parameter.

10. A method according to any preceding claim wherein the second distribution is a normal distribution, and a parameter of the normal distribution is determined based on the accuracy schedule.

11. A method according to any preceding claim, further comprising training the Bayesian Flow Network, wherein during training the method comprises:setting initial input distribution parameters that define an input distribution for each category;and for a plurality of iterations:determining a sender distribution that defines a distribution for each category based on a data sample and an accuracy parameter determined for the current iteration;generating a sample from the sender distribution;using Bayesian inference to generate a new input distribution based on the sample and an existing input distribution;inputting parameters of the new input distribution into the Bayesian Flow Network to generate output parameters that define an output distribution for each category;generating a receiver distribution based on the output distribution parameters and the accuracy parameter for the next iteration;determining the value of a loss function based on a difference between the sender distribution and the receiver distribution; andupdating parameters of the Bayesian Flow Network based on the output of the loss function.

12. A method according to claim 11, wherein:the accuracy parameter is parameterised in an accuracy schedule that defines the accuracy parameter at each time,a parameter of the accuracy schedule is included in the loss function, andthe parameter is learned during the training of the Bayesian Flow Network.

13. A method according to claim 12, wherein a separate accuracy schedule is used for each the plurality of sequence positions, and a parameter of the accuracy schedule for each sequence position is learned during the training of the Bayesian Flow Network.

14. A program that, when executed by an information processing apparatus, causes the information processing apparatus to perform a method according to any of claims 1 to 13.

15. An information processing apparatus comprising a processor and a storage, wherein the storage comprises instructions that when executed causes the information processing apparatus to perform a method that generates a sample with a trained Bayesian Flow Network for development of an amino acid product, the sample being 5 an amino acid sequence comprising the identity of an amino acid token at each of a plurality of sequence positions, the method comprising for a plurality of iterations:sampling a vector, y, from a first distribution that depends on a prediction, an accuracy schedule that increases accuracy as the iterations progress, and isotropic noise;generating a second distribution around the sampled value, y; and10 inputting the parameters of the second distribution to the trained Bayesian FlowNetwork to obtain an updated prediction;wherein after the plurality of iterations, the predictions identify the generated sample amino acid sequence.

Images