NeoBert model and application thereof in identifying tumor neoantigen

Abstract

The invention provides a NeoBert model and application thereof in identifying tumor neoantigen, the NeoBert model comprises a sequence input device, an embedding block, an encoder block and a prediction block, the accuracy rate of the NeoBert model trained by the invention in the aspect of polypeptide-HLA binding prediction is 93% and is obviously superior to that of NetMHCpan, the accuracy rate of the NetMHCpan is 75%, and a new tumor neoantigen prediction model is provided.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical technology, and in particular relates to a NeoBert model and an application thereof in identifying neotumor antigens. Background Art

[0002] HLA, or human leukocyte antigen, is an expression product of the human major histocompatibility complex (MHC). It has more than 200 encoding genes, all located on chromosome 6. Tumor-specific antigens (TSAs), also known as tumor neoantigens, are short peptide antigens displayed by the major histocompatibility complex on the surface of tumor cells. These unique TSAs are not present in normal tissues, but can bind to HLA in abnormal tissues, thereby becoming recognizable to T cells as foreign antigens with the ability to trigger tumor-specific T cell responses, making them promising candidates for cancer vaccines. The current traditional methods for identifying these classic neoantigens rely heavily on whole genome / whole exome sequencing data, and several widely studied classic neoantigen databases, including CEDAR, CAPED (https: / / caped.icp.ucl.ac.be), TANTIGEN, NEPdb, dbPepNeo, NEOdb, and TSNAdb, have been widely explored for their usefulness in cancer immunotherapy.

[0003] Although the coding regions of the genome only account for 40% of the entire genome, and the remaining 60% are non-coding regions, there is increasing evidence that non-classical mechanisms, including transcriptional variants (e.g., alternative splicing, intron retention, cryptic introns, and transposon-derived chimeric transcripts) and proteasomal cryptic peptides (cis / trans proteasomal processing), contribute to the formation of a larger repertoire of non-classical neoantigens. However, the collection and curation of non-classical neoantigen data remains limited. For neoantigens derived from non-coding upstream open reading frames (uORFs) identified by RNA-seq, there is currently only one database, SPENCER, which contains 2806 mass spectrometry data from 55 studies. However, this dataset is not derived from immunoproteomics, which predicts non-coding RNA (ncRNA) translation and uses a sliding window algorithm to split each non-coding peptide into 8-14-mer fragments. Regarding protein proteasomal processed neoantigens, the Proteomedb database contains such entities. Nevertheless, it requires in vitro synthesis of 80 peptides, followed by proteasomal digestion and detection using various mass spectrometry methods. The peptides were detected using a developed method called invitroSPI, so these neoantigens were not present in the tumor samples.

[0004] Detection and sequencing of HLA-bound peptides by liquid chromatography tandem mass spectrometry (LC-MS / MS) offers a unique advantage of directly learning about how peptides are processed and presented endogenously by cells. We collected and organized a large dataset of non-redundant non-classical neoantigen peptides, including 35,574 neoantigen-HLA pairs from different types of RNA and protein-based data from 14 publications. This comprehensive compilation allowed us to compare their antigenic properties with peptides derived from somatic mutations (represented by CEDAR), tumor-associated peptides (represented by HLAthena), and neoantigens derived from infectious diseases (represented by IEDB), including length, motifs, entropy, and hydrophobicity.

[0005] These differences prompted us to train prediction methods for peptide-HLA (pHLA) binding. Currently, prediction methods for pHLA binding can be roughly divided into four categories: structure-based methods, scoring function-based methods, machine learning-based methods, and comprehensive methods that combine multiple factors. Structure-based methods analyze the binding structure of HLA and peptides, while scoring function-based techniques evaluate peptides based on sequence-based properties. Machine learning-based methods extract features from peptides or HLAs and train models for binding prediction. Deep learning methods, including CNNs, RNNs, and attention mechanisms, have good performance due to their good accuracy and efficiency. Recently, a converter-based model was developed that can be used to predict peptide-HLA class I binding and mutant peptides for vaccine design. However, it is only trained on classical peptides and cannot fully explain the complex interactions between HLA and peptide sequences. Therefore, we integrated classical and non-classical neoantigen data to train and obtain the NeoBert model of the present invention. Summary of the invention

[0006] After the present invention collects data, 70% of the data is used to train the NeoBert model, and the remaining 30% of the data is used for verification and comparison of the NeoBert model. The model determines whether the peptide is a tumor neoantigen by determining whether the peptide can bind to HLA. Based on this, the present invention is completed.

[0007] In a first aspect, the present invention provides a NeoBert model, which consists of four consecutive components, including:

[0008] Sequence Input: Masked HLA and peptide sequences are packaged into a sentence (i.e.<masked HLA,maskedpeptide> ) for input;

[0009] Embedding block: adds position embeddings to amino acid (i.e. token) embeddings and fragment embeddings to generate sequence embeddings;

[0010] Encoder block: contains 12 BertLayers, each of which contains a masked multi-head self-attention mechanism and a feature optimization block for learning representations;

[0011] Prediction block: The aggregated sequence representation is used to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position.

[0012] Furthermore, in the sequence input device, each pair of HLA and peptide sequence is represented as a sentence, for example, X=([CLS], HLA sequence, [SEP], peptide sequence).

[0013] Furthermore, in the sequence input device, the final hidden feature of the first tag ([CLS]) is used as the sequence representation for classification, while the ([SEP]) tag distinguishes the two sequences, and the randomly masked amino acids of the two sequences are used to train the deep bidirectional representation.

[0014] Furthermore, in the sequence input device, the input peptide and HLA sequences are padded to a length of 128 to accommodate the variable input length.

[0015] Furthermore, in the embedding block, for each token, the learned embedding (E A or E B ∈R H ) to indicate whether it belongs to HLA or peptide, so as to facilitate the segmentation of the two sequences, where H is the embedding dimension, and WordPiece embedding is used to encode the amino acid embedding (E Ai or E Bj ∈R H ).

[0016] Furthermore, in the embedding block, position embedding (E i ∈R H ) to encode the position (X) of the amino acid in the sentence.

[0017] Further, in the embedding block, the input representation in tokens (or tokens in HLA) is created by adding its token, segment, and position embeddings as follows:

[0018] E i =E A +E Arn +E i

[0019] Furthermore, in the encoder block, multi-head self-attention technology is used to combine the bidirectional cross attention between two sequences by concatenating the sequences.

[0020] Further, in the encoder block, the mechanism includes mapping the query Q to a set of key-value (KV) pairs and obtaining an output, wherein the KV pairs store sequence elements in memory.

[0021] Furthermore, in the encoder block, the attention score is based on the correlation or similarity between Q and K, indicating the importance of information, and a higher attention score (ie, V) indicates a stronger focus on the corresponding information.

[0022] Furthermore, in the encoder block, the feature optimization module combines the fully connected layer with LayerNorm and Dropout to learn the enhanced representation (H i ) for the ith token.

[0023] Furthermore, in the encoder block, the connection sequence of HLA and peptide with a length less than 128 is filled with 0 to 128.

[0024] Furthermore, in the prediction block, two tasks were used to fine-tune the NeoBert model: HLA peptide binding prediction and masked amino acid prediction.

[0025] Furthermore, the HLA-peptide binding prediction is to input the aggregate sequence representation of each HLA-peptide pair into a sigmoid layer to predict the signature:

[0026] Y′=Sigmoid(W (2) (Tanh(W (1) H 0 +b (1) )))

[0027] Among them, W (1) , W (2) , and b (1) is a learnable parameter, H 0 Denotes the representation of special tokens [CLS], and the corresponding loss function is summarized as follows:

[0028]

[0029] Where N represents the set of HLA peptide pairs containing positive binding data and negative data. The generated data are similar to those in previous studies.

[34] .

[0030] Going one step further, a masked amino acid model predicts the positional probabilities of 20 different amino acids in the sequence based on their sequence representation:

[0031] P i =Softmax(WH i +b)

[0032] Among them, W and b are learnable parameters, H iRepresents the representation of token. Its loss function is summarized as follows:

[0033]

[0034] Among them, Mask represents the set of positions of all masked amino acids, Pos represents the set of positive binding pairs, and the total loss function of the NeoBert model is summarized as follows:

[0035]

[0036] In a second aspect, the present invention provides an application of the NeoBert model as described in the first aspect in identifying tumor neoantigens, wherein the model determines whether a peptide is a tumor neoantigen by determining whether the peptide can bind to HLA.

[0037] Furthermore, if the peptide binds to HLA, the peptide is predicted to be a tumor neoantigen.

[0038] In a third aspect, a method for identifying tumor neoantigens comprises the following steps:

[0039] 1) Input peptide and HLA sequences in the sequence inputter. Masked HLA and peptide sequences are packaged into one sentence (i.e., <masked HLA, masked peptide>) for input;

[0040] 2) The embedding block adds the position embedding to the amino acid (i.e., token) embedding and fragment embedding to generate a sequence embedding;

[0041] 3) The encoder block uses multi-head self-attention technology to combine the bidirectional cross attention between two sequences by concatenating the sequences;

[0042] 4) The prediction block uses the aggregated sequence representation to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position;

[0043] 5) Determine whether the peptide can bind to HLA.

[0044] Furthermore, in the sequence input device, each pair of HLA and peptide sequence is represented as a sentence, for example, X=([CLS], HLA sequence, [SEP], peptide sequence).

[0045] Furthermore, in the sequence input device, the final hidden feature of the first tag ([CLS]) is used as the sequence representation for classification, while the ([SEP]) tag distinguishes the two sequences, and the randomly masked amino acids of the two sequences are used to train the deep bidirectional representation.

[0046] Furthermore, in the sequence input device, the input peptide and HLA sequences are padded to a length of 128 to accommodate the variable input length.

[0047] Furthermore, in the embedding block, for each token, the learned embedding (E A or E B ∈R H ) to indicate whether it belongs to HLA or peptide, so as to facilitate the segmentation of the two sequences, where H is the embedding dimension, and WordPiece embedding is used to encode the amino acid embedding (E Ai or E Bj ∈R H ).

[0048] Furthermore, in the embedding block, position embedding (E i ∈R H ) to encode the position (X) of the amino acid in the sentence.

[0049] Further, in the embedding block, the input representation in tokens (or tokens in HLA) is created by adding its token, segment, and position embeddings as follows:

[0050] E i =E A +E Am +E i

[0051] Furthermore, in the encoder block, multi-head self-attention technology is used to combine the bidirectional cross attention between two sequences by concatenating the sequences.

[0052] Further, in the encoder block, the mechanism includes mapping the query Q to a set of key-value (KV) pairs and obtaining an output, wherein the KV pairs store sequence elements in memory.

[0053] Furthermore, in the encoder block, the attention score is based on the correlation or similarity between Q and K, indicating the importance of information, and a higher attention score (ie, V) indicates a stronger focus on the corresponding information.

[0054] Furthermore, in the encoder block, the feature optimization module combines the fully connected layer with LayerNorm and Dropout to learn the enhanced representation (H i ) for the ith token.

[0055] Furthermore, in the encoder block, the connection sequence of HLA and peptide with a length less than 128 is filled with 0 to 128.

[0056] Furthermore, in the prediction block, two tasks were used to fine-tune the NeoBert model: HLA peptide binding prediction and masked amino acid prediction.

[0057] Furthermore, the HLA-peptide binding prediction is to input the aggregate sequence representation of each HLA-peptide pair into a sigmoid layer to predict the signature:

[0058] Y′=Sigmoid(W (2) (Tanh(W (1) H 0 +b (1) )))

[0059] Among them, W (1) ,W( 2) , and b (1) is a learnable parameter, H 0 Denotes the representation of special tokens [CLS], and the corresponding loss function is summarized as follows:

[0060]

[0061] Where N represents the set of HLA peptide pairs containing positive binding data and negative data. The generated data are similar to those in previous studies.

[34] .

[0062] Going one step further, a masked amino acid model predicts the positional probabilities of 20 different amino acids in the sequence based on their sequence representation:

[0063] P i =Softmax(WH i +b)

[0064] Among them, W and b are learnable parameters, H i Represents the representation of token. Its loss function is summarized as follows:

[0065]

[0066] Among them, Mask represents the set of positions of all masked amino acids, Pos represents the set of positive binding pairs, and the total loss function of the NeoBert model is summarized as follows:

[0067]

[0068] Furthermore, if the peptide binds to HLA, it is predicted that the peptide is a tumor neoantigen; if the peptide does not bind to HLA, it is predicted that the peptide is not a tumor neoantigen.

[0069] In a fourth aspect, the present invention provides a system for identifying tumor neoantigens, the system comprising:

[0070] Sequence input module, used to input peptide and HLA sequences, masked HLA and peptide sequences are packaged into a sentence;

[0071] Sequence processing module, used to process input sequences;

[0072] The binding probability prediction module uses the aggregated sequence representation to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position;

[0073] The result output module determines whether the peptide is a tumor neoantigen by judging whether the peptide can bind to HLA.

[0074] Furthermore, in the sequence processing module, the embedding block adds position embedding to the amino acid (i.e., token) embedding and fragment embedding to generate sequence embedding, and the encoder block utilizes the multi-head self-attention technique to effectively combine the bidirectional cross attention between two sequences by concatenating the sequences.

[0075] Furthermore, in the binding probability prediction module, the linear binary classifier uses the aggregated sequence representation to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position.

[0076] Furthermore, if the peptide binds to HLA, the peptide is predicted to be a tumor neoantigen.

[0077] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the third aspect.

[0078] Beneficial Effects

[0079] The NeoBert model trained by the present invention has an accuracy rate of 93% in predicting peptide-HLA binding, which is significantly better than NetMHCpan, while the accuracy rate of NetMHCpan is 75%, providing a new prediction model for tumor neoantigens. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is the comparison result of the prediction performance of the NeoBert model of the present invention and NetMHCpan. DETAILED DESCRIPTION

[0081] The specific embodiments of the present invention are further described below. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the embodiments described below can be combined with each other as long as they do not conflict with each other.

[0082] The experimental methods in the following examples are conventional methods unless otherwise specified, and the experimental materials used in the following examples are commercially available unless otherwise specified.

[0083] Example 1 Training of NeoBert Model

[0084] 1. Dataset Collection

[0085] 1.1 Collection of RNA-derived neoantigen data

[0086] From existing literature (Ouspenskaia, T. et al. Unannotated proteins expand the MHC-I-restricted immunopeptidome in cancer. Nat Biotechnol 40, 209-217, doi:10.1038 / s41587-021-01021-3 (2022); Chong, C. et al. Integrated proteogenomic deepsequencing and analytics accurately identify non-canonical peptides in tumorimmunopeptidomes. Nat Commun 11, 1293, 692doi:10.1038 / s41467-020-14968-9 (2020); Smart, A.C. et al. Intron retention is a source of neoepitopes in cancer. NatBiotechnol 36, 1056-1058, doi:10.1038 / nbt.4239 (2018); Kong, Y. et al. Transposableelement expression in tumors is associated with immune infiltration andincreased antigenicity. Nat Commun 10, 5228, doi:10.1038 / s41467-019-13035-2 (2019); Schwarz, S. et al. T cells of colorectal cancer patients'stimulated byneoantigenic and cryptic peptides better recognize autologous tumor cells. JImmunother Cancer 10, doi:10.1136 / jitc-2022-005651 (2022); Zhao, Q.etal. Proteogenomics Uncovers a Vast Repertoire of Shared Tumor-SpecificAntigens in Ovarian Cancer. Cancer Immunol Res 8, 544-555, doi:10.1158 / 2326-6066.CIR-19-0541(2020); Laumont,CMet al.Noncoding regions are the main source of targetable tumor-specific antigens.Sci Transl Med 10, doi:10.1126 / scitranslmed.aau5516(2018); Erhard, F., Dolken, L., Schilling, B. & Schlosser, A. Identification of the Cryptic HLA-I Immunopeptidome. Cancer Immunol Res 8,1018-1026,doi:10.1158 / 2326-6066.CIR-19-0886(2020); Laumont,CMet al.Globalproteogenomic analysis of human MHC class I-associated peptides derived from non-canonical reading frames.Nat Commun 7, 10238, doi:10.1038 / ncomms10238(2016).) and annotated the following information for each entry: "PMID", "Cell.line", "Peptide", "Length", "Genome", "Chromosome", "Start", "Stop", "Loci", "Transcript_strand", "Overlapping genomic feature", "HLA_locus", "HLA_allele", "Ensembl_gene_id", and "Protein_name". .

[0087] 1.2 Collection of protein-derived neoantigen data

[0088] Data from 4 previous studies were collected (Faridi, P. et al. A subset of HLA-I peptides are not genomically templated: Evidence for cis-and trans-spliced ​​peptideligands. Sci Immunol 3, doi: 10.1126 / sciimmunol.aar3947 (2018); Liepe, J. et al. A large fraction of HLA class I ligands are proteasome-generated splicedpeptides.Science 354.354-358.doi:10.1126 / science.aaf4384(2016);71120 Rolfs, Z., Solntsev, SK, Shortreed, MR, Frey, BL & Smith, LMGlobal Identification of Post-Translationally Spliced ​​Peptides with Neo-Fusion.J Proteome Res, 349-358, doi: 10.1021 / acs.jproteome.8b00651(2019); Faridi, P. et al. Spliced ​​Peptides and Cytokine-Driven Changes in the Immunopeptidome of Melanoma. Cancer Immunol Res8, 1322-1334, doi: 10.1158 / 2326-6066.CIR-19-0894(2020).).

[0089] Annotation data were downloaded from UniProtKB / Swiss-Prot (https: / / www.uniprot.org / ) to annotate proteins and obtain their corresponding genomic coordinates. For the two cis datasets, “PMID: 27846572” and “PMID: 30315122”, the Parentprotein(s) format was standardized to “EntryEntry.namelocation” and protein annotations were added under “GeneUniprot”.

[0090] For "PMID: 32938616", the names of peptides and proteins were extracted. For "PMID 30346791", there were "Full Sequence" and "Protein Name" columns containing multiple peptides and proteins connected by "or". These entries were split to list each sequence and protein name relationship separately, which were then matched with the protein, gene name and Ensemble from the UniProt annotation file. For unmatched entries, the FASTA files of the corresponding proteins were downloaded from UniProtKB.

[0091] 1.3 IEDB, CEDAR, HLAthena data

[0092] IEDB data can be downloaded at https: / / www.iedb.org. HLAthena data can be downloaded at https: / / hlathena.tools. CEDAR data can be downloaded at https: / / cedar.iedb.org.

[0093] 2. NeoBert was trained on 70% of HLA-peptides from five different datasets

[0094] NeoBert consists of four sequential components: (i) masked HLA and peptide sequences are packaged into a sentence (i.e., <masked HLA, masked peptide>) for input; (ii) the embedding block adds the position embedding to the amino acid (i.e., token) embedding and fragment embedding to generate a sequence embedding; (iii) the encoder block contains 12 BertLayers, each of which contains a masked multi-head self-attention mechanism and a feature optimization block for learning representations; (iv) a linear binary classifier uses the aggregated sequence representation to predict binding probabilities, while the masked amino acid model predicts masked amino acids based on the representation of each corresponding position. Specifically:

[0095] (i) Input sequence

[0096] Each pair of HLA and peptide sequence is represented as a sentence, for example X = ([CLS], HLA sequence, [SEP], peptide sequence). The final hidden features of the first token ([CLS]) are used as the sequence representation for classification, while the ([SEP]) token distinguishes the two sequences. Random masked amino acids of the two sequences are used to train the deep bidirectional representation. The input peptide and HLA sequences are padded to a length of 128 to accommodate the variable input length.

[0097] (ii) Sequence Embedding

[0098] For each token, add the learned embedding (E Aor E B ∈R H ) to indicate whether it belongs to HLA or peptide, which facilitates the segmentation of the two sequences, where H is the embedding dimension. WordPiece embedding is used to encode amino acid embeddings (E Ai or E Bj ∈R H In addition, position embedding (E i ∈R H ) to encode the position (X) of the amino acid in the sentence. Therefore, the input representation in tokens (or tokens in HLA) is created by adding its token, segment, and position embeddings as follows:

[0099] E i =E A +E Arn +E i

[0100] (ii) Encoder Block

[0101] The multi-head self-attention technique is used to effectively combine the bidirectional cross-attention between two sequences by concatenating the sequences. The mechanism involves mapping the query Q to a set of key-value (KV) pairs and obtaining the output, where the KV pairs store the sequence elements in memory. The attention score is based on the correlation or similarity between Q and K, indicating the importance of the information. A higher attention score (i.e., V) indicates a stronger focus on the corresponding information. In addition, the feature optimization module combines a fully connected layer with LayerNorm and Dropout to learn an enhanced representation (H i ) for the i-th token. For the concatenated sequences of HLA and peptide with length less than 128, we pad them with 0 to 128. Therefore, non-amino acid features play no role in calculating attention.

[0102] (iv) Prediction Block

[0103] Two tasks were employed to fine-tune the NeoBert model: HLA-peptide binding prediction and masked amino acid prediction. Specifically, we input the aggregated sequence representation of each HLA-peptide pair into a sigmoid layer to predict the label:

[0104] Y′=Sigmoid(W (2) (Tanh(W (1) H 0 +b (1) )))

[0105] Among them, W (1) , W (2) , and b (1) is a learnable parameter, H 0represents the representation of special tokens [CLS]. The corresponding loss function is summarized as follows:

[0106]

[0107] Where N represents the set of HLA peptide pairs containing positive binding data and negative data. The generated data are similar to those in previous studies.

[34] At the same time, a masked amino acid model predicts the positional probabilities of 20 different amino acids in the sequence based on their sequence representation:

[0108] P i =Softmax(WH i +b)

[0109] Among them, W and b are learnable parameters, H i Represents the representation of token. Its loss function is summarized as follows:

[0110]

[0111] Among them, Mask represents the set of positions of all masked amino acids, and Pos represents the set of positive binding pairs. Therefore, the total loss function of the NeoBert model is summarized as follows:

[0112]

[0113] Example 2 Validation of the NeoBert model and comparison of prediction performance with NetMHCpan

[0114] After NeoBert was trained on 70% of the HLA-peptide pairs from five different datasets containing classical and non-classical neoantigens, the prediction performance of the NeoBert model and NetMHCpan were compared using the remaining 30% of the data from five different datasets.

[0115] The performance of each prediction model was evaluated based on the following metrics: accuracy, Matthews correlation coefficient (MCC), F1 score, and area under the receiver operating characteristic curve (AUC).

[0116]

[0117] Among them, TP, FP, FN, and TN represent true positive, false positive, false negative, and true negative.

[0118] like Figure 1As shown, the benchmark results show that: 1) NeoBert has an overall accuracy of 93%, while NetMHCpan has an accuracy of 75%; 2) NeoBert outperforms NetMHCpan in predicting classical and non-classical neoantigens, and only performs slightly worse in RNA-derived neoantigens; 3) NeoBert provides more consistent prediction results than NetMHCpan in different databases, indicating that it is able to capture nonlinear diversification patterns from different sources.

Claims

1. A NeoBert model, the model consists of four consecutive components, including: Sequence Input: Masked HLA and peptide sequences are packaged into a sentence (i.e.<masked HLA,masked peptide> ) for input; Embedding block: adds position embeddings to amino acid (i.e. token) embeddings and fragment embeddings to generate sequence embeddings; Encoder block: contains 12 BertLayers, each of which contains a masked multi-head self-attention mechanism and a feature optimization block for learning representations; Prediction block: The aggregated sequence representation is used to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position.

2. The model of claim 1, wherein in the sequence input, each pair of HLA and peptide sequences is represented as a sentence, i.e., X = ([CLS], HLA sequence, [SEP], peptide sequence), the final hidden feature of the first tag ([CLS]) is used as the sequence representation for classification, and the ([SEP]) tag distinguishes the two sequences, the random masked amino acids of the two sequences are used to train the deep bidirectional representation, and the input peptide and HLA sequences are padded to a length of 128 to accommodate the variable input length; in the embedding block, for each token, the learned embedding (E A or E B ∈R H ) to indicate whether it belongs to HLA or peptide, so as to facilitate the segmentation of the two sequences, where H is the embedding dimension, and WordPiece embedding is used to encode the amino acid embedding (E Ai or E Bj ∈R H ), in the embedding block, position embedding (E i ∈R H ) to encode the position (X) of the amino acid in the sentence, the input representation in tokens (or tokens in HLA) is created by adding its token, segment, and position embeddings as follows: E i =E A +W Am +E i .

3. The model of claim 1, wherein in the encoder block, a multi-head self-attention technique is used to combine bidirectional cross-attention between two sequences by concatenating the sequences, wherein the mechanism includes mapping a query Q to a set of key-value (KV) pairs and obtaining an output, wherein the KV pairs store sequence elements in memory; an attention score is based on the correlation or similarity between Q and K, indicating the importance of information, and a higher attention score (i.e., V) indicates a stronger focus on the corresponding information; and a feature optimization module combines a fully connected layer with LayerNorm and Dropout to learn an enhanced representation (H i ) is used for the i-th marker; for the HLA and peptide connection sequence with a length less than 128, it is filled with 0 to 128.

4. The model of claim 1, wherein in the prediction block, two tasks are used to fine-tune the NeoBert model: HLA peptide binding prediction and masked amino acid prediction, wherein the HLA peptide binding prediction is to input the aggregate sequence representation of each HLA-peptide pair into the S-type layer to predict the label: Y'=Sigmoid(W (2) (Tanh(W (1) H 0 +b (1) ))) in, W (1) ,W (2) , and b (1) is a learnable parameter, H 0 Denotes the representation of special tokens [CLS], and the corresponding loss function is summarized as follows: Where N represents the set of HLA peptide pairs containing positive binding data and negative data. The generated data are similar to those in previous studies. [34] ; A masked amino acid model predicts the positional probabilities of 20 different amino acids in a sequence based on their sequence representation: P i =Softmax(WH i +b) Among them, W and b are learnable parameters, H i The loss function of the token is summarized as follows: Among them, Mask represents the set of positions of all masked amino acids, Pos represents the set of positive binding pairs, and the total loss function of the NeoBert model is summarized as follows:

5. Application of the NeoBert model as described in the first aspect in identifying tumor neoantigens, wherein the model determines whether a peptide is a tumor neoantigen by judging whether the peptide can bind to HLA. If the peptide binds to HLA, the peptide is predicted to be a tumor neoantigen.

6. A method for identifying tumor neoantigens, the method comprising the following steps: 1) Input peptide and HLA sequences into the sequence inputter, mask HLA and peptide sequences are packaged into a sentence (i.e.<maskedHLA,masked peptide> ) for input; 2) The embedding block adds the position embedding to the amino acid (i.e., token) embedding and fragment embedding to generate a sequence embedding; 3) The encoder block uses multi-head self-attention technology to combine the bidirectional cross attention between two sequences by concatenating the sequences; 4) The prediction block uses the aggregated sequence representation to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position; 5) Determine whether the peptide can bind to HLA.

7. The method of claim 6, wherein in the sequence input, each pair of HLA and peptide sequences is represented as a sentence, i.e., X = ([CLS], HLA sequence, [SEP], peptide sequence), the final hidden feature of the first tag ([CLS]) is used as the sequence representation for classification, and the ([SEP]) tag distinguishes the two sequences, the random masked amino acids of the two sequences are used to train the deep bidirectional representation, and the input peptide and HLA sequences are padded to a length of 128 to accommodate the variable input length; in the embedding block, for each token, the learned embedding (E A or E B ∈R H ) to indicate whether it belongs to HLA or peptide, so as to facilitate the segmentation of the two sequences, where H is the embedding dimension, and WordPiece embedding is used to encode the amino acid embedding (E Ai or E Bj ∈R H ), in the embedding block, position embedding (E i ∈R H ) to encode the position (X) of the amino acid in the sentence, the input representation in tokens (or tokens in HLA) is created by adding its token, segment, and position embeddings as follows: E i =E A +E Am +E i In the encoder block, a multi-head self-attention technique is used to combine the bidirectional cross attention between two sequences by concatenating the sequences. The mechanism includes mapping the query Q to a set of key-value (KV) pairs and obtaining the output, where the KV pairs store the sequence elements in memory. The attention score is based on the correlation or similarity between Q and K, indicating the importance of the information. The higher the attention score (i.e., V), the stronger the attention to the corresponding information. The feature optimization module combines the fully connected layer with LayerNorm and Dropout to learn the enhanced representation (H i ) is used for the i-th marker, and for the connection sequence of HLA and peptide with a length less than 128, it is filled with 0 to 128.

8. The method of claim 6, wherein in the prediction block, two tasks are used to fine-tune the NeoBert model: HLA peptide binding prediction and masked amino acid prediction, wherein the HLA peptide binding prediction is to input the aggregate sequence representation of each HLA-peptide pair into the S-type layer to predict the label: Y'=Sigmoid(W (2) (Tanh(W (1) H 0 +b(1) ))) in, W (1) ,W (2) , and b (1) is a learnable parameter, H 0 Denotes the representation of special tokens [CLS], and the corresponding loss function is summarized as follows: Among them, N represents the set of HLA peptide pairs containing both positive binding data and negative data, and the generated data are similar to those in previous studies; A masked amino acid model predicts the positional probabilities of 20 different amino acids in a sequence based on their sequence representation: P i =Softmax(WH i +b) Among them, W and b are learnable parameters, H i The loss function of the token is summarized as follows: Among them, Mask represents the set of positions of all masked amino acids, Pos represents the set of positive binding pairs, and the total loss function of the NeoBert model is summarized as follows: If a peptide binds to HLA, it is predicted to be a tumor neoantigen.

9. A system for identifying tumor neoantigens, the system comprising: Sequence input module, used to input peptide and HLA sequences, masked HLA and peptide sequences are packaged into a sentence; Sequence processing module, used to process input sequences; The binding probability prediction module uses the aggregated sequence representation to predict the binding probability, while the masked amino acid model predicts the masked amino acid based on the representation of each corresponding position; The result output module determines whether the peptide is a tumor neoantigen by judging whether the peptide can bind to HLA.

10. A computer-readable storage medium having a computer program stored thereon, which implements the steps of the method according to claim 6 when the program is executed by a processor.

Images