A tumor vaccine neoantigen optimization method based on a monte carlo tree search algorithm

Abstract

The application discloses a tumor vaccine neoantigen optimization method based on a Monte Carlo tree search algorithm. The method searches for potential antigen polypeptides that have not been discovered by experiments with the help of MCTS, forms a high-quality training data set, and then trains a neural network. The neural network can be used for mutation optimization of antigens so that they have higher immunogenicity. The application can be based on the patient's own neoantigen and its specific HLA phenotype customization optimization. Compared with other existing methods, more vaccine candidate peptides with strong HLA affinity and high immunogenicity potential can be generated. After measuring the replacement rate of the reference peptide replaced by the candidate peptide, it is found that the experimental results are consistent with the design expectations, further highlighting that the application can effectively identify and MHC molecules with stronger stability and affinity. The peptide generates valuable antigen sequences with high immunogenicity potential, and provides an effective solution for cancer vaccine design.

Description

Technical Field

[0001] This invention belongs to the field of cancer immunotherapy and antigen-peptide vaccine design technology, and more specifically relates to a method for generating high-quality antigen-peptide vaccines by simultaneously using reinforcement learning and deep neural networks. Background Technology

[0002] Tumor immunotherapy refers to the use of the body's own immune system to eliminate tumor cells. In 2013, this therapy was named the most important scientific breakthrough of the year by *Science* magazine, and in 2018 it won the Nobel Prize in Medicine and Physiology. Due to mutations in the genome of tumor cells, new proteins are produced within human tumor cells. After hydrolysis, these proteins yield antigenic peptides not present in normal cells, called neoantigens. Some neoantigens bind to human leukocyte antigens (HLA) to form peptide-HLA (pHLA) complexes, which are presented on the cell surface and recognized by CD8+ T cells, thereby activating the human immune response.

[0003] To enhance anti-tumor immune responses and improve the immunogenicity and efficacy of vaccines, the design of personalized cancer vaccines using neoantigens and their mutants has become a promising strategy. These mutants can achieve a stronger immune response by enhancing T cell recognition or by generating cross-reactivity to cover more mutated tumor cells.

[0004] However, existing design methods face several challenges. Rational designs based on molecular dynamics simulations often employ computational methods such as free energy perturbation, aiming to enhance the binding affinity of antigenic peptides to HLA by introducing amino acid mutations; however, these methods are inefficient and time-consuming. Generative models based on data and machine learning training mostly focus on generating antigenic peptides similar to those in databases, but the limited amount of existing experimental data results in a lack of generalization. Summary of the Invention

[0005] To address the shortcomings in the aforementioned technical background, this invention provides a method for optimizing neoantigens in tumor vaccines based on the Monte Carlo Tree Search (MCTS) algorithm. This method starts with a large number of random peptides and progressively mutates them using the MCTS algorithm until they become peptides found in a standard database. The entire mutation optimization process generates a large amount of previously unseen high-quality data, which is then aggregated as a training dataset. This dataset is used to train a neural network to understand how to optimize an antigen mutation to achieve higher immunogenicity.

[0006] This invention is achieved using the following technical solution:

[0007] A method for optimizing neoantigens in tumor vaccines based on the Monte Carlo tree search algorithm, characterized by the following steps:

[0008] 1) Generate a large number of random antigenic peptides using a uniform probability distribution, and use them as initial antigenic peptides;

[0009] 2) Perform single mutations on the initial antigenic peptide multiple times consecutively;

[0010] 3) If within the specified number of optimization steps N episode If the target is achieved, i.e., an antigenic peptide is found in the standard database, then parameter z = 1 is set; otherwise, if the target is not achieved, z = -1 is set. The mutation probabilities π(a|s) corresponding to all antigenic peptides in the single mutation process in step 2) are combined with the z value representing whether the target has been achieved (π(a|s), z) as a training dataset to train the neural network. The standard database is constructed as follows: based on the patient's own HLA phenotype, antigenic peptides that can bind to it are collected from the Immune Epitope Database (IEDB) to form a standard database corresponding to the corresponding HLA.

[0011] 4) Repeat steps 1)-3)N game The trained neural network is then compared with the best existing neural network to determine the best neural network, which is the customized neural network corresponding to the HLA phenotype.

[0012] 5) By using neoantigen peptide sequences as input to a customized neural network, a large number of highly immunogenic antigen sequences can be rapidly generated.

[0013] In the above technical solution, further, the method for repeatedly performing single mutations on the initial antigenic peptide is as follows: repeatedly performing N mutations using the MCTS algorithm. self The first "mutation simulation" is used to obtain the mutation distribution probability, and the initial antigen peptide is mutated based on the mutation distribution probability.

[0014] The method of repeatedly performing N using the MCTS algorithm is described. self The second "mutation simulation" yields the mutation distribution probability, specifically including the following steps:

[0015] Selection Phase: Starting from the root node, i.e., the current antigen peptide s0, when exploring downwards to step L, i.e., after L mutations, a leaf node s that has never been seen before is encountered. L When the time is reached, the exploration is stopped; during the exploration process, the mutation action a is executed for the t-th time. t According to the formula Choose the action to be performed, among which, leaf node s t All executable actions, `cpuct` is a parameter representing the breadth of exploration, and `Z` is the number of edges. The sum of the number of times it was visited; For prior probability, For the edge Number of visits For the value of the action;

[0016] Extended evaluation phase: utilizing neural network f θ (·) for leaf node s L Predict the probability of amino acid mutation p net (s L ) and the success rate of achieving the target v net (s L Two values: (p) net (s L ), v net (s L ))=f θ (s L Simultaneously, for node s L Each edge (s) L a) Initialize the stored statistical information: {N(s L , a)=0, W(s) L , a) = v net (s L ), Q(s) L , a) = v net (s L ), P(s L , a) = p net (s L )},N(s L a) Characterize the edge (s) L a) Number of visits, W(s) L a) represents the total value of the actions, Q(s) L a) represents the average action value, P(s) L a) represents the selection of edge (s) L The prior probability of a);

[0017] Backtracking phase: Update the stored statistics for each edge encountered during the intermediate process of the selection phase. Specifically, the method is: number of visits N(s) t a t )=N(s t a t +1, total action value W(s) t ,a t )=W(s t a t )+v net (s LAverage motion value The above three stages are repeated continuously and iteratively. self After that, the mutation distribution probability is obtained. Where ε is a temperature parameter characterizing the abundance of exploration. This indicates the mutation actions that can be performed on the root node s0.

[0018] Furthermore, the use of neural network f θ (·) for leaf node s L Predict the probability of amino acid mutation p net (s L ) and the success rate of achieving the target v net (s L Two values, the specific method is as follows:

[0019] First, the antigen peptide sequence is inserted into the amino acid embedding layer EmbeddingA. A The embedding layer vector is obtained by parsing (·); the dimension of the embedding layer vector is d. m The amino acid embedding layer EmbeddingA A (·) consists of an embedding layer (Embedding(·)) and a position embedding layer (PE(·)):

[0020] EmbeddingA A (s L =Embedding(s) L )+PE(s L )

[0021] Among them, leaf node s L The input is represented by PE(·), which is constructed as follows:

[0022]

[0023] i represents the number of dimensions, PE pos,2i and PE pos,2i+1 These represent the 2i-th and 2i+1-th values ​​of the embedding vector corresponding to the amino acid at the pos-th position on the antigen peptide sequence, respectively, where i = 0, 1, ... and 2i ≤ d. m ,2i+1≤d m ;

[0024] Next, the output of the amino acid embedding layer is processed using the decoder module of the Transformer neural network model. The first line of information in the processed result is extracted and processed using a linear layer to obtain the output value. This output value is processed in two ways: one is at the policy end, where a scalar of (1, 20 × n) is output by the linear layer to represent the predicted amino acid mutation probability p.net (s L Another approach involves processing the value at the value side, using a combination of a linear layer and a softmax function to output a value that represents the success rate v in achieving the target. net (s L ).

[0025] Further, the mutation probability π(a|s) corresponding to all antigen peptides in the single mutation process of step 2) is combined with the z value representing whether the target has been achieved (π(a|s), z) as the training dataset. Specifically, the method is as follows: for every N optimizations... it e r Each random initial antigenic peptide is considered as one round. In the first 90% of optimization steps of this round, as long as the specified number of optimization steps N is reached... episode Within the target dataset, finding an antigen peptide with a success rate prediction value exceeding 0.95 in a standard database is considered achieving the target. For the remaining 10%, finding an antigen peptide in the standard database is also considered achieving the target. All (π(a|s), z) data from this round are combined to form a training dataset.

[0026] The beneficial effects of this invention are as follows:

[0027] Compared to traditional rational design methods based on molecular dynamics simulations, the method of this invention offers faster generation speed and higher efficiency. Compared to other generative models trained using data and machine learning, this method is more generalizable because it is not limited to existing databases. Compared to other existing methods, the method of this invention can generate more vaccine candidate peptides with strong HLA affinity and greater immunogenic potential. Attached Figure Description

[0028] Figure 1 To optimize the antigen peptide process based on the method of this invention.

[0029] Figure 2 To evaluate the performance of different optimization methods using NetMHC 4.0 as the predictor. Detailed Implementation

[0030] For ease of understanding, each antigenic peptide can be viewed as a chessboard with n columns and 20 rows. The number of columns n represents the number of amino acids in the antigenic peptide, and the number of rows indicates the number of different types of amino acids (there are 20 naturally occurring amino acids). Figure 1 The process of optimizing antigen peptides based on the method of this invention can rapidly generate a large number of highly immunogenic antigen sequences and place them into a target database.

[0031] A method for optimizing neoantigens in tumor vaccines based on the Monte Carlo tree search algorithm is described in the following steps: First, a large number of random antigenic peptides are generated using a uniform distribution probability as initial antigenic peptides.

[0032] Next, these initial antigenic peptides were subjected to a large number of single mutations in succession. Before mutation, N was repeatedly performed according to the MCTS algorithm. self This "mutation rehearsal" yields the mutation distribution probability π, which represents the probability of amino acid mutations at different positions. In other words, each position on an n×20 chessboard will have a probability distribution. The mutation strategy is then determined based on this probability π. Single mutations were performed on the antigenic peptide.

[0033] If the specified number of optimization steps N episode If the target is achieved (i.e., an antigenic peptide is found in the standard database), the parameter z = 1 is set; otherwise, if the target is not achieved, z = -1 is set. The mutation probabilities π(a|s) corresponding to all antigenic peptides in the single mutation process are combined with the z value (π(a|s), z) representing whether the target has been achieved, and used as a training dataset to train the neural network. The standard database consists of antigenic peptides that can bind to HLA phenotypes, selected from the immune epitope database IEDB.

[0034] Repeat step N above. game Each time, the neural network after each training iteration is compared with the best existing training model. Specifically, N is optimized using two neural networks. arena A random initial peptide is used, and the requirement is that the antigen peptide in the standard database must be found to achieve the goal. The one that achieves the goal the most wins and becomes the best neural network, which is the customized neural network corresponding to the HLA phenotype.

[0035] By using neoantigen peptide sequences as input to a customized neural network, a large number of highly immunogenic antigen sequences can be rapidly generated.

[0036] The "mutation pre-visit" process is implemented using a Monte Carlo Tree Search (MCTS) reinforcement learning algorithm similar to AlphaGo Zero. Unlike traditional MCTS algorithms, this algorithm incorporates a neural network, allowing it to predict the probability of reaching the target without requiring continuous pre-visiting until the actual target is achieved. Each node in the search tree corresponds to an antigenic peptide sequence, and nodes are connected by corresponding edges (s, a), where a represents the specific mutation action performed on the antigenic peptide at node s, a∈A(s), and A(s) represents the set of executable mutation actions. Each edge stores the following statistics: {N(s, a), W(s, a), Q(s, a), P(s, a)}, where N(s, a) represents the number of times the edge is visited, W(s, a) represents the total action value, Q(s, a) represents the average action value, and P(s, a) represents the prior probability of selecting the edge. The "mutation pre-visit" process iterates through N... selfEach "mutation rehearsal" consists of three phases: selection, extended evaluation, and retrospective phases, as detailed below:

[0037] Selection Phase: Starting from the root node, i.e., the current antigen peptide s0, when exploring downwards for L steps (L mutations), a leaf node s that has never been seen before is encountered. L The search is terminated at this point. To ensure both depth and breadth in the search process, during the intermediate exploration phase (t < L), the mutation action a is executed for the t-th time. t According to the formula Choose the action to be performed, among which, leaf node s t All executable actions, c puct Z is a parameter representing the breadth of exploration, where Z is the number of edges. The sum of the number of times it was visited; For prior probability, For the edge Number of visits Action value is determined by the algorithm. In the early stages of the search, actions with high prior probabilities and low access frequency are favored; in the later stages, actions with high value are prioritized. The extended evaluation phase utilizes a neural network f. θ (·) for leaf node s L Predict the probability of amino acid mutation p net (s L ) and the success rate of achieving the target v net (s L Two values: (p) net (s L ), v net (s L ))=f θ (s L Simultaneously, for node s L Each edge (s) L a) Initialize the stored statistical information: {N(s L , a)=0, W(s) L , a) = v net (s L ), Q(s) L , a) = v net (s L ), P(s L , a) = p net (s L )},N(s L a) Characterize the edge (s) L a) Number of visits, W(s) L a) represents the total value of the actions, Q(s) L a) represents the average action value, P(s)L , a) represents the prior probability of choosing this edge.

[0038] Backtracking Phase: Update the stored statistics for each edge encountered during the intermediate process of the selection phase. The specific update method is: number of visits N(s) t a t )=N(s t a t +1, total action value W(s) t a t )=W(s t a t )+v net (s L Average motion value Essentially, v net (s L The success rate of optimizing to a standard database is used to update the value function of each action.

[0039] Repeat the above three stages N in a continuous iterative manner self After that, the mutation distribution probability is obtained. Where τ is a temperature parameter characterizing the abundance of exploration. This indicates the mutation actions that can be performed on the root node s0.

[0040] Exit the "mutation rehearsal" process and take real action on the current antigenic peptide s0. This action is determined by the mutation distribution probability π(a|s0) from the "mutation pre-simulation" results. Subsequently, the antigen peptide optimized by mutation based on the mutation distribution probability becomes the new root node, and the above "mutation pre-simulation" must be performed before taking any actual action.

[0041] The mutation probabilities π(a|s) of all antigenic peptides in the single mutation process are combined with the z-value representing whether the target has been achieved (π(a|s), z) to form a training dataset for training the neural network. Specifically, for each N... iter Each random initial antigenic peptide is considered as one round. In the first 90% of optimization steps of this round, as long as the specified number of optimization steps N is reached... episode Within this round, encountering an antigenic peptide with a predicted success rate exceeding 0.95 in a standard database is considered achieving the target. For the remaining 10%, finding an antigenic peptide in the standard database is required to achieve the target. All (π(a|s), z) data from this round are combined to form a training dataset for training the neural network.

[0042] A total of N were performed. game Round optimization. Data from each round of play is stored in the training dataset, but the dataset retains a maximum of [number missing] rounds. In each round of data, any historical data exceeding a certain threshold will be forgotten. The loss function used during the training of the neural network is loss = (zv... net ) 2 -π T logp net +γ||θ|| 2 θ represents the parameters of the neural network.

[0043] The core of this invention utilizes a self-attention mechanism to understand the nature of immune antigen peptides, predict the distribution probability of different amino acids at various positions in the next mutation, and optimize the success rate to a standard database. This is achieved using a neural network f... θ (·) for leaf node s L Predict the probability of amino acid mutation p net (s L ) and the success rate of achieving the target v net (s L Two values, the specific method is as follows:

[0044] First, the antigen peptide sequence is inserted into the amino acid embedding layer. AA The embedding layer vector is obtained by parsing (·); the dimension of the embedding layer vector is d. m The amino acid embedding layer EmbeddingA A (·) consists of the common embedding layer Embedding(·) and the position embedding layer PE(·):

[0045] EmbeddingA A (s L =Embedding(s) L )+PE(s L )

[0046] Among them, leaf node s L The input is represented by PE(·), which is constructed as follows:

[0047]

[0048] i represents the number of dimensions, PE pos，2i and PE pos，2i+1 These represent the 2i-th and 2i+1-th values ​​of the embedding vector corresponding to the amino acid at the pos-th position on the antigen peptide sequence, respectively, where i = 0, 1, ... and 2i ≤ d. m ,2i+1≤d m .

[0049] Next, the output of the amino acid embedding layer is processed using the decoder module of the Transformer neural network model. This module consists of two core layers: a multi-head self-attention mechanism layer and a fully connected feedforward network layer for feature processing. Residual connections are used around both sub-layers, and layer normalization is performed. The first line of information from the decoder module's output is extracted and processed using a linear layer to obtain the output. This output value undergoes two processing steps: one at the policy level, where the linear layer outputs a scalar (1, 20 × n) to represent the predicted amino acid mutation probability p. net (s L Another approach involves processing the value at the value side, using a combination of a linear layer and a softmax function to output a value that represents the success rate of optimization to a standard database, i.e., achieving the target success rate v. net (s L ).

[0050] Through the above training process, a customized neural network can be obtained based on the patient's HLA phenotype. Specific neoantigen peptide sequences extracted from the patient are used as input to the neural network, enabling the rapid generation of a large number of highly immunogenic and valuable antigen sequences.

[0051] The advantages of this invention are illustrated below by comparing it with several other existing methods:

[0052] Using NetMHC4.0 as the predictor, three criteria were employed to evaluate the quality of optimized antigen peptide generation. These three criteria are:

[0053] 1. The proportion of antigenic peptides that have a strong affinity for HLA (NetMHC4.0 predicted rank ≤ 0.5)

[0054] 2. The proportion of generated antigenic peptides with weak affinity for HLA (NetMHC 4.0 prediction: 0.5%). <rank≤2.0)

[0055] 3. The percentage of antigenic peptides that have stronger affinity than the initiating peptide.

[0056] Taking HLA*02:01 as an example, nine antigenic peptides with experimental data that bind to it were used as starting peptides for optimization. The methods used were as follows:

[0057] "R." (Random) means that based on the initial peptide, a mutation site is randomly selected with uniform probability, and another amino acid is randomly mutated using a uniform probability distribution.

[0058] "I.+B." (IEDB+BLOSUM) means that based on the initial peptide, a mutation site is randomly selected with uniform probability, and a different amino acid is mutated using a probability distribution of an IEDB database combined with the BLOSUM62 substitution probability.

[0059] “T.” (TransPHLA-AOMP) is an algorithm published in 2022 entitled “A transformer-based model to predict peptide-HLA class I binding and optimize mutated peptides for vaccine design”. The Automatically Optimized Mutated Peptides (AOMP) part can optimize antigen peptides to enhance binding affinity.

[0060] "V24" indicates that the present invention is based on the initial peptide optimization, with the additional requirement that the generated antigenic peptide has no more than 4 mutations compared to the initial peptide.

[0061] "V2" indicates that the present invention is based on the initial peptide optimization and no other settings are made.

[0062] Comparative analysis revealed that, regardless of whether additional modifications are made to the invention, it is significantly superior to other methods (such as...). Figure 2 ).

[0063] Finally, flow cytometry was used to validate the optimized candidate peptides based on the BING4 protein fragment CQWGRLWQL and the protein phosphatase PHLPP1 (563-571) fragment mutant ILCETCLIV. The peptide replacement rate (i.e., the proportion of the reference peptide replaced by the candidate peptide) of each antigenic peptide was measured. The experimental results were consistent with the design expectations, and the mutants were superior to the starting peptides (as shown in Tables 1 and 2). These results further highlight that the present invention can effectively recognize peptides with stronger stability and affinity that bind to MHC molecules, and has the ability to generate valuable antigen sequences with high immunogenic potential, providing an effective solution for cancer vaccine design.

[0064] Table 1. Experimental validation results of optimized antigenic peptides based on CQWGRLWQL

[0065]

[0066] Table 2. Experimental validation results of optimized antigenic peptides based on ILCETCLIV

[0067]

Claims

1. A method for optimizing neoantigens in tumor vaccines based on the Monte Carlo tree search algorithm, characterized in that, Includes the following steps: 1) Generate a large number of random antigenic peptides using a uniform probability distribution, and use them as initial antigenic peptides; 2) Perform single mutations on the initial antigenic peptide multiple times consecutively; 3) If within the specified number of optimization steps Once the objective is achieved—that is, the antigen peptide is found in the standard database—the parameters are set. Conversely, if the goal is not achieved, then set The mutation probabilities corresponding to all antigenic peptides in the single mutation process of step 2) are calculated. Whether the characterization objective has been achieved Value Combination This serves as the training dataset for training the neural network. Represents a node. This indicates a mutation action; the standard database is composed of antigenic peptides that can bind to HLA phenotypes, selected from the immune epitope database IEDB. 4) Repeat steps 1)-3). The trained neural network is then compared with the best existing neural network to determine the best neural network, which is the customized neural network corresponding to the HLA phenotype. 5) By using neoantigen peptide sequences as input to a customized neural network, a large number of highly immunogenic antigen sequences can be rapidly generated; The method for repeatedly performing single mutations on the initial antigenic peptide is as follows: The MCTS algorithm is used to repeatedly perform... The mutation pre-run is performed to obtain the mutation distribution probability, and the initial antigen peptide is mutated based on the mutation distribution probability. The MCTS algorithm is used repeatedly. The second "mutation simulation" yields the mutation distribution probability, specifically including the following steps: Selection phase: Starting from the root node, i.e., the current antigen peptide. Let's set off, exploring downwards to the... Step, i.e., mutation Next, I encountered a leaf node I had never seen before. When the time limit is reached, the exploration is terminated; during the exploration process, the mutation action is executed for the t-th time. According to the formula Choose the action to be performed, among which, leaf node All executable actions, , It is a parameter characterizing the breadth of exploration. For the edge The sum of the number of times it was visited; For prior probability, For the edge Number of visits For the value of the action; Extended evaluation phase: utilizing neural networks Leaf nodes Predicting the probability of amino acid mutations and the success rate of achieving the target Two values: Simultaneously for nodes Each edge Initialize the stored statistics: , Representation edge Number of visits This represents the total value of the actions. Indicates the average value of an action. Indicates the selection of edges The prior probability; Backtracking Phase: Update the stored statistics for each edge encountered during the intermediate process of the selection phase. The specific update method is: visit count. Total value of actions Average action value ; Repeat the above three stages continuously. After that, the mutation distribution probability is obtained. : ;in, It is a temperature parameter that characterizes the richness of exploration. Represents the root node Executable mutation actions.

2. The method for optimizing neoantigens in tumor vaccines based on the Monte Carlo tree search algorithm according to claim 1, characterized in that, The use of neural networks Leaf nodes Predicting the probability of amino acid mutations and the success rate of achieving the target Two values, the specific method is as follows: First, the antigen peptide sequence is inserted into the amino acid intercalation layer. The process involves parsing to obtain the embedding layer vector; the dimension of the embedding layer vector is... The amino acid intercalation layer By embedding layer and position embedding layer composition: Among them, leaf nodes Indicates input, The creation method is as follows: Indicates the number of dimensions. and These represent the first and second lines of the antigen peptide sequence. The embedding layer vector corresponding to the amino acid at the i-th position is the first... The value and the first One value, and ; Next, the output of the amino acid embedding layer is processed using the decoder module of the Transformer neural network model. The first line of information in the processed result is extracted and processed using a linear layer to obtain the output value. The output value undergoes two processing steps simultaneously: one is processing at the policy level, where a linear layer outputs a... A scalar value used to characterize the predicted amino acid mutation probability. Another approach involves processing the value at the value side, using a combination of a linear layer and a softmax function to output a value that represents the success rate of achieving the target. .

3. The method for optimizing neoantigens in tumor vaccines based on the Monte Carlo tree search algorithm according to claim 1, characterized in that, In step 3), the mutation probabilities corresponding to all antigen peptides in the single mutation process of step 2) are... Whether the characterization objective has been achieved Value Combination The specific method for using this as the training dataset is as follows: Each optimization One random initial antigenic peptide is considered as one round, prior to which In this optimization process, as long as the specified number of optimization steps is reached... Within, the predicted success rate of a standard database exceeds [a certain value]. The antigenic peptide is considered to have achieved the target, and the remaining ones are considered to have achieved the target. The requirement is that the antigenic peptide must be found in a standard database to be considered as achieving the goal; All of the round The data are combined to form a training dataset.

Images