Prediction model training method, computer device and computer storage medium

Abstract

The embodiment of the application discloses a kind of prediction model training method, computer equipment and computer storage medium.The joint learning of amino acid sequence and molecular information sequence of two " languages " is carried out in pre-training phase, and the parameter optimization of enzyme-substrate interaction judgment task is carried out in fine-tuning phase, and the prediction model constructed by the method can naturally fuse the amino acid sequence characteristics of enzyme and the molecular information sequence characteristics of substrate.The model is no longer isolatedly analyzed enzyme or substrate when processing enzyme-substrate pair, but the two are jointly modeled as an organic whole.This cross-modal feature fusion capability enables the model to more deeply understand the nature mechanism of enzyme-substrate interaction, thereby effectively breaking through the limitation of " single language " model when processing heterogeneous data, significantly improving the generalization ability and prediction accuracy in the enzyme-substrate interaction prediction task of cross enzyme family or far evolutionary distance.

Description

Technical Field

[0001] This application relates to the field of bioinformatics, specifically to a prediction model training method, computer equipment, and computer storage medium. Background Technology

[0002] Enzymes are biological macromolecules (mainly proteins) that significantly accelerate biochemical reactions by specifically binding to substrates, making them crucial in drug development, biomanufacturing, and synthetic biology. Accurate identification of enzyme-substrate interactions is essential for biotechnology and drug discovery. Machine learning methods have been applied to enzyme-substrate interaction prediction, but they typically only generalize well to evolutionarily related enzymes. In recent years, advancements in Natural Language Processing (NLP) have driven the development of specialized large-scale language models for biological sequences (amino acid sequences, nucleic acid sequences, SMILES strings). Among these, SMILES (Simplified Molecular Input Line Entry System) strings are a linear symbol system that explicitly describes molecular structures using ASCII strings.

[0003] These models excel at capturing complex sequence patterns and structural features, but in tasks such as protein design and molecular property prediction, they are mostly "monolingual" models, meaning they can only model single types of biological sequences (such as only amino acid sequences or only SMILES strings). They struggle to effectively integrate the complementary information contained in the two heterogeneous sequence data of enzyme amino acid sequences and substrate SMILES strings, failing to jointly model the key features of enzymes and substrates. This results in limited performance in enzyme-substrate interaction prediction tasks that span enzyme families or have large evolutionary distances. Summary of the Invention

[0004] This application provides a predictive model training method, computer device, and computer storage medium for constructing a bilingual large language model that enables it to jointly learn the amino acid sequence language of proteins and the molecular information language of substrate molecules, thereby establishing a "cross-language" understanding of enzyme-substrate interactions.

[0005] A first aspect of this application provides a method for training a prediction model, the method comprising:

[0006] Obtain an initial prediction model and a pre-training dataset; the pre-training dataset includes amino acid sequences and molecular information sequences.

[0007] The pre-trained dataset is input into the initial prediction model so that the initial prediction model predicts the lexical at the target position in the amino acid sequence and / or the molecular information sequence based on the lexical context in the pre-trained dataset.

[0008] The initial prediction model is optimized based on the word prediction results at the target position and the actual words at the target position to obtain a pre-trained prediction model;

[0009] A fine-tuning dataset is obtained, which includes the amino acid sequences of multiple target enzymes and the molecular information sequences of multiple paired molecules corresponding to each target enzyme; the multiple paired molecules include substrate molecules that interact with the target enzymes and non-substrate molecules that do not interact with the target enzymes.

[0010] The fine-tuned dataset is input into the pre-trained prediction model so that, for each target enzyme, the pre-trained prediction model determines whether there is an interaction between the target enzyme and each of its paired molecules based on the amino acid sequence of the target enzyme and the molecular information sequence of its paired molecules, and adjusts the model parameters of the pre-trained prediction model based on the determination result and the actual interaction label between the target enzyme and each of its paired molecules.

[0011] A second aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect described above.

[0012] A third aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0013] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0014] By jointly learning the two "languages" of amino acid sequences and molecular information sequences during the pre-training phase, and optimizing parameters for enzyme-substrate interaction prediction tasks during the fine-tuning phase, the prediction model constructed by this method can naturally integrate the amino acid sequence features of enzymes and the molecular information sequence features of substrates. When processing enzyme-substrate pairs, the model no longer analyzes enzymes or substrates in isolation, but models them as an organic whole. This cross-modal feature fusion capability enables the model to gain a deeper understanding of the essential mechanism of enzyme-substrate interactions, thus effectively overcoming the limitations of "monolingual" models in handling heterogeneous data, and significantly improving the generalization ability and prediction accuracy in enzyme-substrate interaction prediction tasks across enzyme families or with large evolutionary distances. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the prediction model training method in an embodiment of this application;

[0016] Figure 2This is a flowchart illustrating an exemplary application scenario of the prediction model training method in this application.

[0017] Figure 3 This is a schematic diagram of a computer device in an embodiment of this application. Detailed Implementation

[0018] This application provides a predictive model training method, computer device, and computer storage medium for constructing a bilingual large language model that enables it to jointly learn the amino acid sequence language of proteins and the molecular information language of substrate molecules, thereby establishing a "cross-language" understanding of enzyme-substrate interactions.

[0019] The proposed approach offers a general model for predicting small molecule substrates of enzymes. It processes enzyme sequences and molecular features using multiple independent modules, then combines the feature processing results to predict enzyme-substrate interactions. Furthermore, a deep learning method utilizing positive and unlabeled samples is proposed for discovering toxin-degrading enzymes, which also employs separate feature extraction modules for enzymes and substrates.

[0020] However, the aforementioned approach, due to its separate processing architecture, limits the understanding of the interaction between enzyme and molecular features. Existing models use separate feature extraction modules to process enzymes and substrates before fusing the features. This separate architecture cannot directly and fully capture and encode the complex cross-lingual interaction patterns between the two biomolecules within the model, thus limiting its prediction accuracy and deeper understanding of the interaction mechanisms.

[0021] Secondly, a unified biomolecular representation cannot be achieved. Existing protein language models or molecular language models are "monolingual," meaning they learn within a closed modality. This prevents the models from establishing a shared semantic space in which amino acid sequences and substrate SMILES strings can be represented simultaneously and equally. Consequently, it fails to fundamentally understand that enzyme-substrate interactions are interactions between entities using two different "languages."

[0022] To address the aforementioned issues, the purpose of this application is to construct a unified bilingual large language model that can jointly learn the amino acid sequence language of proteins and the molecular information language of substrate molecules. This allows for the direct establishment of a "cross-language" understanding of enzyme-substrate interactions within the model, enabling high-precision interaction prediction, efficient bidirectional retrieval, and further extending the ability to generate enzyme sequences based on substrate molecules.

[0023] The core of this application's embodiments lies in treating the amino acid sequence of an enzyme and the molecular information sequence of a substrate molecule (such as the string "SMILES") as two different "languages," and constructing a bilingual model capable of understanding and generating both languages ​​based on a large language model architecture (such as the GLM General Language Model). This model learns the general syntax and semantics of both languages ​​through pre-training on a large-scale bilingual corpus, and then fine-tunes it on a high-quality enzyme-substrate pair dataset, thereby gaining the ability to predict downstream tasks such as enzyme-substrate interactions and perform bidirectional retrieval. Furthermore, by constructing a substrate-enzyme generation dataset, the model is continuously trained to enable it to generate enzyme sequences conditionally (e.g., with substrate molecule cues) and unconditionally. The GLM General Language Model is a general language model architecture whose pre-training objective incorporates autoregressive blank filling, enabling unidirectional and bidirectional attention within a single encoder structure.

[0024] The prediction model training method in the embodiments of this application is described below:

[0025] Please see Figure 1 One embodiment of the prediction model training method in this application includes:

[0026] 101. Obtain the initial prediction model and the pre-training dataset; the pre-training dataset includes amino acid sequences and molecular information sequences;

[0027] The method in this embodiment can be applied to a computer device, which can deploy an initial prediction model. This initial prediction model can be the GLM model architecture described above, or other advanced large language model architectures, such as various variants of Transformer, state-space models like Mamba, as long as it can handle the mixed input of the two different "languages" and perform effective joint training. No limitation is made here.

[0028] This pre-training dataset is used to pre-train the model. It can be obtained from public databases, literature, or experimental data, and contains a large number of amino acid sequences (such as sequences of various proteins and enzymes) and molecular information sequences (such as SMILES strings of various organic molecules and drug molecules) from different sources. These sequence data do not require pre-annotation of enzyme-substrate interactions; the aim is to allow the model to learn the basic grammatical rules, structural features, and potential relationships between the two "languages," such as specific residue combination patterns in amino acid sequences and the representation of functional groups in molecular information sequences. Through learning from this massive amount of data, the model can initially form a general understanding of the two biomolecular sequences.

[0029] The molecular information sequence refers to a string or sequence that can be used to describe molecular structural information, as long as it can be parsed by the model and converted into a word sequence for processing. This molecular information sequence can be the string SMILES, or other representations of the molecule, such as SELFIES (SELF-referencing Embedded Strings), molecular diagrams, 3D coordinates, etc.

[0030] For example, the UniRef50 protein cluster can be obtained from the UniProt database, and the SMILES-250M molecular dataset can be obtained from Huggingface. This allows for the processing of approximately 4.2 million biomolecular sequences (each with 1024 tokens), comprising approximately 4.3 billion tokens (lexical units), to form a pre-training corpus. Each token represents an independent basic biomolecular unit. Specifically, for the amino acid sequence language of proteins, tokenization is performed on individual amino acid residues (e.g., single-letter or three-letter codes corresponding to 20 common amino acids such as Ala and Ser); for the SMILES string language of substrate molecules, tokenization is performed on basic building blocks such as atomic symbols (e.g., C, O, N), chemical bonds (e.g., -, =, #), ring numbers (e.g., 1, 2), and special symbols in the SMILES syntax.

[0031] In some optional implementations, when acquiring the pre-training dataset, amino acid sequences and molecular information sequences can be acquired. For conflicting characters in the molecular information sequence that have the same form as characters in the amino acid sequence but different meanings, the conflicting characters are mapped to specific special form characters to distinguish the semantic differences between the same form characters in the amino acid sequence and the molecular information sequence.

[0032] For example, to avoid conflicts between identical characters in the amino acid sequence and the SMILES string that have different meanings, a conflict-free vocabulary can be constructed, and conflicting characters in the SMILES can be remapped. When characters in the SMILES string are formally identical to characters in the protein amino acid sequence but represent different biochemical meanings, these SMILES characters are specifically re-encoded or labeled to eliminate ambiguity.

[0033] For example, the single letter 'A' in an amino acid sequence represents alanine, while in the SMILES string, 'A' might represent the element symbol argon. This is not common in the description of small organic molecules. However, to completely avoid any potential confusion, characters in SMILES that are identical to the single-letter codes of amino acids (such as 'A', 'C', 'D', etc.) can be remapped by adding specific markers before and after them (such as '#A#', '#C#') or replacing them with entirely new unique identifiers (such as 'Smile_A', 'Smile_C'). This allows the model to clearly distinguish the same characters in these two different "languages," ensuring that during pre-training, the model accurately learns the unique grammar and semantic rules of each language without comprehension bias due to overlapping character forms. The construction of this conflict-free vocabulary is the foundation for joint learning of two heterogeneous biomolecular languages ​​within the same model.

[0034] In addition, special tokens such as <+>, [CLS], [MASK], [True], and [False] can be added to the vocabulary to support pre-training and downstream tasks. For example, in the vocabulary design of a bilingual pre-training corpus, in addition to the basic tokens for amino acid sequences and SMILES strings mentioned above, a series of special tokens with specific functions can be introduced. Among them, the <+> token is designed to clearly separate the amino acid sequence of the enzyme and the SMILES string of the substrate, so that when the model processes mixed inputs containing both, it can clearly identify the boundary between the two "languages" and provide a clear structural indication for the subsequent cross-language attention mechanism. The [CLS] token (i.e., the Classification token) is placed at the beginning of each input sequence. Its hidden state after model processing will serve as a comprehensive representation of the entire sequence and can be used in subsequent downstream tasks such as classification and retrieval, serving as a key basis for judging enzyme-substrate interaction relationships or making similarity comparisons.

[0035] The [MASK] token is a core component of the autoregressive blank-filling pre-training objective. During pre-training, some tokens in the input sequence are randomly replaced with [MASK], and the model needs to predict the masked original tokens based on lexical context information. This mechanism helps the model learn dependencies and semantic associations within and across languages, thereby improving its deep understanding of sequences. The [True] and [False] tokens primarily serve the construction of binary judgment tasks during the pre-training phase. For example, in constructed enzyme-substrate pairs, the model needs to predict the truth value of the interaction relationship: [True] represents the presence of an interaction, and [False] represents its absence. Through training on such tasks, the model can initially learn discriminative features to distinguish whether an enzyme and substrate interact.

[0036] The introduction of these special tokens enables the constructed bilingual pre-training corpus to not only contain rich original biomolecular sequence information, but also to possess the structural foundation to support complex pre-training objectives and diverse downstream tasks, laying an important foundation for large language models to learn bilingual universal representations and cross-language interactive understanding capabilities.

[0037] 102. Input the pre-training dataset into the initial prediction model so that the initial prediction model predicts the lexical at the target position in the amino acid sequence and / or the molecular information sequence based on the lexical context in the pre-training dataset;

[0038] In this step, the target position can be a randomly selected word position to be masked, or it can be the position of the next word in the sequence that is exposed to the model.

[0039] Therefore, in some alternative implementations, during the pre-training of the initial prediction model, the words at the target position can be masked, and the initial prediction model can predict the words at the target position based on the words before and after the target position. In other alternative implementations, the words in the sequence preceding the target position can also be exposed to the initial prediction model, and the initial prediction model can predict the words at the target position based on the words preceding the target position.

[0040] "Making" refers to selecting a portion of tokens in a sequence and temporarily hiding or replacing them with special markers. This tests the model's ability to understand and reason about contextual information, thereby enabling the model to learn the dependencies and overall semantics between tokens in the sequence. This masking mechanism simulates scenarios where information is missing or needs to be predicted in biological sequences, allowing the model to learn how to infer missing information from context during the pre-training phase. This is crucial for handling situations where sequence information may be incomplete or where it is necessary to predict the overall interaction relationship based on a partial sequence in subsequent tasks such as enzyme-substrate interaction prediction.

[0041] For example, two objectives can be used: "mask recovery" and "next token prediction." With a 50% probability, one objective can be randomly applied to each batch of sequences for training. That is, within each training batch, for all biomolecular sequences (including both amino acid sequences and SMILES strings), a pre-training objective is randomly selected for the entire batch. Specifically, there is a 50% chance that all sequences in the batch will be trained using the mask recovery objective; and a 50% chance that all sequences in the batch will be trained using the next token prediction objective. This batch-level objective selection strategy ensures that the model can learn two different prediction tasks in a balanced way during training, thereby comprehensively improving its understanding and generation capabilities for both "languages."

[0042] This process is achieved through the autoregressive blank-filling mechanism of GLM. During pre-training, amino acid sequences and SMILES strings are randomly mixed in each batch, forcing the model to learn to distinguish and understand two different "languages." Specifically, in the model input layer, both amino acid sequences and SMILES strings are treated as independent text sequences, requiring no additional modality transformation or feature engineering. For the masked recovery target, 15% (or other percentages) of the tokens in the sequence are randomly selected for masking, and the model needs to predict the original masked tokens based on contextual information. For the next token prediction target, starting from the beginning of the sequence, the model predicts subsequent tokens one by one until the end of the sequence.

[0043] By alternating between these two pre-training objectives, the model can not only learn the grammatical rules within the amino acid sequence (such as the order of amino acid residues, conserved regions, etc.) and the structural features of the SMILES string (such as atomic connection methods, functional group representations, etc.), but also gradually master the ability to distinguish between these two different "language" patterns through mixed input of the two "language" sequences within a batch, laying the foundation for subsequent cross-language understanding and interactive modeling.

[0044] 103. Optimize the initial prediction model based on the word prediction results at the target position and the actual word at the target position to obtain a pre-trained prediction model;

[0045] In this step, the training loss of the model is constructed by calculating the difference between the target word predicted by the model and the actual word at that position (e.g., using the cross-entropy loss function). Then, using the backpropagation algorithm, the parameters of the initial prediction model (such as attention weights, linear layer weights, and biases) are adjusted based on this loss value, so that the model's word prediction ability on the pre-training dataset is continuously improved. This optimization process can be iterated for multiple training epochs until the model's performance on the pre-training task reaches the preset convergence condition (e.g., the loss value no longer decreases significantly, the prediction accuracy on the validation set no longer improves, etc.). The model obtained at this point is the pre-trained prediction model. This pre-trained prediction model has initially acquired the ability to understand the general syntax and semantics of the two "languages" of amino acid sequences and molecular information sequences, and can capture the potential relationships between them.

[0046] Optionally, during pre-training, the AdamW optimizer can be used with an initial learning rate of 2e-5 and a linear learning rate decay strategy. The number of training epochs is dynamically adjusted according to the corpus size and model convergence, typically set to 10-20 epochs, to ensure that the model fully learns the universal representation of bilingual languages.

[0047] 104. Obtain a fine-tuning dataset, which includes the amino acid sequences of multiple target enzymes and the molecular information sequences of multiple paired molecules corresponding to each target enzyme; the multiple paired molecules include substrate molecules that interact with the target enzymes and non-substrate molecules that do not interact with the target enzymes;

[0048] This fine-tuning dataset is used to further fine-tune the model parameters of the aforementioned pre-trained prediction model. It can be obtained from experimental validation data, literature reports, or high-throughput screening results, and must contain detailed information for the specific target enzyme. The data sample corresponding to each target enzyme consists of its amino acid sequence and the molecular information sequences of multiple paired molecules. These paired molecules are clearly distinguished as substrate molecules (i.e., molecules known to catalyze or specifically bind to the enzyme) and non-substrate molecules (i.e., molecules determined not to interact with the enzyme). For example, for the target enzyme "trypsin," substrate molecules may include derivatives of its natural substrates such as arginine or lysine (e.g., benzoyl-L-arginine ethyl ester), while non-substrate molecules may be selected from molecules whose structure does not match the active site of trypsin (e.g., alanine methyl ester).

[0049] For example, experimentally validated enzymes can be obtained from the UniProt / Swiss-Prot database, and AI tools (such as ChatGPT) can be used to read the source literature corresponding to the enzyme's EC number (Enzyme Commission number) to screen out more reliable "golden enzymes." In other words, AI-assisted judgment is used to assess the strength of experimental evidence, clarity of description, and reliability of conclusions regarding the interaction between the enzyme and the reported substrate. This method successfully identifies a batch of "golden enzymes" with more precise annotation information and more sufficient experimental evidence, improving the overall annotation accuracy of the dataset by 5% compared to directly using the original annotations from the database.

[0050] Simultaneously, the SMILES strings of substrate molecules can be retrieved from the ChEBI database. Furthermore, for each real substrate of each enzyme, three times the number of molecules are randomly selected as negative samples. This resulted in the construction of a high-confidence dataset containing 160,984 enzyme-substrate pairs. Specifically, for each experimentally validated positive enzyme-substrate pair in the dataset (i.e., a known enzyme interacts with a known substrate), three molecules are randomly selected from a molecular library containing a large number of non-substrate molecules. These molecules may have some similarity in structure or chemical properties to the positive sample substrate, but are known not to interact with the enzyme. These are then used as the corresponding negative samples.

[0051] For example, if the true substrate of an enzyme E is S, then when constructing the dataset, in addition to retaining the positive sample (E, S), three molecules M1, M2, and M3 from the molecular library will be randomly selected to construct three negative samples: (E, M1), (E, M2), and (E, M3). In this way, the ratio of positive to negative samples in the final fine-tuning dataset reaches 1:3. This ratio provides the model with sufficient positive sample features to learn the correct enzyme-substrate interaction pattern, while introducing an appropriate number of negative samples enhances the model's ability to identify non-interaction scenarios, thereby improving the model's discriminative power and prediction accuracy during fine-tuning.

[0052] The selection of negative samples is not completely random. Instead, it avoids selecting molecules that are too different from the actual substrate structure or that are obviously impossible to be substrates. This ensures that the negative samples are "challenging" and encourages the model to learn more essential enzyme-substrate interaction characteristics, rather than simple structure matching.

[0053] 105. Input the fine-tuned dataset into the pre-trained prediction model so that for each target enzyme, the pre-trained prediction model determines whether there is an interaction between the target enzyme and each of its paired molecules based on the amino acid sequence of the target enzyme and the molecular information sequence of its paired molecules, and adjusts the model parameters of the pre-trained prediction model based on the determination result and the actual interaction label between the target enzyme and each of its paired molecules.

[0054] In this step, the bilingual general representation learned in the pre-training phase is transferred to a specific enzyme-substrate interaction judgment task, and the model parameters are fine-tuned to suit the characteristics of this specific task. During fine-tuning, the input to the pre-trained prediction model is a mixed sequence composed of the amino acid sequence of the target enzyme and the molecular information sequence of the paired molecule (such as the SMILES string) connected by <+> tokens, with a [CLS] token added at the beginning of the sequence. After the model encodes the entire mixed sequence, the hidden state corresponding to the [CLS] token is used to represent the comprehensive features of the enzyme-substrate pair.

[0055] Subsequently, this comprehensive feature is mapped to a binary classification space (i.e., [True] or [False]) through a linear classification layer, thereby outputting a judgment result indicating whether the target enzyme interacts with its paired molecule. Based on this judgment result and the true interaction label of the enzyme-substrate pair ([True] if interaction exists, [False] if not), the prediction loss is calculated using the binary cross-entropy loss function. Next, using the backpropagation algorithm, the parameters of the pre-trained prediction model (especially the upper-layer network parameters) are fine-tuned based on this loss value, allowing the model to gradually adapt to the specific task of enzyme-substrate interaction prediction and optimizing its ability to discriminate the interaction relationship between the enzyme and the substrate.

[0056] By iteratively training on fine-tuning datasets, the model can continuously refine its parameters, thereby more accurately capturing the correlation patterns between the amino acid sequence features of enzymes and the molecular structure features of substrates, ultimately improving the prediction accuracy of new and unseen enzyme-substrate interactions.

[0057] In some optional implementations, when inputting the fine-tuning dataset into the pre-trained prediction model, the amino acid sequences and molecular information sequences in the fine-tuning dataset can first be concatenated (e.g., using the <+> connector) to obtain fine-tuned concatenated sequences. A classification marker (e.g., [CLS] token) is added to the beginning position of each fine-tuned concatenated sequence, and a multilayer perceptron classification head is added to the output layer of the pre-trained prediction model. The fine-tuned concatenated sequence with the added classification marker is then input into the pre-trained prediction model for fine-tuning training. The classification marker is used to instruct the pre-trained prediction model to perform a task of determining whether there is an interaction between the target enzyme and its paired molecule for the fine-tuned concatenated sequence.

[0058] The [CLS] token is an abbreviation for "Classification," a special token commonly used in natural language processing tasks. It is usually added to the beginning of the input sequence and its function is to output a vector that represents the global information of the entire input sequence in the last layer of the model, so that it can be used for downstream tasks such as classification.

[0059] Specifically, during the fine-tuning phase, when the enzyme's amino acid sequence and the substrate's SMILES string are fed into the model as input, the special symbol [CLS] is first appended to the beginning of the entire input sequence. For example, assuming the enzyme's amino acid sequence is represented as "[ENZ]Seq1[SEP]" and the substrate's SMILES string is represented as "[SUB]Smiles1[SEP]" (where [ENZ] and [SUB] are special markers used to distinguish between the two languages, and [SEP] is the separator), then the complete input sequence after adding the [CLS] token becomes "[CLS][ENZ]Seq1[SEP]+[SUB]Smiles1[SEP]". Thus, when the model processes this input sequence, the [CLS] token undergoes the same attention mechanism and feature extraction process as all other tokens. Ultimately, in the model's output layer, the hidden state vector corresponding to the [CLS] token contains a comprehensive understanding of the entire enzyme-substrate sequence pair, which can be directly input into the subsequent MLP classification head to determine whether there is an interaction.

[0060] In other words, the task of model fine-tuning is to define enzyme-substrate interaction prediction as a binary classification problem. Its technical effectiveness is mainly reflected in the following aspects:

[0061] First, the introduction of the [CLS] token provides a comprehensive global representation of the entire enzyme-substrate sequence pair. During pre-training, the model primarily learns the syntax and semantics of the two "languages" (amino acid sequences and molecular information sequences) and their local dependencies within the context. In the fine-tuning phase, for binary classification prediction (interaction or non-interaction), a "sentence vector" or "discourse vector" that integrates the overall information of the enzyme sequence and substrate molecular information sequences is needed. The [CLS] token, in the final layer of the model, aggregates the key features of the entire input sequence (including enzyme and substrate information) to form a fixed-dimensional vector for classification.

[0062] Secondly, the addition of a Multilayer Perceptron (MLP) classifier head is crucial for adapting the general representation capabilities of the pre-trained model to specific downstream tasks. While the pre-trained prediction model already possesses the ability to understand two biomolecule sequences, its output is a representation of each token in the sequence or a preliminary global representation of [CLS], which is not directly applicable to binary classification. The MLP classifier head, through further nonlinear transformation and feature learning of the global representation of the [CLS] token, can map it to two category spaces: "interacting" and "non-interacting," thereby achieving the prediction function. The parameters of this classifier head are learned from scratch or randomly initialized during fine-tuning, ensuring that the model can be optimized for the specific task of enzyme-substrate interactions.

[0063] Finally, this approach maintains the integrity of the pre-trained model's main structure and the knowledge it has already learned, adapting to new tasks by adding only a small number of new parameters (classification heads). It is both efficient and makes full use of the advantages of pre-training, avoiding the high cost and overfitting risk of training a model from scratch.

[0064] In this embodiment, through joint learning of the two "languages"—amino acid sequences and molecular information sequences—during the pre-training phase, and parameter optimization for enzyme-substrate interaction judgment tasks during the fine-tuning phase, the prediction model constructed by this method can naturally integrate the amino acid sequence features of enzymes and the molecular information sequence features of substrates. When processing enzyme-substrate pairs, the model no longer analyzes enzymes or substrates in isolation, but rather models them as an organic whole. For example, the model can identify the spatial matching and chemical interaction patterns between specific combinations of amino acid residues in the enzyme's active pocket (such as catalytic triplet) and specific functional groups (such as hydroxyl and carboxyl groups) in the substrate molecule. This cross-modal feature fusion capability allows the model to understand the essential mechanism of enzyme-substrate interactions more deeply, thereby effectively overcoming the limitations of "monolingual" models when processing heterogeneous data, and significantly improving the generalization ability and prediction accuracy in enzyme-substrate interaction prediction tasks across enzyme families or with large evolutionary distances.

[0065] based on Figure 1 In one optional implementation of the illustrated embodiment, after fine-tuning the pre-trained prediction model, the fine-tuned pre-trained prediction model can be continuously trained. Specifically, a generated training dataset can be obtained, which includes the amino acid sequence of each target enzyme and the molecular information sequence of the target enzyme's substrate. This generated training dataset is input into the pre-trained prediction model with adjusted model parameters. The pre-trained prediction model then predicts the lexical units at the target positions in the amino acid sequence and / or molecular information sequence based on the lexical context in the generated training dataset. Furthermore, the pre-trained prediction model can be optimized based on the lexical unit prediction results at the target positions and the actual lexical units at the target positions to obtain the target prediction model.

[0066] The training dataset can be obtained by sequentially concatenating category terms, molecular cue terms, the molecular information sequence of the target enzyme's substrate, protein cue terms, and the amino acid sequence of the target enzyme for each target enzyme. This yields a sequence sample corresponding to the target enzyme. Based on at least one sequence sample corresponding to each of the multiple target enzymes, the training dataset can be determined. Category terms indicate that the current task is enzyme sequence generation, molecular cue terms guide the model to recognize the subsequent input molecular information sequence, and protein cue terms guide the model to recognize the subsequent input amino acid sequence.

[0067] For example, when constructing the training dataset, sequence samples can be built for each "golden enzyme" and its corresponding substrate molecule based on the aforementioned fine-tuning dataset. The format of the sequence samples can be: "Category token" + "Molecular cue token" + "Substrate SMILES string" + "Protein cue token" + "Enzyme amino acid sequence". Here, the "Category token" indicates that the current task is enzyme sequence generation, while the "Molecular cue token" and "Protein cue token" serve as guiding markers for the substrate SMILES string and enzyme amino acid sequence, respectively, helping the model accurately identify the semantic roles of different parts of the input sequence.

[0068] Furthermore, a continuous training strategy can be adopted, using a generated training dataset composed of the aforementioned multiple sequence samples to further pre-train the pre-trained prediction model. During training, a dynamic masking mechanism can be employed. For each batch of input sequences, BERT-style masks and GPT-style masks are applied in a 2:8 ratio. For BERT-style masks, 15% (or other percentages) of the tokens in the sequence are randomly selected for masking, and the model needs to predict the original masked token based on the context of the masked token.

[0069] For the GPT-style mask, which accounts for 80%, the process starts from the beginning of the sequence. All tokens before the "enzyme amino acid sequence" (including "category tokens", "molecular cue tokens", "substrate SMILES strings" and "protein cue tokens") are used as visible context. The tokens in the "enzyme amino acid sequence" are masked bit by bit. The model needs to start from the first token of the "enzyme amino acid sequence" and predict each subsequent token autoregressively based on all the preceding visible context information until the end of the "enzyme amino acid sequence".

[0070] This adjustment to the hybrid masking strategy focuses on enhancing the model's ability to generate sequences based on context, particularly for the generation of the "enzyme amino acid sequence" portion.

[0071] Optionally, regarding optimizer selection and parameter settings, the AdamW optimizer can be used during the continuous training phase, but the initial learning rate is adjusted to 1e-5, slightly lower than in the pre-training phase, to avoid excessive impact on the learned bilingual universal representation during training on new generation tasks. The learning rate decay strategy uses cosine annealing, and the number of training epochs is dynamically adjusted based on the model's generation performance on the validation set (e.g., the effectiveness of generated sequences, matching degree with the substrate, etc.), typically set to 5-10 epochs.

[0072] In each training iteration, after encoding the input sequence, the model captures the potential association between the substrate string SMILES and the enzyme amino acid sequence through a self-attention mechanism, and then generates the predicted token through the output layer. The loss function adopts cross-entropy loss, which calculates the loss between the predicted value and the true value for the masked token (whether it is a predicted token under BERT style or GPT style masking), and updates the model parameters through backpropagation.

[0073] During continuous training, the model learned the mapping rules from substrate molecular structure features to enzyme sequence features through a large number of paired samples of "substrate SMILES string → enzyme amino acid sequence". Specifically, when the input contains a prompt containing the substrate SMILES string, the model can automatically complete the subsequent enzyme amino acid sequence based on the common syntax and semantics of the two "languages" learned in the pre-training stage, as well as the understanding of enzyme-substrate interaction patterns in the fine-tuning stage. This completion is not a simple character stacking, but a "logical generation" performed by the model based on the chemical properties of the substrate (such as functional groups and molecular structure) and known sequence features of the enzyme (such as active sites and conserved regions).

[0074] For example, if the substrate is an ester molecule containing a specific hydroxyl group, the model may tend to include catalytic triplet amino acid residues (such as serine, histidine, and aspartic acid) related to ester bond hydrolysis when generating enzyme sequences. Unconditional generation means that after learning the overall distribution and intrinsic rules of a large number of enzyme sequences, the model can generate novel sequences that conform to the statistical characteristics of natural enzyme sequences (such as amino acid composition preferences and secondary structure tendencies).

[0075] Through this continuous training, the model not only consolidates the bilingual general grammar and semantic knowledge learned in the pre-training phase, but more importantly, it learns how to generate enzyme amino acid sequences with reasonable biological structures and potential functions, either conditioned on the substrate string "SMILES" or without specific substrate cues. For example, when the input contains a cue containing the specific substrate string "SMILES," the model can generate the amino acid sequence of an enzyme that may interact with it based on the molecular structural characteristics of that substrate; while without conditioned generation, the model can generate random enzyme sequences that conform to the basic characteristics of protein sequences based on the overall distribution and structural regularity of amino acid sequences it has learned. This continuous training process effectively integrates knowledge of enzyme-substrate interactions into the model's generative capabilities, enabling the target prediction model to possess the potential for targeted design from substrate to enzyme.

[0076] Therefore, during the model application process, when generating the enzyme amino acid sequence, the molecular information prompt sequence of the substrate molecule can be obtained. This molecular information prompt sequence is then input into the target prediction model, which can then generate the corresponding enzyme amino acid sequence based on this molecular information prompt sequence.

[0077] The molecular information cue sequence must at least contain the molecular information sequence of the substrate, and may optionally include category terms and molecular cue terms to clarify the task type and guide the model in recognizing substrate information. Upon receiving the molecular information cue sequence, the target prediction model, based on its internally learned knowledge of enzyme-substrate interactions and amino acid sequence generation rules, autoregressively generates each token of the enzyme's amino acid sequence until a complete enzyme amino acid sequence is generated. The generated enzyme amino acid sequence can serve as potential candidate enzymes capable of interacting with the substrate, providing direction for enzyme design and discovery.

[0078] In addition to using the molecular information prompt sequence of the substrate molecule as a prompt condition for generating the enzyme amino acid sequence, the desired functional description (text) of the enzyme, the active site structural fragment, or other molecular descriptors can also be used as prompt conditions for generation; no limitation is made here.

[0079] In other application scenarios, the target prediction model can be applied to enzyme-substrate interaction prediction and retrieval. Specifically, when performing enzyme-substrate interaction prediction and retrieval, the amino acid sequence of the enzyme to be tested and the molecular information sequence of at least one molecule to be tested can be obtained. The amino acid sequence of the enzyme to be tested and the molecular information sequence of the molecule to be tested are input into a pre-trained prediction model with adjusted model parameters. Based on the amino acid sequence of the enzyme to be tested and the molecular information sequence of the molecule to be tested, the pre-trained prediction model outputs an interaction index, which can be used to characterize the probability of an interaction between the enzyme to be tested and the molecule to be tested.

[0080] In the prediction scenario, the enzyme sequence and the SMILES string of the molecule to be tested can be input into the fine-tuned pre-trained prediction model according to the format. The model outputs a score between 0 and 1 based on the input sequence, which represents the probability that there is an interaction between the enzyme and the molecule to be tested.

[0081] Specifically, the pre-trained prediction model, with its parameters adjusted, encodes the input amino acid sequence of the enzyme and the sequence of the molecule to be tested. It then uses its internal self-attention mechanism to capture the potential correlation features between these two sequences. Subsequently, the model maps these correlation features to a binary classification task space, passes them through the sigmoid activation function of the output layer, and finally obtains a normalized probability value. A probability value closer to 1 indicates a higher probability that the model predicts an interaction between the enzyme and the molecule; conversely, a probability value closer to 0 indicates a lower probability of interaction. This probability output serves as an important reference for subsequent experimental verification or further screening, providing a quantitative indicator for the rapid identification of enzyme-substrate interactions.

[0082] In the retrieval scenario, for a given enzyme (or molecule), the fine-tuned pre-trained prediction model is used to calculate its interaction score with all candidate molecules (or enzymes) in the database in turn, and the Top-K results are returned in order of score, that is, the K candidate molecules (or enzymes) with the highest probability of interacting with the enzyme (or molecule).

[0083] Specifically, in the retrieval scenario, a database containing a large number of candidate molecule (or enzyme) sequence information is first constructed. Given a target enzyme, its amino acid sequence is combined with the molecular information sequence of each candidate molecule in the database to form enzyme-substrate sequence pairs. These pairs are then sequentially input into a fine-tuned pre-trained prediction model. The model processes each sequence pair and outputs the corresponding interaction index (i.e., score). After all candidate molecules have been calculated, the system sorts the candidate molecules according to their scores from highest to lowest, selecting the top K molecules as the candidate substrates with the highest probability of interacting with the target enzyme and returning them to the user. Similarly, given a target molecule, its molecular information sequence is combined with the amino acid sequence of each candidate enzyme in the database. The model calculates the interaction index and sorts the pairs, returning the Top-K most likely interacting candidate enzymes.

[0084] This predictive model-based search method can efficiently screen potential enzyme-substrate interaction pairs from massive databases, significantly reducing the blind spots and costs of traditional experimental screening. It provides powerful tools for the design of biocatalytic reactions, the construction of metabolic pathways, and the discovery of drug targets. For example, in the field of industrial biocatalysis, by inputting a target substrate molecule with a specific chemical structure, this search function can quickly find enzymes that may have catalytic activity against it from enzyme databases, providing precise candidates for enzyme screening and modification, and accelerating the development of new catalytic processes.

[0085] Figure 2 An exemplary flow illustrating an application scenario of the method of this application is shown in the figure. In the data preparation flow of part a of the figure, during the training data preparation stage, sequence data such as UniRef50 protein cluster and SMILES-250M molecular dataset can be obtained from the database to construct a bilingual corpus. During the model pre-training data preparation stage, the interaction data of the above-mentioned "golden enzyme" and its corresponding substrate molecules can be obtained from public databases or online resources of bioinformatics and cheminformatics such as Web (Web of Science), Swiss-Prot or ChEBI, thereby constructing a fine-tuning dataset containing enzyme-substrate pairing data.

[0086] In the model training process illustrated in section b, during the pre-training phase, the initial prediction model (e.g., Initial GLM) is pre-trained using biomolecular sequence data such as amino acid sequences and molecular information sequences from the aforementioned bilingual corpus. During pre-training, the protein amino acid sequences or substrate molecule molecular information sequences are randomly masked. The model learns the contextual information of the masked tokens to predict the original tokens, thereby mastering the general grammatical rules and semantic representations of protein and molecular sequences. After pre-training, the model enters the fine-tuning phase, where the parameters of the pre-trained model are adjusted using a constructed fine-tuning dataset, allowing the model to focus on learning specific patterns and features of enzyme-substrate interactions. Through this model training process, a pre-trained bilingual prediction model (e.g., Bilingual GLM) is obtained. In the diagram, S is the start marker of the sequence, E is the end marker, B = Begin (the beginning of the sequence segment), and CLS represents the classification task marker.

[0087] In the bilingual task flow shown in part c of the diagram, the prediction model (such as Enzyme-GLM) trained above can be used to perform the prediction task of enzyme-substrate interaction and output a binary prediction result, namely, whether there is an interaction or no interaction.

[0088] Therefore, the enzyme-substrate interaction prediction model trained by the method of this application embodiment possesses superior performance. In the enzyme-substrate interaction prediction task, this prediction model outperforms previous prediction models such as the ESP and PU-EPP models on the PU-EPP, ESP, Price, and Post-trained datasets, demonstrating stronger generalization ability. Especially in highly imbalanced bidirectional retrieval tasks, the prediction model of this application embodiment achieves a Top-100 accuracy improvement of 17.4% and 40.9% in substrate retrieval and enzyme retrieval, respectively, compared to published models, making it a more effective screening tool.

[0089] Secondly, through t-SNE visualization technology, the predictive model can effectively distinguish between amino acid sequences and SMILES strings, capturing their features at both the token and sequence levels. Simulated mutation experiments demonstrate that the model's decisions rely on an understanding of active sites and deep interaction patterns, exhibiting interpretability.

[0090] The t-SNE visualization refers to the visualization of high-dimensional feature vectors learned by the model in a two-dimensional space using the t-SNE (t-distributed random neighborhood embedding) algorithm. In the visualization results, sequences from different modalities (such as enzyme amino acid sequences and substrate SMILES strings) form relatively concentrated and well-defined clusters. This indicates that the model can accurately identify and distinguish these two different types of biomolecular sequences, and their respective feature representations have significant uniqueness. Simultaneously, within the same modality, sequences with similar structures or functions (such as enzyme sequences catalyzing the same type of reaction, or substrate SMILES strings with similar chemical structures) are relatively close in two-dimensional space. This further demonstrates that the model can not only distinguish different modalities but also capture the intrinsic structural similarities and functional correlations between similar sequences at the sequence level, thus verifying the effectiveness and depth of the model's sequence feature learning.

[0091] The prediction model in this application has strong functional extensibility. Based on the same bilingual model architecture, it can be extended to generate enzyme sequences through continuous, goal-oriented training. Experimental data show that the generated enzyme sequences are structurally close to the natural sequences, and computational predictions show that they have potentially strong interactions with the target substrate and excellent kinetic parameters. Furthermore, the model training does not rely on prior knowledge; the model directly learns the interaction patterns from enzyme sequences to molecular structures without relying on EC number annotations or 3D structural data, providing an effective new tool for annotating a large number of uncharacterized enzyme sequences.

[0092] In addition, the bilingual modeling concept of this application can be extended to other scenarios that require joint processing of two different modalities of biological information, such as protein-nucleic acid interaction prediction, drug-target interaction prediction (combining target protein sequences with drug molecule SMILES), and joint analysis of multi-omics data (such as transcriptome sequences and metabolite structures). It has broad cross-modal learning potential and provides new ideas and methods for the integration and analysis of complex multimodal data in the field of bioinformatics.

[0093] The computer device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 3 One embodiment of the computer device in this application includes:

[0094] The computer device 300 may include one or more central processing units (CPUs) 301 and a memory 305, wherein the memory 305 stores one or more applications or data.

[0095] The memory 305 can be volatile or persistent storage. The program stored in the memory 305 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 301 can be configured to communicate with the memory 305 and execute the series of instruction operations stored in the memory 305 on the computer device 300.

[0096] Computer device 300 may also include one or more power supplies 302, one or more wired or wireless network interfaces 303, one or more input / output interfaces 304, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0097] The central processing unit 301 can perform the aforementioned... Figure 1 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments are not described in detail here.

[0098] This application also provides a computer storage medium, one embodiment of which includes: the computer storage medium storing instructions, which, when executed on a computer, cause the computer to perform the aforementioned... Figure 1 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training a prediction model, characterized in that, The method includes: Obtain an initial prediction model and a pre-training dataset; the pre-training dataset includes amino acid sequences and molecular information sequences. The pre-trained dataset is input into the initial prediction model so that the initial prediction model predicts the lexical at the target position in the amino acid sequence and / or the molecular information sequence based on the lexical context in the pre-trained dataset. The initial prediction model is optimized based on the word prediction results at the target position and the actual words at the target position to obtain a pre-trained prediction model; A fine-tuning dataset is obtained, which includes the amino acid sequences of multiple target enzymes and the molecular information sequences of multiple paired molecules corresponding to each target enzyme; the multiple paired molecules include substrate molecules that interact with the target enzymes and non-substrate molecules that do not interact with the target enzymes. The fine-tuned dataset is input into the pre-trained prediction model so that, for each target enzyme, the pre-trained prediction model determines whether there is an interaction between the target enzyme and each of its paired molecules based on the amino acid sequence of the target enzyme and the molecular information sequence of its paired molecules, and adjusts the model parameters of the pre-trained prediction model based on the determination result and the actual interaction label between the target enzyme and each of its paired molecules.

2. The method according to claim 1, characterized in that, The method further includes: A training dataset is generated, which includes the amino acid sequence of each target enzyme and the molecular information sequence of the substrate of the target enzyme. The generated training dataset is input into the pre-trained prediction model with adjusted model parameters, so that the pre-trained prediction model predicts the lexical at the target position in the amino acid sequence and / or the molecular information sequence based on the lexical context in the generated training dataset. The pre-trained prediction model is optimized based on the word prediction results at the target position and the actual words at the target position to obtain the target prediction model.

3. The method according to claim 2, characterized in that, The process of obtaining the generated training dataset includes: For each target enzyme, the category term, molecular cue term, molecular information sequence of the substrate of the target enzyme, protein cue term, and amino acid sequence of the target enzyme are sequentially spliced ​​together to obtain the sequence sample corresponding to the target enzyme. The generated training dataset is determined based on at least one sequence sample corresponding to each of the multiple target enzymes; The category terminology is used to indicate that the current task is enzyme sequence generation, the molecular cue terminology is used to guide the model to identify the molecular information sequence portion of the subsequent input, and the protein cue terminology is used to guide the model to identify the amino acid sequence portion of the subsequent input.

4. The method according to claim 2, characterized in that, The method further includes: Obtain the molecular information prompt sequence of the substrate molecule, and input the molecular information prompt sequence into the target prediction model so that the target prediction model generates the corresponding enzyme amino acid sequence based on the molecular information prompt sequence.

5. The method according to claim 1, characterized in that, The process of obtaining the pre-trained dataset includes: Obtain the amino acid sequence and the molecular information sequence; For conflicting characters in the molecular information sequence that have the same form but different meaning as characters in the amino acid sequence, the conflicting characters are mapped to specific special form characters to distinguish the semantic differences between the same form characters in the amino acid sequence and the molecular information sequence.

6. The method according to claim 1, characterized in that, The initial prediction model predicts the lexical units at target positions in the amino acid sequence and / or the molecular information sequence based on the lexical context in the pre-training dataset, including: The words at the target position are masked, and the initial prediction model predicts the words at the target position based on the words in the positions before and after the target position. And / or, The initial prediction model exposes the lexical units of the positions preceding the target position to the initial prediction model, and the initial prediction model predicts the lexical units of the target position based on the lexical units of the positions preceding the target position.

7. The method according to claim 1, characterized in that, The step of inputting the fine-tuned dataset into the pre-trained prediction model includes: The amino acid sequences and molecular information sequences in the fine-tuning dataset are spliced ​​together to obtain the fine-tuned spliced ​​sequence; A classification label is added at the beginning of each of the fine-tuned splicing sequences, and a multilayer perceptron classification head is added to the output layer of the pre-trained prediction model. The fine-tuned splicing sequence with the added classification label is then input into the pre-trained prediction model. The classification marker is used to instruct the pre-trained prediction model to perform a task of determining whether there is an interaction between the target enzyme and the paired molecule on the fine-tuned spliced ​​sequence.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the amino acid sequence of the enzyme to be tested and the molecular information sequence of at least one molecule to be tested; The amino acid sequence of the enzyme to be tested and the molecular information sequence of the molecule to be tested are input into the pre-trained prediction model with adjusted model parameters, so that the pre-trained prediction model outputs an interaction index based on the amino acid sequence of the enzyme to be tested and the molecular information sequence of the molecule to be tested. The interaction index is used to characterize the probability that there is an interaction between the enzyme to be tested and the molecule to be tested.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images