Bioactive peptide model pre-training method and device
By fusing graph neural networks with large language models, a basic model of bioactive peptides is constructed, which solves the problem of low cross-modal alignment accuracy, enables efficient application in multi-task transfer, and improves the model's generalization performance and downstream task execution capability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-04-03
AI Technical Summary
Existing technologies struggle to effectively integrate the structural, sequence, and functional descriptions of bioactive peptides, resulting in low cross-modal alignment accuracy, poor model generalization ability, and difficulty in transferring to downstream tasks with small sample sizes.
We employ a method that integrates graph neural networks and large language models. We construct a basic model of bioactive peptides using a multimodal dataset, extract features using a path-aware dynamic graph attention network and a large language model, and perform self-supervised pre-training using a cross-modal Transformer to achieve the alignment and fusion of structural and semantic features.
It enables the efficient application of bioactive peptide models in multi-task transfer, improves cross-modal alignment accuracy and robustness, enhances the generalization performance of the model in small sample scenarios, and supports a variety of downstream tasks such as bioactivity prediction and novel peptide generation.
Smart Images
Figure CN121789792A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of artificial intelligence and bioinformatics, and in particular relates to a method and apparatus for pre-training a bioactive peptide model. Background Technology
[0002] Bioactive peptides are short-chain amino acid sequences with specific biological functions, such as antimicrobial peptides, antioxidant peptides, umami peptides, and antitumor peptides, which are of great value in the pharmaceutical, food, and cosmetic industries. Traditional research relies on experimental screening and molecular simulation, which is costly and time-consuming. In recent years, artificial intelligence technology has provided a new paradigm for peptide function prediction and design. Graph Neural Networks (GNNs) are adept at processing molecular graph structure data and can effectively model the chemical bonds, spatial distances, and topological relationships between amino acid residues, thereby capturing the three-dimensional conformational features of peptides. However, GNNs have difficulty understanding long-range dependencies in sequences and semantic information in functional annotation texts. On the other hand, large language models, represented by BERT and GPT, have shown strong capabilities in processing amino acid sequences and scientific literature, and can extract contextual semantic features from sequences and texts. However, LLMs are essentially sequence models and cannot directly process non-Euclidean graph structure data. Although some studies have attempted to combine GNNs and LLMs for protein or multimodal drug discovery, a dedicated, deep pre-training framework that integrates graph structure and sequence-text semantics is still lacking for the specific object of bioactive peptides. Existing methods generally suffer from the following problems:
[0003] Separate modeling of images and text fails to achieve fine-grained alignment; simple splicing or shallow fusion leads to cross-modal semantic gaps; there is a lack of self-supervised task design tailored to peptide characteristics; and it is difficult to transfer to downstream tasks with small sample sizes (such as the generation of novel antimicrobial peptides). Therefore, there is an urgent need for a pre-trained base model that can efficiently integrate the structural, sequence, and functional description information of peptides and support multi-task transfer. Summary of the Invention
[0004] The purpose of this invention is to provide a pre-training method and device for a bioactive peptide basic model based on the fusion of graph neural networks and large language models, which solves the problems of insufficient multimodal information fusion, low cross-modal alignment accuracy, and poor model generalization ability in the prior art, and constructs a transferable, scalable, and high-performance general basic model for bioactive peptides.
[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:
[0006] This invention relates to a method and apparatus for pre-training a bioactive peptide model, comprising the following steps:
[0007] S101. Collect graph structure data and sequence description data of bioactive peptides to form a multimodal dataset;
[0008] S102. Process the graph structure data using a graph neural network to extract structural feature vectors;
[0009] S103. Process the sequence description data using a large language model to extract semantic feature vectors;
[0010] S104. Align the structural feature vector and the semantic feature vector to the same embedding space through contrastive learning to obtain the aligned multimodal features;
[0011] S105. Input the aligned multimodal features into the fusion model for pre-training to construct a basic model of bioactive peptides;
[0012] S106. Apply the bioactive peptide basic model to downstream tasks through transfer learning to obtain specific task results.
[0013] The present invention is further configured such that, in step S101, molecular structure data of bioactive peptides are obtained from a public database and converted into graph structure data, while the corresponding amino acid sequences, functional annotation texts and experimental metadata are obtained as sequence description data; and the obtained data are cleaned, labeled and enhanced to form a balanced multimodal dataset.
[0014] The present invention is further configured such that, in step S102, a path-aware dynamic graph attention network is used to perform message passing and aggregation on the graph structure data to generate node-level embeddings, and the node-level embeddings are converted into graph-level structure feature vectors through a global pooling operation.
[0015] The present invention is further configured such that, in step S103, the sequence description data is segmented and encoded, input into a pre-trained large language model, and the contextual semantic features of the sequence and text are extracted by the Transformer encoder to generate a semantic feature vector.
[0016] The present invention is further configured such that, in step S104, positive and negative sample pairs are constructed, wherein the positive samples are the structural and semantic features of the same bioactive peptide, and the negative samples are the features of different peptides; the projection head is optimized using an iteratively optimized and enhanced InfoNCE loss function to achieve cross-modal alignment.
[0017] The present invention is further configured such that, in step S105, a cross-modal Transformer is used as a fusion model to design a self-supervised pre-training task, including masked language modeling, graph node prediction and cross-modal reconstruction;
[0018] We introduce a cross-modal alignment framework inspired by the idea of image and text contrastive learning. It combines a large language model dedicated to peptide sequence modeling with a graph neural network based on graph sampling aggregation convolution. Through a learnable projection head, the embedding vectors of the two modalities are mapped to a unified shared semantic space, and the model is optimized using a symmetric cross-entropy loss function.
[0019] The present invention is further configured such that, in step S106, a task-specific header is added for the downstream task, and a small amount of labeled data is used for fine-tuning to adapt to specific task requirements, such as tasks such as biological activity prediction, sequence generation or structure optimization.
[0020] A bioactive peptide model pre-training device, characterized in that it comprises:
[0021] The data preparation unit is used to perform step S101 in claim 1;
[0022] A structural feature extraction unit is used to perform step S102 in claim 1;
[0023] A semantic feature extraction unit is used to perform step S103 in claim 1;
[0024] A feature alignment unit is used to perform step S104 in claim 1;
[0025] A pre-trained unit for performing step S105 of claim 1;
[0026] An application unit is used to perform step S106 of claim 1.
[0027] A storage medium comprising stored instructions, wherein when the instructions are executed, the device containing the storage medium is controlled to perform the pre-training method for the bioactive peptide basic model.
[0028] An electronic device includes a memory and one or more instructions, wherein one or more instructions are stored in the memory, and is configured with one or more processors to execute a pre-training method for a bioactive peptide basic model.
[0029] The present invention has the following beneficial effects:
[0030] 1. This invention achieves unified representation learning of three types of heterogeneous data: molecular diagram structure, amino acid sequence, and functional description text. It breaks through the information limitations of traditional single-modal models, enabling the model to simultaneously perceive the three-dimensional spatial conformation, sequence evolution law, and biological semantic connotation of peptides, thereby gaining a more comprehensive and essential molecular understanding.
[0031] 2. This invention introduces a path-aware dynamic graph attention network, which integrates prior knowledge of biological pathways such as KEGG into the graph message passing process, significantly enhancing the ability to model the interactions of key residues. By adopting an iteratively optimized and enhanced InfoNCE loss function and a CLIP-inspired contrastive learning framework, it effectively narrows the distance between the structural and semantic features of the same peptide in the embedding space, while widening the distance between features of different peptides, thus greatly improving the accuracy and robustness of cross-modal alignment.
[0032] 3. This invention includes a multi-task self-supervised pre-training strategy of masked language modeling, graph node prediction, and cross-modal reconstruction. It adopts a three-stage training paradigm of "single-modal pre-training, joint comparative fine-tuning combined with instruction optimization". It not only makes full use of massive unlabeled peptide data, but also enhances the model's understanding and response to natural language instructions through high-quality PeptideQA questions and answers generated by GPT-4o, thereby improving the model's generalization performance in small sample or even zero sample scenarios.
[0033] 4. The basic model constructed in this invention has strong transfer capabilities. It can efficiently support a variety of downstream tasks by simply adding a lightweight task head and fine-tuning it with a small amount of labeled data. These tasks include high-precision bioactivity prediction, targeted generation of novel functional peptide sequences, and optimization of peptide molecule three-dimensional structure folding. In actual tests, all indicators are significantly better than existing single-modality or simple fusion methods.
[0034] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.
[0036] Figure 1 A flowchart of a pre-training method for a bioactive peptide basic model provided in an embodiment of the present invention;
[0037] Figure 2 This is a schematic diagram of the data extraction and cross-modal alignment process;
[0038] Figure 3 A schematic diagram for designing a self-supervised pre-training task;
[0039] Figure 4 This is a schematic diagram illustrating the migration of downstream tasks.
[0040] Figure 5 This is a structural block diagram of the pre-training device provided in an embodiment of the present invention;
[0041] Figure 6 This is a hardware structure diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0043] Please see Figure 1-4 The present invention provides a pre-training method for a bioactive peptide model, comprising the following steps:
[0044] S101. Constructing a multimodal dataset: Obtain molecular structure data, amino acid sequences, functional annotation text, and experimental metadata of bioactive peptides from public databases; convert the molecular structure data into graph structure data using tools such as RDKit, where nodes represent amino acid residues and edges represent chemical bonds or spatial distances; clean, label, and enhance the original data to form a balanced multimodal dataset.
[0045] S102. Extract structural feature vectors and process graph structure data using a path-aware dynamic graph attention network: Prior knowledge of biological pathways is introduced during message passing to dynamically adjust attention weights; the update formula is as follows:
[0046]
[0047] in, Attention coefficient;
[0048] Fusion path weight factor path-emb ij The path embedding between nodes i and j is extracted from a biological pathway database; the node-level embeddings are aggregated into a graph-level structure feature vector g∈R using global average pooling. d ;
[0049] S103. Extracting Semantic Feature Vectors: The sequence description data, including the amino acid sequence "ACDEFG…" and the functional text "This peptide has potent antibacterial activity…", is segmented and encoded, and input into a large language model similar to PeptideBERT. This LLM is based on the Transformer architecture and pre-trained on a large-scale peptide sequence corpus, possessing the ability to deeply understand the semantics of biological sequences. The [CLS] token or the sequence mean is taken as the final semantic feature vector t∈R. d .
[0050] S104. Cross-modal feature alignment: Construct a contrastive learning framework to align structural features g and semantic features t to the same embedding space.
[0051] Positive sample pairs: those from the same peptide (g i ,t i );
[0052] Negative sample pairs: from different peptides (g i ,t j ),j≠i;
[0053] Introducing a CLIP-inspired contrastive learning mechanism: mapping g and t to a shared semantic space using two independent projection heads;
[0054] An enhanced InfoNCE loss function using iterative optimization:
[0055]
[0056] Where sim is the cosine similarity, τ is the temperature parameter, and Ω iter This is an iterative optimization term based on historical negative samples, which improves alignment stability.
[0057] S105. Pre-training the fusion model: Input the aligned multimodal features into the cross-modal Transformer fusion model and perform the following self-supervised pre-training tasks:
[0058] Masked Language Modeling (MLM): Randomly mask 15% of the amino acid or text tokens, which are then reconstructed by the model;
[0059] Graph node attribute prediction: Mask partial node features to predict their original type or physicochemical properties;
[0060] Cross-modal reconstruction: reconstructing sequences using only graph features, or reconstructing graph structures using only sequences;
[0061] Joint optimization using symmetric cross-entropy loss function:
[0062]
[0063] Where, p g and p t , where are the probability distributions of the outputs of GNN and LLM, respectively, and H is the cross-entropy;
[0064] Implement a two-stage pre-training strategy:
[0065] (1) Each modality is pre-trained independently, and binary cross-entropy is used to predict peptide activity properties;
[0066] (2) Joint comparison and fine-tuning, using the above comparison and reconstruction tasks;
[0067] Further, an instruction tuning stage is introduced: a ProteinQA-like bioactive peptide question-and-answer dataset is constructed, containing sequence, structure, and function summaries as well as question-and-answer pairs. High-quality QA samples are generated using GPT-4o, and the model is fine-tuned to improve its understanding and response capabilities to bioactive queries.
[0068] S106. Downstream task transfer application: Based on the pre-trained model, add lightweight task headers for specific tasks.
[0069] Activity prediction task: Add a fully connected classifier for umami intensity scoring, anticancer activity assessment, etc.
[0070] Sequence generation task: Connect to an autoregressive decoder to generate new peptide sequences with specific functions, such as highly active antimicrobial peptides;
[0071] Structural optimization task: Predicting the folding conformation of peptides or optimizing the side chain arrangement;
[0072] Efficient migration can be achieved by using a small amount of labeled data for end-to-end fine-tuning.
[0073] Example 1: Data Preparation and Feature Extraction
[0074] Download 10,000 peptide structure files with known activities from the PDB, and use RDKit to parse them into molecular graphs. The nodes represent 20 standard amino acids, and the edges include covalent bonds and Cα–Cα distances. Non-bonded interactions were observed. Corresponding sequences and functional descriptions were obtained from UniProt and BIOPEP-UWM. After deduplication and quality filtering, 8,500 high-quality samples were obtained.
[0075] The graph structure is input into a SAGEConv-based GNN (2 layers, 512 hidden dimensions), combined with a path-aware dynamic GAT mechanism, to output graph-level features g. The sequence and text are concatenated and input into a PeptideBERT-like LLM (12-layer Transformer, vocabulary containing 20 amino acids + special tokens + natural language words), to output semantic features t.
[0076] Example 2: Cross-modal alignment and pre-training
[0077] A contrastive learning task with a batch size of 256 was constructed. The projector was a 2-layer MLP (512→256). The temperature parameter τ = 0.07, and the iterative optimization term weight λ = 0.1. Pre-training was performed on an 8×A100 GPU with a total of 200,000 steps.
[0078] Weighted portfolio of losses from self-supervised tasks:
[0079]
[0080] Example 3: Downstream Task Application
[0081] Umami peptide activity prediction: Fine-tuned on the UMAMI dataset, the accuracy reached 92.3%, which is better than GNN alone (85.1%) or LLM (87.6%).
[0082] Antimicrobial peptide generation: Given the instruction of "high activity and low toxicity", the model generates 100 new sequences, and 78% of them have potential activity as verified by AMP Scanner;
[0083] Structure folding prediction: RMSD error was reduced by 18% on the CASP-like peptide test set.
[0084] Please see Figure 5 This embodiment also provides a pre-training device for a bioactive peptide basic model, which includes a data preparation unit, a structural feature extraction unit, a semantic feature extraction unit, a feature alignment unit, a pre-training unit, and an application unit.
[0085] The data preparation unit automatically acquires peptide structure, sequence, and functional annotation data from public databases such as PDB, UniProt, and BIOPEP-UWM, uses tools like RDKit to convert the 3D structure into graph data, and performs cleaning, annotation, and enhancement operations to form a high-quality multimodal dataset. The structural feature extraction unit deploys a hybrid architecture based on SAGEConv and a path-aware dynamic graph attention network, combining residue path embedding information extracted from the KEGG pathway database to dynamically adjust attention weights during message passing, ultimately generating graph-level structural feature vectors through global pooling. The semantic feature extraction unit loads a PeptideBERT-like large language model pre-trained on large-scale peptide sequences to extract the structure and function of the spliced amino acid sequences. The text is encoded to output context-aware semantic feature vectors. The feature alignment unit maps structural and semantic features to a shared embedding space through two independent projection heads and uses the InfoNCE loss function, which includes momentum queues and iterative optimization mechanisms, to achieve high-precision cross-modal alignment. The pre-training unit integrates a cross-modal Transformer backbone network, supports self-supervised tasks such as masked language modeling, graph node prediction, and cross-modal reconstruction, and has two-stage training scheduling and instruction fine-tuning capabilities. It can automatically save model checkpoints and support breakpoint resume training. The application unit provides a flexible task adaptation interface, which can dynamically load classifiers, regression heads, or sequence generation decoders according to downstream needs, and output various results, including activity scores, new peptide sequences, or three-dimensional structures. It can also be encapsulated as a RESTful API for external systems to call.
[0086] Please see Figure 6 The electronic device of the present invention includes a memory, one or more processors, and a communication interface. The memory stores instructions that, when executed by the processor, cause the device to perform all the steps of the aforementioned method. The storage medium contains the instructions and controls the device to execute the method of the present invention. The device may be a workstation, a server, or a cloud virtual machine, and is not limited to a specific hardware form.
[0087] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
Claims
1. A method for pre-training a bioactive peptide model, characterized in that, Includes the following steps: S101. Collect graph structure data and sequence description data of bioactive peptides to form a multimodal dataset; S102. Process the graph structure data using a graph neural network to extract structural feature vectors; S103. Process the sequence description data using a large language model to extract semantic feature vectors; S104. Align the structural feature vector and the semantic feature vector to the same embedding space through contrastive learning to obtain the aligned multimodal features; S105. Input the aligned multimodal features into the fusion model for pre-training to construct a basic model of bioactive peptides; S106. Apply the bioactive peptide basic model to downstream tasks through transfer learning to obtain specific task results.
2. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S101, molecular structure data of bioactive peptides are obtained from public databases and converted into graph structure data. At the same time, the corresponding amino acid sequences, functional annotation texts and experimental metadata are obtained as sequence description data. The obtained data are cleaned, labeled and enhanced to form a balanced multimodal dataset.
3. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S102, a path-aware dynamic graph attention network is used to perform message passing and aggregation on the graph structure data to generate node-level embeddings, and then the node-level embeddings are converted into graph-level structure feature vectors through a global pooling operation.
4. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S103, the sequence description data is segmented and encoded, input into a pre-trained large language model, and the contextual semantic features of the sequence and text are extracted by the Transformer encoder to generate a semantic feature vector.
5. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S104, positive and negative sample pairs are constructed, where positive samples are the structural and semantic features of the same bioactive peptide, and negative samples are the features of different peptides. The projection head is optimized using an iteratively optimized and enhanced InfoNCE loss function to achieve cross-modal alignment.
6. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S105, a cross-modal Transformer is used as the fusion model, and a self-supervised pre-training task is designed, including masked language modeling, graph node prediction, and cross-modal reconstruction. We introduce a cross-modal alignment framework inspired by the idea of image and text contrastive learning. It combines a large language model dedicated to peptide sequence modeling with a graph neural network based on graph sampling aggregation convolution. Through a learnable projection head, the embedding vectors of the two modalities are mapped to a unified shared semantic space, and the model is optimized using a symmetric cross-entropy loss function.
7. The pre-training method for bioactive peptide models according to claim 1, characterized in that, In step S106, a task-specific header is added for the downstream task, and a small amount of labeled data is used for fine-tuning to adapt to the specific task requirements, such as biological activity prediction, sequence generation, or structure optimization.
8. An apparatus for use in the pre-training method for bioactive peptide models according to any one of claims 1-7, characterized in that, include: The data preparation unit is used to perform step S101 in claim 1; A structural feature extraction unit is used to perform step S102 in claim 1; A semantic feature extraction unit is used to perform step S103 in claim 1; A feature alignment unit is used to perform step S104 in claim 1; A pre-trained unit for performing step S105 of claim 1; An application unit is used to perform step S106 in claim 1.
9. A storage medium, characterized in that, The storage medium includes stored instructions, which, when executed, control the device containing the storage medium to perform the pre-training method for the bioactive peptide basic model as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes a memory and one or more instructions, wherein one or more instructions are stored in the memory, and one or more processors are configured to execute the bioactive peptide basic model pre-training method as described in any one of claims 1 to 7.