Three-dimensional molecular text understanding method based on hybrid expert model and large language model

Abstract

The present invention discloses a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model, the three-dimensional molecular text understanding method comprising: constructing a new MoE layer of a hybrid expert model, the MoE layer comprising multiple experts, encoding task information text and text context through a cross-modal projector and inputting them into the routing of the hybrid expert model; training molecular encoders for processing different tasks, and integrating the trained molecular encoders into a unified 3D molecular encoder. The present invention introduces a text-assisted MoE layer to inject task information into routing input, thereby enhancing the model's ability to process specific tasks, and provides a general model that can simultaneously understand 2D, 3D molecules and various different molecular properties. By combining multiple molecular encoders, a comprehensive understanding of molecular characteristics is improved. By designing a molecule-text mapper, the deep fusion between molecular features and text is promoted, and the model's ability to understand three-dimensional molecular text is improved.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal biochemical molecular modeling, and more specifically to a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model. Background Art

[0002] The rapid development of language models (LM) has triggered a series of innovations in multiple disciplines. In particular, LM has performed well in text-based molecular understanding tasks, such as question-answering tasks in the fields of chemistry and medicine, and has demonstrated strong knowledge learning capabilities through pre-training on a large amount of biochemical literature. Recognizing the potential of LM in using extensive biochemical knowledge to complete molecular-related tasks, molecule-text modeling has gradually become an emerging research direction. Current methods mainly focus on combining text with 1D molecular sequences and 2D molecular graphs. For example, the application of contrastive learning methods enables LM to simultaneously learn input 2D graph data as well as molecular strings and text, which has promoted the development of tasks such as molecule-text retrieval and molecular description generation.

[0003] However, most of these studies have ignored 3D molecular structures, which are crucial in understanding molecular dynamics, protein-ligand interactions, enzyme functions, and other biomolecular phenomena. There are multiple problems in seamlessly integrating 3D molecular encoders into language models to achieve 3D molecule-text understanding. First, the problem of 3D molecule-text alignment, that is, it is difficult to map 3D molecular representations to an input space that the language model can understand; second, the problem of fine-tuning instructions based on 3D molecules, that is, how to adjust the model to better follow human instructions for tasks related to 3D molecules; third, the problem of unified encoders for multimodal molecules and tasks, that is, molecule-text understanding usually needs to handle queries covering different molecular properties (such as quantum chemical properties and drug-related properties), while understanding multiple molecular modalities (such as 1D, 2D, and 3D). However, current molecular encoders are typically pre-trained for only a single property or a single modality. This limitation hinders the ability to build a general model for molecule-text understanding. Fourth, the high computational cost, that is, directly integrating multiple molecular encoders in parallel into the 3D molecular language model will significantly increase the computational cost, and the computational cost will increase linearly with the increase in the number of encoders. At the same time, there are also challenges in effectively selecting the most relevant molecular encoders to adapt to specific molecular task requirements. Summary of the invention

[0004] In this embodiment, a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model is provided to solve the problem in the related art that 3D molecule-text alignment cannot be achieved, that is, it is difficult to map the 3D molecular representation to an input space that can be understood by the language model.

[0005] In a first aspect, an embodiment of the present invention provides a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model, the three-dimensional molecular text understanding method comprising:

[0006] A new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model;

[0007] Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder;

[0008] The input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data; the input 2D molecular graph is encoded using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph;

[0009] Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0010] The model is trained and used for the three-dimensional molecular text understanding task.

[0011] Optionally, a new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model, including:

[0012] Extract the original molecular markers from the 3D molecular structure, encode the task information text and text context through the cross-modal projector, and generate text context representation;

[0013] The textual context representation is provided as input to the routing along with the raw molecule tokens;

[0014] A probability function is designed to match an expert for each original molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the original molecular markers and focus on molecular coding.

[0015] Optionally, a probability function is designed to match an expert for each raw molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the raw molecular markers and focus on molecular coding, including:

[0016] set up For a group Experts, For molecular markers, is the representation of text context;

[0017] Define the MoE layer as:

[0018]

[0019] in, Molecular marker Assign to The probability of an expert; Molecular marker Assign to the first The probability corresponding to an expert is The molecular markers are input into experts; from The experts choose the one with the highest probability experts; is the gating weight of the route.

[0020] Optionally, molecular encoders for different tasks are trained and integrated into a unified 3D molecular encoder, including:

[0021] Pre-train the corresponding molecular encoder model through representative subtasks, including prediction of drug-related molecular properties, solubility-related properties, quantum chemical properties, and self-supervised learning;

[0022] Using the self-supervised learning model of the molecular encoder, we fine-tuned the other three tasks to obtain four specialized molecular encoder models;

[0023] Initialize the FFN weights of the 3D molecular encoder by copying the weights of the molecular encoder model;

[0024] The weights of the self-attention layer of the 3D molecular encoder are set by averaging the weights of the four molecular encoder models.

[0025] Optionally, the input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data, including:

[0026] make represents a molecule, where and Represent atomic nodes and their characteristics respectively, Represents the 3D coordinates of the node;

[0027] Each pair of atoms is represented by a 3D coordinate system Initialize the invariant spatial position encoding, which is derived from the Euclidean distance between atom pairs;

[0028] The representations of atoms and atom pairs are processed through a self-attention mechanism to generate molecular representations with 3D spatial information;

[0029] 3D Molecular Encoder Perform the molecular encoding process to obtain a representation of each atom:

[0030]

[0031] in, Indicates atoms, X is the molecule Representation.

[0032] Optionally, the molecule-text mapper includes a molecule transformer and a text transformer, and the molecule transformer and the text transformer share a self-attention layer;

[0033] The molecule Transformer is used to process molecule features, and the text Transformer is used to process text.

[0034] Optionally, the atom representation and the atom embedding representation are projected into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation, including:

[0035] Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate molecular labels, wherein the molecular labels include 2D molecular labels and 3D molecular labels;

[0036] A mixed tag sequence, including text tags, 2D molecular tags, 3D molecular tags and SMILES tags, is passed as input to a pre-trained language model for processing to generate a complete prompt representation.

[0037] Compared with the prior art, the three-dimensional molecular text understanding method based on the hybrid expert model and the large language model of the present invention has the following beneficial effects:

[0038] The present invention solves the problem that the traditional MoE layer is difficult to select the right expert by introducing a text-assisted MoE layer. By injecting task information into the routing input, the model's ability to handle specific tasks is enhanced, and a general model that can simultaneously understand 2D, 3D molecules and various molecular properties is provided. By combining multiple molecular encoders, it not only covers the structural information of molecules (such as the connection and spatial position between atoms), but also considers various physical and chemical properties of molecules, thereby improving the comprehensive understanding of molecular properties. By designing a molecule-text mapper (Q-Former), the representation of the 2D / 3D molecular encoder is effectively interacted with the learnable query tag through a cross-attention mechanism, which promotes the deep fusion between molecular features and text, and improves the model's ability to understand three-dimensional molecular text.

[0039] In a second aspect, an embodiment of the present invention provides a three-dimensional molecular text understanding method and device based on a hybrid expert model and a large language model, comprising:

[0040] An MoE layer construction module is used to construct a new MoE layer of a hybrid expert model, wherein the MoE layer includes multiple experts, and encodes the task information text and text context through a cross-modal projector and inputs them into the routing of the hybrid expert model;

[0041] 3D molecular encoder building module, which is used to train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder;

[0042] The encoding module is used to encode the input 3D molecular data using a 3D molecular encoder to obtain a representation of each atom; and to encode the input 2D molecular graph using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph;

[0043] A text output module, used for projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0044] The training module is used to train the model. The trained model is used for the three-dimensional molecular text understanding task.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0046] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional molecular text understanding method based on a hybrid expert model and a large language model as described in the first aspect.

[0047] Compared with the prior art, the beneficial effects of the three-dimensional molecular text understanding device, electronic device and storage medium based on a hybrid expert model and a large language model of the present invention are the same as the three-dimensional molecular text understanding method based on a hybrid expert model and a large language model described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a flowchart of a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model in an embodiment of the present invention;

[0050] Figure 2 This is a diagram of the pre-training process of the 3D molecular encoder in an embodiment of the present invention;

[0051] Figure 3 It is the overall architecture diagram of the model in the embodiment of the present invention;

[0052] Figure 4 It is a structural block diagram of a three-dimensional molecular text understanding device based on a hybrid expert model and a large language model in an embodiment of the present invention;

[0053] Figure 5 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] In an embodiment of the present invention, a three-dimensional molecular text understanding method based on a hybrid expert model and a large language model is provided, which can also be understood as proposing a molecular-text understanding framework or a molecular-text understanding model based on a hybrid expert model and a large language model. Figure 1 is a flow chart of the three-dimensional molecular text understanding method based on the hybrid expert model and the large language model of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0057] S100, constructing a new MoE layer of a hybrid expert model, wherein the MoE layer includes multiple experts, and encoding task information text and text context through a cross-modal projector and inputting them into the routing of the hybrid expert model;

[0058] It should be noted that the standard MoE layer contains multiple experts (i.e., feedforward neural network, FFN) and a router, which adaptively selects experts based on the input tags. In the 3D molecule-text understanding task, the router has difficulty selecting the correct expert because its input lacks task information. Figure 3As shown in Figure 2, the context of the 3D molecule-text understanding task usually contains task information. These text contexts are encoded by using a cross-modal projector and input to the routing together with the original molecule labels. Meanwhile, the expert only receives the original molecule labels and focuses on the molecule encoding. In this way, the routing decision is guided by both text features and molecular features without changing the expert's input space.

[0059] Specifically, in this embodiment, a new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model, including:

[0060] Extract the original molecular markers from the 3D molecular structure, encode the task information text and text context through the cross-modal projector, and generate text context representation;

[0061] The textual context representation is provided as input to the routing along with the raw molecule tokens;

[0062] A probability function is designed to match an expert for each original molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the original molecular markers and focus on molecular coding.

[0063] A probability function is designed to match an expert for each raw molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the raw molecular markers and focus on molecular coding, including:

[0064] set up For a group Experts, For molecular markers, is the representation of text context;

[0065] Define the MoE layer as:

[0066]

[0067] in, Molecular marker Assign to The probability of an expert; Molecular marker Assign to the first The probability corresponding to an expert is The molecular markers are input into experts; from The experts choose the one with the highest probability experts; is the gating weight of the route.

[0068] In this embodiment, setting , selecting the expert with the highest probability when processing each molecular marker. This method can fully consider the task information in the process of molecular marker processing, thereby effectively improving the performance of the 3D molecule-text understanding task.

[0069] S200, training molecular encoders for different tasks and integrating the trained molecular encoders into a unified 3D molecular encoder (three-dimensional molecular encoder);

[0070] A "branch training hybrid" strategy is adopted to integrate multiple molecular encoders (Uni-Mol) specializing in different tasks into a unified 3D molecular encoder (3D-UMEncoder).

[0071] Specifically, molecular encoders for different tasks are trained and integrated into a unified 3D molecular encoder, including:

[0072] The corresponding molecular encoder model is pre-trained through representative subtasks, such as Figure 2 As shown, the subtasks include prediction of drug-related molecular properties, prediction of solubility-related properties, prediction of quantum chemical properties, and self-supervised learning;

[0073] Using the self-supervised learning model of the molecular encoder, we fine-tuned the other three tasks to obtain four specialized molecular encoder models;

[0074] Initialize the FFN (Feed Forward Neural Network) weights of the 3D Molecular Encoder by copying the weights of the Molecular Encoder model;

[0075] The weights of the self-attention layer of the 3D molecular encoder are set by averaging the weights of the four molecular encoder models.

[0076] like Figure 3As shown, in this embodiment, the corresponding molecular encoder (Uni-Mol) model is pre-trained using the following representative molecular tasks: prediction of drug-related molecular properties; prediction of solubility-related properties; prediction of quantum chemical properties; self-supervised learning. To achieve this goal, the self-supervised learning model of the molecular encoder (Uni-Mol) is used, and the other three tasks are fine-tuned to finally obtain four specialized molecular encoder (Uni-Mol) models. Subsequently, we initialize the FFN weights of the 3D molecular encoder (3D-UMEncoder) by copying the weights of these specialized models. For the self-attention layer in the 3D molecular encoder (3D-UMEncoder), the weights of the four pre-trained molecular encoders (Uni-Mol) are averaged. It should be noted that this weighted averaging method is also applied to other modules, such as the node embedding module. Through this method, the 3D molecular encoder (3D-UMEncoder) can share knowledge in multiple molecular tasks while maintaining the specialization of each task, improving the multi-task learning ability of the model and providing greater versatility and flexibility for various molecular tasks.

[0077] S300, using a 3D molecular encoder to encode the input 3D molecular data (three-dimensional molecular data) to obtain a representation of each atom in the 3D molecular data; using a 2D molecular encoder (two-dimensional molecular encoder) to encode the input 2D molecular graph (two-dimensional molecular graph) to obtain an embedded representation of atoms in the 2D molecular graph;

[0078] Specifically, the input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data, including:

[0079] make represents a molecule, where and Represent atomic nodes and their characteristics respectively, Represents the 3D coordinates of the node;

[0080] Each pair of atoms is represented by a 3D coordinate system Initialize the invariant spatial position encoding, which is derived from the Euclidean distance between atom pairs;

[0081] The representations of atoms and atom pairs are processed through a self-attention mechanism to generate molecular representations with 3D spatial information;

[0082] 3D Molecular Encoder Perform the molecular encoding process to obtain a representation of each atom:

[0083]

[0084] in, Indicates atoms, X is the molecule Representation.

[0085] In this embodiment, the position code is derived based on the Euclidean distance between atom pairs and is robust to rotation or translation.

[0086] Furthermore, the 2D molecular encoder uses an encoder based on a graph neural network (GNN). Specifically, a five-layer graph isomorphism network (GINE) is used. Represents the 2D graph structure of the input molecule, using the given 2D graph encoder , we can get the embedded representation of the atom:

[0087]

[0088] in, Indicates Atoms in 2D molecular diagram Embedded in .

[0089] S400, projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0090] First of all, it should be noted that the molecular text mapper includes a three-dimensional molecular text mapper and a two-dimensional molecular text mapper. The molecular text mapper includes a molecular transformer and a text transformer. The molecular transformer and the text transformer share a self-attention layer.

[0091] The molecule transformer is used to process molecular features, and the text transformer is used to process text.

[0092] Furthermore, the text transformer follows the same architecture as the Bidirectional Representation Encoder (BERT), while the molecule transformer adds a cross-attention module between the self-attention and feed-forward modules to extract molecular features. Specifically, the molecule transformer maintains Given a 2D / 3D molecule input, the query token can interact with the representation of the 2D / 3D molecule encoder via a criss-cross attention module.

[0093] Furthermore, the language model adopts Galactica-1.3B as the backbone LM, which is extensively pre-trained on scientific corpora, enabling it to perform well in downstream tasks in scientific fields such as molecular description generation and molecular question answering.

[0094] In this embodiment, the atom representation and the atom embedding representation are projected into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation, including:

[0095] Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate molecular labels, wherein the molecular labels include 2D molecular labels and 3D molecular labels;

[0096] A mixed tag sequence, including text tags, 2D molecular tags, 3D molecular tags and SMILES tags, is passed as input to a pre-trained language model for processing to generate a complete prompt representation.

[0097] Specifically, 2D molecules and 3D molecules (i.e. and ) are input into the 2D molecule encoder and 3D molecule encoder respectively. These representations are then projected into the input space of LM through 2D and 3D molecule-text mappers to generate molecular labels and Next, the mixed tag sequence, including text tags, 2D and 3D molecular tags, and SMILES tags, is passed as input to the language model for processing. The complete prompt is represented as:

[0098] ,

[0099] in, Indicates system prompts and task-specific instructions, is the SMILES representation of the molecule, Indicates the length of the input sequence.

[0100] Based on the above, the language model (LM) is trained through the next token prediction task to generate text responses. Referring to the advanced research of MoE, we introduce a load balancing loss to ensure that the tokens are evenly distributed among experts. This auxiliary loss can be expressed as:

[0101]

[0102] in represents the number of experts, Indicates in batch Allocate to The proportion of experts’ marking Is the router in batch Middle The probability ratio assigned by experts, It is a batch The number of tokens in . The loss is generated by combining the autoregression of the language model , our total loss can be expressed as:

[0103]

[0104] in, is a hyperparameter that balances these two losses.

[0105] S500: training the model, and using the trained model for a three-dimensional molecular text understanding task.

[0106] In this embodiment, the training of the model is divided into three stages, and the goal of the first stage is to enhance the ability of the mapper to extract molecular features that are closely aligned with the corresponding text. At the same time, integrate 2D and 3D molecular representations as early as possible, and preliminarily train the ability of routing to assign tags. This process is achieved by jointly training 2D and 3D molecular encoders and mappers. Specifically, the routing is first trained for 15 rounds, and the other parameters of the 3D molecular encoder are kept frozen. The purpose of this stage is to allow the randomly initialized routing to adapt to the pre-trained 3D molecular encoder. In this process, the 2D molecular encoder and the 2D and 3D mappers are continuously trained to ensure comprehensive learning and integration. Subsequently, based on this checkpoint, the entire 3D molecular encoder, 2D molecular encoder, and 2D and 3D mappers are trained together for 35 rounds, focusing on the molecular-text retrieval task. Through this series of training steps, we can gradually achieve the effective fusion of 2D and 3D molecular representations and improve the performance of the model in tasks such as molecular-text retrieval.

[0107] In the second stage, the 2D molecule encoder and 3D molecule encoder as well as the 2D molecule text mapper and 3D molecule text mapper (i.e., Q-Former) are combined with LM. Based on the training in the first stage, our goal is to iterate the dataset and warm up the router and mapper. This process aims to make the router familiar with all molecules and prepare for the next stage of training. At the same time, the mapper is trained to map the molecular representation to the input space of the language model. Specifically, the other parameters of the 3D molecule encoder are frozen during the training process, and only the router is trained. During this process, the low-rank adapter (LoRA) tuning technique is also used to fine-tune the LM to enhance its ability to execute various instructions and generate text.

[0108] In the third stage, all datasets are used to fully train the 2D molecular encoder and 3D molecular encoder, 2D and 3D mappers (2D molecular text mapper and 3D molecular text mapper), and LM. Specifically, the 3D molecular encoder is fully trained to enhance its ability to encode 3D molecules and improve the ability of routing in assigning molecular labels to the top-k experts with the highest probability. At the same time, the 2D molecular encoder and 2D and 3D mappers continue to be trained to ensure better alignment and integration. In addition, in order to integrate a large amount of chemical knowledge from the 3D Molecular Center Instruction Tuning Dataset (3D-MoIT) and improve the ability of the language model to execute various instructions, the language model LM is fully fine-tuned. Through comprehensive training in this stage, the ability of each component to work together is further improved to ensure that the model performs well in a variety of molecular tasks, while being able to effectively understand and generate complex texts related to molecules.

[0109] The following example analyzes the advantages of the proposed molecule-text understanding framework (MTU-LLM) over other molecule-text understanding methods, such as the method using 1D molecular characters and 2D molecular graphs, in tasks such as molecule-text retrieval and molecular description generation.

[0110]

[0111] Table 1: Comparison of molecule-text retrieval performance of MTU-LLM and other methods on PubChem dataset

[0112] We first evaluated the performance of the MTU-LLM model, as well as Sci-BERT, KV-PLM, and MoMu on the molecule-text retrieval task on a downstream subset of the PubChem dataset, and evaluated the performance of the data within each batch of 64 samples and on the entire test set using the accuracy and recall @ 20 metrics. The experimental results are shown in Table 1. The MTU-LLM model significantly outperforms existing baseline models, including 1D molecule-aware molecular language models (such as Sci-BERT, KV-PLM) and 2D molecule-aware language models (such as MoMu-S and MoMu-K). This improvement can be attributed to two key factors. First, MTU-LLM outperforms MoMu (both of which are pre-trained on the same PubChem molecule-text pairs), indicating that the mapper achieves significant performance improvements through multi-objective pre-training across different granularities. This distinguishes the mapper from other models that primarily rely on molecule-text comparison learning objectives. Second, the scale of the PubChem dataset provides a clear advantage, which is demonstrated by The performance of MTU-LLM is verified. Our re-implemented MoMu model uses 301K molecule-text pairs in pre-training, while the original MoMu model only uses 15K pairs, which significantly improves the performance compared to the original model. The increased dataset size provides the MTU-LLM model with more diverse and rich training examples, enabling it to better capture the molecule-text relationship. Overall, the excellent performance of MTU-LLM emphasizes the effectiveness of 2D and 3D molecular representation learning in stage 1, and proves that the mapper is able to extract molecular features that are closely related to text descriptions;

[0113]

[0114] Table 2: Comparison of molecular description generation performance of MTU-LLM and other methods on PubChem dataset

[0115] Furthermore, we compared the performance of the MTU-LLM model with MolT5 and MoMu on the task of generating molecular descriptions on the PubChem dataset. The naming system of molecules is essentially closely related to the identification of molecular structures, including features such as hydrocarbon chains and benzene rings. Therefore, the model can accurately predict these names, demonstrating its ability to understand molecular structures. To this end, our PubChem dataset increases the complexity of the task by retaining the molecular names, positioning it as a task that combines molecular image generation and name prediction, and is evaluated by BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR. The experimental results are shown in Table 2. The MTU-LLM model far surpasses other models in all evaluation indicators. For example, it surpasses the second-place MoMu-Large, improving by 14.35 on BLEU-2, 15.42 on BLEU-4, and 15.4 on METEOR. These excellent results highlight the ability of this method to generate accurate molecular descriptions. This verifies that the model and training strategy can achieve accurate molecule-text understanding.

[0116] The embodiment of the present invention also provides a three-dimensional molecular text understanding method device based on a hybrid expert model and a large language model, and the device is used to implement the above method embodiment, which has been described and will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceived.

[0117] like Figure 4 As shown, Figure 4 : is a structural block diagram of a three-dimensional molecular text understanding method device based on a hybrid expert model and a large language model in the present invention, the device comprising:

[0118] The MoE layer construction module 101 is used to construct a new MoE layer of a hybrid expert model, wherein the MoE layer includes multiple experts, and encodes the task information text and text context through a cross-modal projector and inputs them into the routing of the hybrid expert model;

[0119] A 3D molecular encoder construction module 102 is used to train molecular encoders for processing different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder;

[0120] The encoding module 103 is used to encode the input 3D molecular data using a 3D molecular encoder to obtain a representation of each atom; and to encode the input 2D molecular graph using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph;

[0121] A text output module 104, configured to project the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0122] The training module 105 is used to train the model, and the trained model is used for the three-dimensional molecular text understanding task.

[0123] Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0124] A new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model;

[0125] Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder;

[0126] The input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data; the input 2D molecular graph is encoded using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph;

[0127] Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0128] The model is trained and used for the three-dimensional molecular text understanding task.

[0129] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0130] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0131] A new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model;

[0132] Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder;

[0133] The input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data; the input 2D molecular graph is encoded using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph;

[0134] Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation;

[0135] The model is trained and used for the three-dimensional molecular text understanding task.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional molecular text understanding method based on a hybrid expert model and a large language model, characterized in that: The three-dimensional molecular text understanding method comprises: A new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model; Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder; Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder, including: Pre-train the corresponding molecular encoder model through representative subtasks, including prediction of drug-related molecular properties, solubility-related properties, quantum chemical properties, and self-supervised learning; Using the self-supervised learning model of the molecular encoder, we fine-tuned the other three tasks to obtain four specialized molecular encoder models; Initialize the FFN weights of the 3D molecular encoder by copying the weights of the molecular encoder model; The weights of the self-attention layer of the 3D molecular encoder are set by averaging the weights of the four molecular encoder models; The input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data; the input 2D molecular graph is encoded using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph; Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation; The model is trained and used for the three-dimensional molecular text understanding task.

2. The three-dimensional molecular text understanding method based on a hybrid expert model and a large language model according to claim 1, characterized in that: A new MoE layer of a hybrid expert model is constructed, wherein the MoE layer includes multiple experts, and the task information text and text context are encoded by a cross-modal projector and input into the routing of the hybrid expert model, including: Extract the original molecular markers from the 3D molecular structure, encode the task information text and text context through the cross-modal projector, and generate text context representation; The textual context representation is provided as input to the routing along with the raw molecule tokens; A probability function is designed to match an expert for each original molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the original molecular markers and focus on molecular coding.

3. The three-dimensional molecular text understanding method based on a hybrid expert model and a large language model according to claim 2, characterized in that: A probability function is designed to match an expert for each raw molecular marker, and the top k experts with the highest probability are selected from all experts to process the molecular markers. The experts only receive the raw molecular markers and focus on molecular coding, including: set up For a group Experts, For molecular markers, is the representation of text context; Define the MoE layer as: in, Molecular marker Assign to The probability of an expert; Molecular marker Assign to the first The probability corresponding to an expert is The molecular markers are input into experts; from The experts choose the one with the highest probability experts; is the gating weight of the route.

4. The three-dimensional molecular text understanding method based on a hybrid expert model and a large language model according to claim 1, characterized in that: The input 3D molecular data is encoded using a 3D molecular encoder to obtain a representation of each atom in the 3D molecular data, including: make represents a molecule, where and Represent atomic nodes and their characteristics respectively, Represents the 3D coordinates of the node; Each pair of atoms is represented by a 3D coordinate system Initialize the invariant spatial position encoding, which is derived from the Euclidean distance between atom pairs; The representations of atoms and atom pairs are processed through a self-attention mechanism to generate molecular representations with 3D spatial information; 3D Molecular Encoder Perform the molecular encoding process to obtain a representation of each atom: in, Indicates atoms, X is the molecule Representation.

5. The three-dimensional molecular text understanding method based on a hybrid expert model and a large language model according to claim 1, characterized in that: The molecule-text mapper includes a molecule transformer and a text transformer, and the molecule transformer and the text transformer share a self-attention layer; The molecule Transformer is used to process molecule features, and the text Transformer is used to process text.

6. The three-dimensional molecular text understanding method based on a hybrid expert model and a large language model according to claim 1, characterized in that: The atom representation and atom embedding representation are projected into the input space of the pre-trained language model through a pre-built molecule text mapper to generate a complete prompt representation, including: Projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate molecular labels, wherein the molecular labels include 2D molecular labels and 3D molecular labels; A mixed tag sequence, including text tags, 2D molecular tags, 3D molecular tags and SMILES tags, is passed as input to a pre-trained language model for processing to generate a complete prompt representation.

7. A three-dimensional molecular text understanding device based on a hybrid expert model and a large language model, characterized in that: include: An MoE layer construction module is used to construct a new MoE layer of a hybrid expert model, wherein the MoE layer includes multiple experts, and encodes the task information text and text context through a cross-modal projector and inputs them into the routing of the hybrid expert model; 3D molecular encoder building module, which is used to train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder; Train molecular encoders for different tasks and integrate the trained molecular encoders into a unified 3D molecular encoder, including: Pre-train the corresponding molecular encoder model through representative subtasks, including prediction of drug-related molecular properties, solubility-related properties, quantum chemical properties, and self-supervised learning; Using the self-supervised learning model of the molecular encoder, we fine-tuned the other three tasks to obtain four specialized molecular encoder models; Initialize the FFN weights of the 3D molecular encoder by copying the weights of the molecular encoder model; The weights of the self-attention layer of the 3D molecular encoder are set by averaging the weights of the four molecular encoder models; The encoding module is used to encode the input 3D molecular data using a 3D molecular encoder to obtain a representation of each atom; and to encode the input 2D molecular graph using a 2D molecular encoder to obtain an embedded representation of atoms in the 2D molecular graph; A text output module, used for projecting the atom representation and the atom embedding representation into the input space of the pre-trained language model through a pre-built molecular text mapper to generate a complete prompt representation; The training module is used to train the model. The trained model is used for the three-dimensional molecular text understanding task.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the three-dimensional molecular text understanding method based on a hybrid expert model and a large language model is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional molecular text understanding method based on a hybrid expert model and a large language model as described in any one of claims 1 to 6 are implemented.

Images