Method for biomedical hypothesis generation representation learning based on hybrid expert model

Abstract

This invention belongs to the field of biomedical and information technology, and relates to a method for biomedical hypothesis generation representation learning based on a hybrid expert model. First, the GraphSAGE model is used to aggregate the neighbor information of each entity to obtain an initial representation of the entity. Then, the rich interaction modes between the two entities in each entity pair are deeply mined. Finally, a hybrid expert model is used to learn multiple interaction features of the entity pair to improve the model's feature representation capability. This invention is mainly used for biomedical hypothesis generation research tasks in the field of natural language processing.

Description

Technical Field

[0001] This invention belongs to the field of biomedical and information technology, and relates to a method for biomedical hypothesis generation representation learning based on a hybrid expert model. Background Technology

[0002] With the rapid development of biomedicine and information technology, the number of biomedical documents has grown rapidly, forming an unprecedented treasure trove of knowledge. Some seemingly isolated documents may conceal undiscovered connections, which, once revealed, could become the fulcrum for new scientific discoveries. However, individual researchers' knowledge alone is insufficient to penetrate such a vast ocean of text. Biomedical hypothesis generation technology was proposed against this backdrop: its goal is to automatically extract existing knowledge from scattered documentary evidence and deduce implicit propositions not yet explicitly stated by humans, providing actionable scientific hypotheses for subsequent experimental verification, thereby accelerating the systematic production of new knowledge.

[0003] Traditional hypothesis generation methods primarily rely on the co-occurrence frequency of entities in literature as a proxy for association strength. This approach assumes that the more frequently two entities appear in the same document, the more likely they are to be associated. However, high-frequency co-occurrence does not equate to biological significance; it may simply be statistical noise or a domain hotspot effect, leading to many false associations being mistakenly identified as potential discoveries. In recent years, machine learning-based models have significantly improved the accuracy of association mining, but they generally assume that the biomedical knowledge space is static and invariant, ignoring the dynamic nature of the continuously evolving domain knowledge. In fact, entity semantics and association strength evolve non-stationarily over time; this dynamic trajectory itself contains crucial information for predicting future associations, yet it has not been adequately modeled by early models.

[0004] To capture the temporal evolution of entity semantics, recent research has begun to introduce temporal representation learning frameworks, representing entities as continuously updated vector sequences over time. Building upon this, Recurrent Neural Networks (RNNs) and their variants are used to characterize the dynamic trajectories of entity pair relationships, capturing the evolutionary trends of entity associations by modeling semantic differences between adjacent time steps. To further learn the direct dependencies between any two time steps in a time series, some research has introduced attention mechanisms. These methods focus on the temporal evolution information of entity pairs, i.e., the smoothness and differences in entity pair embeddings between any two adjacent time steps, thereby capturing the temporal evolution process of entity pairs.

[0005] However, current work primarily focuses on the temporal continuity of entity pair relationships, lacking in-depth exploration of their complex and diverse interaction patterns. Biomedical hypothesis generation is an important research direction in the field of bioinformatics. Its core objective is to infer and generate scientific hypotheses with potential research value that have not yet been experimentally verified, based on existing knowledge such as massive biomedical literature and experimental data. Biomedical entities are the basic semantic units constituting hypotheses, encompassing elements such as diseases, genes, and drugs, and are the core research objects of hypothesis generation. The essence of hypothesis generation is to uncover the potential relationships between two biomedical entities, which is of great significance for advancing further clinical experimental research. In current biomedical hypothesis generation research, researchers simply represent an entity pair as a concatenation of the embeddings of the two entities. It is worth noting that the embedding representation of entity pairs has multiple construction methods besides simply concatenating the two entity embeddings. How to explicitly model and integrate these diverse interactions, and thus construct a dynamic association prediction framework that can integrate various interaction features, is a core scientific problem that urgently needs to be addressed in current biomedical hypothesis generation research. Summary of the Invention

[0006] Given the shortcomings of existing methods in feature mining and modeling of biomedical entity pairs, this invention proposes a biomedical hypothesis generation representation learning method based on a hybrid expert model. By utilizing a hybrid expert model, the co-occurrence relationships of biomedical entity pairs in the biomedical hypothesis generation dataset are learned, and multiple entity pair interaction feature representations are aggregated to predict the future co-occurrence relationships of biomedical entity pairs.

[0007] This invention utilizes a Mixture of Experts (MoE) model to learn representations of entity pairs in biomedical literature for use in biomedical hypothesis generation datasets. First, the GraphSAGE model is used to aggregate neighbor information for each entity, obtaining an initial representation. Then, the rich interaction patterns between the two entities in each pair are deeply mined. Finally, a Mixture of Experts model is employed to learn various interaction features of entity pairs, thereby enhancing the model's feature representation capabilities. This invention is primarily intended for biomedical hypothesis generation research tasks within the field of natural language processing.

[0008] The technical solution of this invention: The biomedical hypothesis generation representation learning method based on a hybrid expert model comprises the following steps: Step 1: Learn initial representations of biomedical entity pairs based on graph sampling aggregation networks. The GraphSAGE graph sampling aggregation network model is used to learn the initial embeddings of biomedical entities in the dataset. At different time steps of the dataset's time series graph, the initial representation of each entity at each time step is obtained by aggregating the neighbor information of each entity at the current time step.

[0009] Step 2: Feature Capture of Biomedical Entity Pairs Based on Multiple Fine-Grained Interaction Modes For each pair of biomedical entities, six ways in which they interact are obtained: concatenation of two entity embeddings, entity 1 embedding, entity 2 embedding, addition of two entity embeddings, subtraction of two entity embeddings, and multiplication of two entity embeddings. These fine-grained entity interaction methods can not only reveal the correlation and differences between entity pair embeddings, but also deeply capture the fusion feature representation of entity pair embeddings.

[0010] Step 3: Learning complex interaction feature representations of biomedical entity pairs based on a hybrid expert model At different time steps of the dataset's time series graph, various interaction features are calculated for the initial embeddings of the two entities in a biomedical entity pair. A linear layer is added after each feature, serving as multiple expert subnetworks in a hybrid expert model. Since the concatenation operation preserves relatively complete information about the two entities, this invention chooses to use the concatenation operation to obtain expert weights. The concatenated embeddings of the two entities are input into a gating network and mapped to a vector space of the same dimension as the total number of experts, thus obtaining a score for each expert. To convert these scores into a probability distribution, a softmax function is applied to obtain the weight for each expert. All experts are ranked according to their weights, and the top-ranked experts are selected. k The experts. This k The weight of each expert is multiplied by its corresponding embedding, and this... k The results of multiplication by the experts are summed to obtain a complex interaction feature representation of the entity pair.

[0011] Step 4: Obtaining Prediction Results Based on Biomedical Hypotheses from Fully Connected Layers For different time steps of the dataset time series graph, the interaction feature representation obtained by the hybrid expert model is input into a fully connected layer and mapped to a one-dimensional embedding. The sigmoid function is then used to convert this embedding into a score to obtain the prediction result for each time step. These prediction results will be used to calculate the loss with the true label during training.

[0012] The beneficial effects of this invention are: This invention proposes a biomedical hypothesis generation representation learning method based on a hybrid expert model. It deeply mines the rich and diverse interaction patterns between entity pairs, effectively learning and fusing multiple interaction features of entity pairs using a hybrid expert model, significantly improving the expressive power and modeling accuracy of biomedical entity pair features. Experimental results show that this invention exhibits excellent performance on multiple biomedical hypothesis generation datasets, effectively improving the accuracy and reliability of biomedical hypothesis generation. Furthermore, this invention can be directly applied to key downstream research such as potential drug-target relationship discovery, pathogenic gene relocation, and molecular mechanism mining, providing efficient and reliable technical support for knowledge discovery and innovative applications in the biomedical field. Attached Figure Description

[0013] Figure 1 This is a basic flowchart of the present invention; Figure 2 This is a schematic diagram of a graph sampling aggregation network; Figure 3 This is a schematic diagram of a hybrid expert model; Figure 4 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0014] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0015] The present invention discloses a biomedical hypothesis generation representation learning method based on a hybrid expert model. First, using GraphSAGE as the backbone, multi-order neighborhoods of each biomedical entity are sampled and aggregated to generate an initial embedding that combines structural context and semantic consistency. Then, various fine-grained interaction methods are designed for biomedical entity pairs to deeply mine the rich interaction features between the two entities in each pair. Finally, a hybrid expert model is introduced, using the aforementioned interaction methods as different expert sub-networks, and gating is used to achieve the fusion of multiple features, thereby significantly improving the model's ability to characterize and generalize complex relationships between entity pairs. The invention includes the following steps: Step 1: Learn initial representations of biomedical entity pairs based on graph sampling aggregation networks. First, proceed as follows Figure 1 The dataset shown is read from the biomedical hypothesis-generated dataset to obtain the time steps. t Entity Relationship Diagram ,in Indicates time step t Biomedical entities on the surface Indicates time step t Co-occurrence of two entities in the same paper Indicates time step t Embedding of entities.

[0016] Next, proceed Figure 1 The graph sampling aggregation network GraphSAGE is used to capture spatial dependencies between nodes. The specific process is as follows: Figure 2 As shown, for time step t For each node, GraphSAGE is used to aggregate data from the nodes. Information about neighbors is used to generate their embeddings: in, Represents the first in GraphSAGE layer, Represents a node In GraphSAGE Layer embedding, Represents a node In GraphSAGE Layer embedding, It is a node The sampled neighbor set, Represents a node Sampling neighbors In GraphSAGE Layer embedding, Represents the ReLU activation function. This represents a function that aggregates the embeddings of neighboring nodes. In the initial layer... In the middle, node The embedding is the time step in the dataset. t upper node The initial embedding, i.e. , Represents a given node in the dataset The initial embedding. In After the aggregation of layers, nodes The embedding is represented as .

[0017] Step 2: Feature Capture of Biomedical Entity Pairs Based on Multiple Fine-Grained Interaction Modes The following will proceed Figure 1 The hybrid expert module is shown. To obtain multiple features and input them into the hybrid expert network, this invention utilizes multiple fine-grained interaction methods at time steps. t Capture features of biomedical entity pairs. For entity pairs Two entities and This invention not only focuses on the embedding of entity pairs formed by splicing them together, but also explores the diverse interaction methods between them.

[0018] This invention will time step t Entity pairs embedded on as well as Interactive computing methods can be summarized into the following six types: The first type is to directly embed and concatenate two entities to obtain entity pair embeddings. The second type is physical. Embedded, in The third type is entity embedding; Embedded, in The fourth method represents entity embedding; it adds two entity embeddings together. The fifth method involves capturing the joint semantics of two entities to enhance entity pair features; the second method involves subtracting the embeddings of the two entities. The process involves capturing the semantic differences between two entities and extracting entity pair difference features; finally, the two entity embeddings are multiplied element-wise. This allows us to obtain the fusion embedding and capture the correlation strength between two entities.

[0019] Step 3: Learning complex interaction feature representations of biomedical entity pairs based on a hybrid expert model like Figure 3 As shown, after embedding the six interaction features obtained in step two, a linear layer is added to each of them, forming multiple expert sub-networks of the hybrid expert model. At time step... t In this process, the formula for transforming each interaction feature into an expert subnetwork is as follows: in, Indicates time step t The first i A network of experts, Indicates time step t The first time i Such interactive feature embedding, and Indicates the first i Learnable parameters of an expert network.

[0020] For the first expert network, the entity pairs obtained by splicing are embedded... By performing expert-level operations, the first expert network was obtained. The same applies to the other expert networks.

[0021] To fully preserve the original information of the two entities, this invention uses entity pair splicing embedding. We obtain weights to measure the contribution of each expert in entity pair interactions. Entity pairs are embedded and concatenated, then input into a gating network, mapped to a vector space of the same dimension as the total number of experts. The softmax function is then applied to obtain the time steps. t Weight of all experts : in, and This represents the learnable parameters of the gated network.

[0022] To select the expert subnetwork that best matches the task, reduce the interference of redundant components on the results, and lower the computational cost required by the model, this invention filters all expert subnetworks, retaining only those with the largest weights. k A subnetwork of experts. Specifically, the subnetwork consists of the expert with the largest weight. k The index corresponding to each expert subnetwork is placed in the time step. t On the selection expert index set For subsequent calculations: in, express The first in Each weight, express Sort by expert weight from largest to smallest. k The number of experts selected.

[0023] The selected expert subnetwork results are multiplied by their corresponding weights to obtain their respective weighted results. Then, the weighted outputs from the selected experts are summed to obtain the weighted sum of the expert outputs. Interaction feature representation as entity pairs: in, Indicates time step t The selection expert index set contains the most weighted indexes. k An index corresponding to each expert Indicates time step t The first A network of experts For the first The weight corresponding to each expert.

[0024] This invention shares hybrid expert model parameters across all time steps. At each time step... t The entity pairs obtained from the graph sampling aggregation network model are embedded into the input hybrid expert model layer. The gating network dynamically selects experts based on the entity pair features at the current time step, achieving parameter reuse while maintaining step-by-step calculation, preventing the introduction of future information and preventing information leakage.

[0025] Thus, the feature representation of biomedical entity pairs is obtained through complex and diverse embedding interaction methods.

[0026] Step 4: Obtaining Prediction Results Based on Biomedical Hypotheses from Fully Connected Layers Based on the fully connected layer, the prediction result is obtained from the entity pair embedding representation obtained in step three. During training, at each time step, the learned biomedical entity pair representation can be transformed into a probability of co-occurrence relations. Specifically, this involves converting the entity pair interaction feature representation obtained from the hybrid expert model, i.e., the weighted sum of expert outputs, into a single representation. Input a fully connected layer, map it to a one-dimensional embedding, and use an activation function to convert this embedding into entity pair co-occurrence probabilities. : in, This indicates the weighted sum of expert outputs. This represents the activation function sigmoid. These are learnable parameters.

[0027] After this step, we can obtain each time step. t The prediction results. Before the last time step, the model's previous... The prediction results at each time step will be compared with the true labels during training to calculate the cross-entropy loss. : in, Indicates the training set at time step t The true label is 1 if it co-occurs, and 0 if it does not co-occur. For the present invention at time step t Predicted co-occurrence probability of biomedical entity pairs Indicates the time step length of the training set. Indicates the number of entity pairs.

[0028] The testing process is the same as the training process, using the trained model to obtain entity pair representations, for the ... This invention predicts co-occurrence relationships among biomedical entities at each time step. The ultimate goal of this invention is to predict future co-occurrence relationships based on known ones.

[0029] The method of this invention was used to conduct biomedical hypothesis generation research experiments on three co-occurrence datasets in the biomedical field. The evaluation index of the experimental results was Micro- F 1 point, Macro- FThe experiment included a score and an AUC (Area Under the CurveScore). The experiments were conducted on three biomedical hypothesis generation datasets in the fields of immunology, virology, and neurology, constructed by Zhou et al. (Zhou H, Jiang H, Yao W, Du X. Learning temporaldifference embeddings for biomedical hypothesis generation [J]. Bioinformatics, 2022, 38: 5253-5261.).

[0030] The specific experimental procedure of this invention is as follows: Figure 4 As shown in Table 1, the hyperparameter settings for the three datasets are as follows: First, the dataset is loaded, including entity and time series graph data. Then, model training begins, and after each training iteration, the results are validated on the validation set, retaining the model parameters with the best validation results. Finally, the optimal model parameters obtained through validation are loaded, and the final model performance metrics are acquired.

[0031] Table 1

[0032] The results on the immunology dataset are shown in Table 2 below.

[0033] Table 2

[0034] The results on the virology dataset are shown in Table 3 below.

[0035] Table 3

[0036] The results on the neurology dataset are shown in Table 4 below.

[0037] Table 4

[0038] This invention is compared with static graph models GraphSAGE and GAT, and dynamic graph models T-pair and TDE on each dataset. The invention performs best on all three datasets, demonstrating its good generalization ability and robustness. Specifically, the invention shows significant advantages on immunology and virology datasets with relatively small dataset sizes; however, on neurology datasets, due to the large data volume and task performance approaching its upper limit, the differences between different models decrease, so the performance advantage of hybrid expert models is not significant. Nevertheless, the invention still outperforms other methods.

[0039] The experimental results fully demonstrate that the biomedical hypothesis generation representation learning based on the hybrid expert model proposed in this invention can deeply explore the rich interaction modes between entity pairs, and learn multiple interaction features of entity pairs through the hybrid expert model, thereby comprehensively improving the feature expression ability of the model.

Claims

1. A method for biomedical hypothesis generation and representation learning based on a hybrid expert model, characterized in that, The steps are as follows: Step 1: Learn initial representations of biomedical entity pairs based on graph sampling aggregation networks. The GraphSAGE graph sampling aggregation network model is used to perform initial embedding learning on the biomedical entities in the dataset. In different time steps of the dataset time series graph, the initial representation of each entity in different time steps is obtained by aggregating the neighbor information of each entity in the current time step; Step 2: Feature Capture of Biomedical Entity Pairs Based on Multiple Fine-Grained Interaction Modes For each pair of biomedical entities, obtain six ways in which they interact: concatenating two entity embeddings, embedding entity 1, embedding entity 2, adding two entity embeddings, subtracting two entity embeddings, and multiplying two entity embeddings. Step 3: Learning complex interaction feature representations of biomedical entity pairs based on a hybrid expert model At different time steps of the dataset's time series graph, various interaction features are calculated for the initial embeddings of the two entities in the biomedical entity pair. A linear layer is added after each feature, serving as multiple expert sub-networks in the hybrid expert model. The concatenated embeddings of the two entities are input into a gating network and mapped to a vector space of the same dimension as the total number of experts, thus obtaining the score for each expert. To convert the scores into a probability distribution, a softmax function is applied to obtain the weight of each expert. All experts are ranked according to their weights, and the top-ranked experts are selected. k The experts; will this k The weight of each expert is multiplied by its corresponding embedding, and this... k The results of multiplication by the experts are summed to obtain the complex interaction feature representation of the entity pair; Step 4: Obtaining Prediction Results Based on Biomedical Hypotheses from Fully Connected Layers For different time steps of the dataset time series graph, the interaction feature representation obtained by the hybrid expert model is input into a fully connected layer and mapped to a one-dimensional embedding. The sigmoid function is then used to convert this embedding into a score to obtain the prediction result for each time step. These prediction results will be used to calculate the loss with the true label during training.

2. The biomedical hypothesis generation representation learning method based on a hybrid expert model according to claim 1, characterized in that, Step one is as follows: Read biomedical hypotheses to generate datasets and obtain time steps. t Entity Relationship Diagram ,in Indicates time step t Biomedical entities on the surface Indicates time step t Co-occurrence of two entities in the same paper Indicates time step t Embedding of entities; using the graph sampling aggregation network model GraphSAGE to capture spatial dependencies between nodes; for time steps t For each node, GraphSAGE is used to aggregate data from the nodes. Information about itself and its neighbors is used to generate its embedding: < img src='' class="img-anchor" img-id="QLYQS_6" / > in, Represents the first in GraphSAGE layer, Represents a node In GraphSAGE Layer embedding, Represents a node In GraphSAGE Layer embedding, It is a node The sampled neighbor set, Represents a node Sampling neighbors In GraphSAGE Layer embedding, Represents the ReLU activation function. This represents a function that aggregates the embeddings of neighbor nodes; in the initial layer In the middle, node The embedding is the time step in the dataset. t upper node The initial embedding, i.e. , Represents a given node in the dataset The initial embedding; in After the aggregation of layers, nodes The embedding is represented as .

3. The biomedical hypothesis generation representation learning method based on a hybrid expert model according to claim 1, characterized in that, Step two is as follows: For entity pairs Two entities and , take time steps t Entity pairs embedded on as well as Interactive computing methods can be summarized into the following six types: The first type is to directly embed and concatenate two entities to obtain entity pair embeddings. The second type is physical entities. Embedded, in The third type is entity embedding; Embedded, in Represents entity embedding; The fourth method is to embed and add two entities. This captures the joint semantics of two entities, enhancing entity pair features; The fifth method is to embed two entities into subtraction. The process involves capturing the semantic differences between two entities and extracting entity pair difference features; finally, the two entity embeddings are multiplied element-wise. This allows us to obtain the fusion embedding and capture the correlation strength between two entities.

4. The biomedical hypothesis generation representation learning method based on a hybrid expert model according to claim 1, characterized in that, Step three is as follows: After embedding the six interaction features obtained in step two, each is layered with a linear layer to form multiple expert subnetworks of the hybrid expert model; at time step... t In this process, the formula for transforming each interaction feature into an expert subnetwork is as follows: in, Indicates time step t The first i A network of experts, Indicates time step t The first time i Such interactive feature embedding, and Indicates the first i The learnable parameters of an expert network; For the first expert network, the entity pairs obtained by splicing are embedded... By performing expert-level operations, the first expert network was obtained. The same applies to the other expert networks; Embedding using entity pairs We obtain weights to measure the contribution of each expert in entity pair interactions; we concatenate the entity pairs and input them into the gating network, mapping them to a vector space of the same dimension as the total number of experts; then we apply the softmax function to obtain the time steps. t Weight of all experts : in, and Represents the learnable parameters of the gated network; All expert subnetworks are filtered, and only those with the highest weights are retained. k A subnetwork of experts; specifically, the subnetwork consists of the expert with the largest weight. k The index corresponding to each expert subnetwork is placed in the time step. t On the selection expert index set middle: in, express The first in Each weight, express Sort by expert weight from largest to smallest. k The number of experts selected; The selected expert subnetwork results are multiplied by their corresponding weights to obtain their respective weighted results; then, the weighted outputs from the selected experts are summed to obtain the weighted sum of the expert outputs. Interaction feature representation as entity pairs: in, Indicates time step t The selection expert index set contains the one with the highest weight. k An index corresponding to each expert Indicates time step t The first A network of experts For the first The weight corresponding to each expert.

5. The biomedical hypothesis generation representation learning method based on a hybrid expert model according to claim 1, characterized in that, Step four is as follows: Based on the fully connected layer, the prediction result is obtained from the entity pair embedding representation obtained in step three. During training, at each time step, the learned biomedical entity pair representation is transformed into a probability of co-occurrence relations. Specifically, the entity pair interaction feature representation obtained from the hybrid expert model, i.e., the weighted sum of expert outputs, is used. Take a fully connected layer as input, map it to a one-dimensional embedding, and then use an activation function to convert this embedding into a fraction. Get each time step t Prediction results: in, This indicates the weighted sum of expert outputs. This represents the activation function sigmoid. These are learnable parameters; Before the last time step, the model's front The prediction results at each time step will be compared with the true labels during training to calculate the cross-entropy loss. : in, Indicates the training set at time step t The true label is 1 if it co-occurs, and 0 if it does not co-occur. In time step t Predicted co-occurrence probability of biomedical entity pairs Indicates the time step length of the training set. Indicates the number of entity pairs; The testing process is the same as the training process, using the trained model to obtain entity pair representations, for the ... Predict co-occurrence relationships of biomedical entities at each time step.

Images