An open-domain scientific knowledge discovery method and device based on a pre-trained language model
By constructing an open-domain scientific knowledge discovery method based on a pre-trained language model, and optimizing the cue representation using external scientific knowledge and continuous spatial semantic information, the method solves the problems of insufficient efficiency and accuracy in scientific knowledge discovery in existing technologies, and achieves more efficient scientific knowledge discovery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-11-08
- Publication Date
- 2026-05-29
AI Technical Summary
Existing pre-trained language models struggle to effectively capture scientific terminology information in scientific knowledge discovery, resulting in poor efficiency and accuracy in knowledge mining.
We construct an open-domain scientific knowledge discovery method based on a pre-trained language model. By extracting triples from a scientific knowledge graph and constructing first and second prompts using external scientific knowledge, we optimize their embedding representations and train the pre-trained language model using a masking task to improve the efficiency and accuracy of knowledge discovery.
By incorporating external scientific knowledge and continuous spatial semantic information, the prompt representation of the pre-trained language model is optimized, which significantly improves the efficiency and accuracy of knowledge discovery and reduces the reliance on manual prompts.
Smart Images
Figure CN115658921B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing and machine learning, and specifically relates to a method and apparatus for discovering open-domain scientific knowledge based on a pre-trained language model. Background Technology
[0002] The development of pre-trained language models has elevated research in the field of natural language processing to a new level. Without manual labeling, they can learn general language representations from massive corpora and significantly improve downstream tasks. Some studies have shown that pre-trained language models possess a certain ability to store information and answer questions; their parameters implicitly contain different kinds of knowledge. This knowledge acquisition is crucial for language models in various downstream tasks. However, like most neural systems, the knowledge in pre-trained language models is encoded in a diffuse manner, making it often difficult to interpret and update.
[0003] Given the development of pre-trained language models, they are widely used to supplement and construct scientific knowledge graphs in the scientific field. Scientific knowledge graphs contain various scientific knowledge recorded by triples (head entity, relation, tail entity). The head and tail entities can be entities from various scientific fields such as biomedicine and chemistry, including diseases, drugs, genes, and molecules. Relationships include inclusion, action, and type.
[0004] Knowledge learned during pre-training can be potentially acquired through fine-tuning or hints. Hints are an effective way to directly acquire this knowledge without any additions. Hints can be divided into manually created hints and automatically learned hints. While manually created hints are intuitive and can indeed perform triplet completion tasks to some extent, this method requires time and experience, especially for some complex completion tasks, where even experienced hint designers may not be able to manually discover the best hints. Automatically learned hints automate the hint template design process but fail to fully capture scientific terminology information.
[0005] Patent document CN114706943A discloses an intent recognition method that uses a pre-trained language model to recognize intent from input text with added prompts. However, the added prompts do not capture scientific terminology, resulting in poor accuracy of the embedded knowledge mined during the learning process.
[0006] Patent document CN114661913A discloses an entity relation extraction method based on a pre-trained language model. It solves the problem that manual intervention is required for the annotation of prompt templates in knowledge mining schemes, which leads to poor knowledge mining efficiency. However, this method still does not capture scientific terminology information, which also leads to poor accuracy of embedded knowledge mined during the learning process. Summary of the Invention
[0007] In view of the above, the purpose of this invention is to provide an open-domain scientific knowledge discovery method and apparatus based on a pre-trained language model, which fully considers external scientific knowledge and learnable knowledge representations, can better acquire knowledge in the pre-trained language model, achieve a better ability to detect knowledge in the pre-trained language model, and improve the efficiency and accuracy of knowledge discovery by the pre-trained language model.
[0008] To achieve the above-mentioned objectives, an embodiment provides an open-domain scientific knowledge discovery method based on a pre-trained language model, comprising the following steps:
[0009] Extract triples (head entity, relation, tail entity) from the scientific knowledge graph, and construct a first prompt and a second prompt for each relation using external scientific knowledge;
[0010] Construct an input template for a pre-trained language model, the input template including a head entity, a first prompt, a second prompt, and a tail entity mask;
[0011] All triples containing the target relation are used as sample data. The head entity of each triple, the first prompt and the second prompt corresponding to the target relation are filled into the input template, and the tail entity is masked to form the input sample data.
[0012] For each target relation, a single pre-trained language model is constructed. The pre-trained language model is then trained for a masking task using the input sample data corresponding to the target relation, thereby optimizing the embedding representation of the first and second prompts.
[0013] By utilizing optimized embedding representations of the first and second prompts and a pre-trained language model, we can predict missing entities in triples containing target relations, thereby enabling the discovery of open-domain scientific knowledge.
[0014] Preferably, the first and second prompts constructed using external scientific knowledge are related to the relations in the triples. The first prompt is used as discrete tokens in the input template, and the second prompt is used as initialization tokens for a continuous spatial vector in the input template.
[0015] Preferably, the first prompt corresponding to the target relationship is filled in the input template in a discrete form, serving as discrete tokens of the target relationship.
[0016] Preferably, the second prompt corresponding to the target relation is filled in the input template in the form of a continuous vector, serving as the initialization tokens of the continuous vector of the target relation.
[0017] Preferably, the tokens in the second prompt corresponding to the target relation are embedded using pre-trained embeddings, and this embedding is used as the initialization of the continuous vector of the target relation.
[0018] Preferably, the pre-trained language model is trained for the masking task using the input sample data corresponding to the target relationship, keeping the parameters of the pre-trained language model fixed, and the following loss function is used and the gradient correction method is used to minimize the probability of the negative logarithm of the input sample data in order to update the embedding representation of the first prompt and the second prompt.
[0019]
[0020] in, Denotes the loss function, D r Let t represent the set of head and tail entity pairs (h,t) of the target relation r. r (h) represents the input sample data formed by filling the input template with the head and tail entity pairs (h,t) containing the target relation r, and P(([MASK]=t)|t r (h) represents t r (h) The mask [MASK] input to the output of the pre-trained language model is equal to the probability value of the tail entity t.
[0021] Preferably, the prediction of missing entities in triples containing target relations is performed using optimized embeddings of the first and second prompts and a pre-trained language model, including:
[0022] The known entities in the triples are used as the head entities, and the embedded representations of the head entities, the optimized first prompt, and the second prompt are filled into the input template to form the input sample data.
[0023] The input sample data is fed into a pre-trained language model, and the predicted probability of the missing entity in the triple is calculated and output.
[0024] To achieve the above-mentioned objectives, an embodiment provides an open-domain scientific knowledge discovery device based on a pre-trained language model, comprising:
[0025] The external knowledge acquisition module is used to extract triples (head entity, relation, and tail entity) from the scientific knowledge graph and construct a first prompt and a second prompt for each relation using external scientific knowledge.
[0026] An input template construction module is used to construct an input template for a pre-trained language model. The input template includes a head entity, a first prompt, a second prompt, and a tail entity mask.
[0027] The input sample data construction module is used to take all triples containing the target relation as sample data, fill the input template with the head entity of each triple, the first prompt and the second prompt corresponding to the target relation, and process the tail entity mask to form the input sample data.
[0028] The training module is used to build a single pre-trained language model for each target relation, and to train the pre-trained language model for the masking task using the input sample data corresponding to the target relation, thereby optimizing the embedding representation of the first and second prompts.
[0029] The application module is used to predict missing entities in triples containing target relations by utilizing optimized embedding representations of the first and second prompts and a pre-trained language model, thereby enabling the discovery of open-domain scientific knowledge.
[0030] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0031] When constructing input sample data, external scientific knowledge is introduced as the first and second prompts for the target relationship, which makes the input sample data contain more scientifically relevant semantic information. When training the pre-trained language model, the semantic information of the continuous space is incorporated, which does not rely entirely on the experience of human prompts. This can more effectively capture the knowledge information in the pre-trained language model, thereby improving the efficiency and accuracy of the pre-trained language model in discovering knowledge. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 This is a flowchart of the open-domain scientific knowledge discovery method based on a pre-trained language model provided in the embodiment;
[0034] Figure 2 This is a flowchart of an open-domain scientific knowledge discovery method based on a pre-trained language model, provided in the embodiment. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0036] To enable pre-trained language models to learn knowledge more effectively and improve their ability to detect knowledge within the language model, thereby enhancing the efficiency and accuracy of knowledge discovery, this embodiment provides a method and apparatus for open-domain scientific knowledge discovery based on a pre-trained language model.
[0037] like Figure 1 As shown in the embodiment, the open-domain scientific knowledge discovery method based on a pre-trained language model includes the following steps:
[0038] Step 1: Extract triples from the scientific knowledge graph, classify the triples according to their relationships, and construct a first and second prompt for each relationship using external scientific knowledge.
[0039] In this embodiment, the triplet extracted from the scientific knowledge graph contains (head entity h, relation r, tail entity t) which includes various scientific knowledge. The head entity includes entities from various scientific fields such as biomedicine and chemistry, such as diseases, drugs, genes, and molecules. The relation includes terms such as include, act on, and type. The tail entity includes entities from various scientific fields such as biomedicine and chemistry, such as diseases, drugs, genes, and molecules.
[0040] In this embodiment, two prompts are constructed for each relationship using external scientific knowledge, wherein the first prompt is t r1 Second prompt t r2 The relation r in the triple can be either equal or unequal. The first prompt serves as the discrete tokens of the relation in the input template, while the second prompt serves as the initialization tokens for the continuous space vector of the relation.
[0041] Step 2: Construct an input template for the pre-trained language model. The input template includes a head entity, a first prompt, a second prompt, and a tail entity mask.
[0042] Objective facts are not independent of each other. Data-driven generated prompts can draw on the distribution of knowledge in the training set to extract knowledge from pre-trained language models, and can even recover objective facts from randomly initialized pre-trained language models. Previous methods (such as AutoPrompt) first search for K superior candidate prompts in a discrete vocabulary, then select and verify them, thus limiting the search space to a discrete space. If replaced with continuous vectors, these might not be true tokens but rather continuous vector representations, lacking the scientific terminology information of discrete tokens. Therefore, this embodiment uses scientific terms to construct the first prompt as discrete tokens, and adds continuous space vectors based on the second prompt as initial tokens. Then, it masks the tokens of triple tail entities to predict these masked tokens. Therefore, the constructed input template is:
[0043] t r =[X][Term]1…[Term] m [P]1…[P] n [MASK]
[0044] Where [X] is the head entity of the triple, [Term]1…[Term] m The discrete tokens t represent scientific terms related to relation r. r1 m is t r1 The number of tokens in the middle, It is a continuous vector in the vector space, where d represents the dimension of the embedded vector, and n is a continuous vector in the vector space. r2 The number of tokens in the middle, [MASK] is the masked tail entity.
[0045] Step 3: Take all triples containing the target relation as sample data, fill the input template with the head entity of each triple, the first prompt and the second prompt corresponding to the target relation, and mask the tail entity to form the input sample data.
[0046] In this embodiment, the relation of interest is taken as the target relation, and all triples containing the target relation are used as sample data for the pre-trained language model corresponding to the target relation. This sample data is converted into input sample data through an input template. Specifically, the first and second prompts corresponding to the target relation are filled in the input template. In the input template, the first prompt corresponding to the target relation exists in a discrete form, serving as discrete tokens of the target relation. The second prompt corresponding to the target relation exists in the form of a continuous vector, serving as initialization tokens for the continuous vector of the target relation.
[0047] In this challenging nonconvex optimization problem, good initialization of continuous vectors is crucial. Therefore, the embodiment considers a more complex form, namely, using a manually constructed second cue to determine the number n and position of continuous vectors [P] for each target relation, and initializing [P] using pre-trained embeddings of tokens in the second cue.
[0048] Simultaneously, the head entity is filled into the input template, and the tail entity is masked to form the input sample data.
[0049] Step 4: Build a single pre-trained language model for each target relation, and train the pre-trained language model for the masking task using the input sample data corresponding to the target relation to optimize the embedding representation of the first and second prompts.
[0050] In this embodiment, each target relation corresponds to a pre-trained language model. The pre-trained language model is trained for the masking task using the input sample data corresponding to the target relation. The parameters of the pre-trained language model are kept constant. The following loss function is used and the gradient correction method is used to minimize the probability of negative logarithm of input sample data in order to update the embedding representation of the first prompt and the second prompt.
[0051]
[0052] in, Denotes the loss function, D r Let t represent the set of head and tail entity pairs (h,t) of the target relation r. r (h) represents the input sample data formed by filling the input template with the head and tail entity pairs (h,t) containing the target relation r, and P(([MASK]=t)|t r (h) represents t r (h) The mask [MASK] input to the output of the pre-trained language model is equal to the probability value of the tail entity t.
[0053] By using the knowledge masking task of a pre-trained language model, we can explore the knowledge information of the pre-trained language model and optimize the embedding representation of the first and second prompts, which is more conducive to the prediction of missing entities in triples.
[0054] Step 5: Utilize the optimized embedding representations of the first and second prompts, along with a pre-trained language model, to predict missing entities in triples containing the target relation, thereby enabling the discovery of open-domain scientific knowledge.
[0055] The prediction of missing entities in triples containing target relations is performed using optimized embeddings of the first and second cues and a pre-trained language model, including:
[0056] The known entities in the triples are used as the head entity, and the head entity, the optimized first prompt, and the second prompt are embedded in the input template to form the input sample data, which is a cloze test statement.
[0057] Input sample data is fed into a pre-trained language model, which calculates and outputs the predicted probability of missing entities in the triples. Based on the predicted probability, missing entities are selected to complete the missing triples and form complete triples, thus realizing the discovery of open-domain scientific knowledge.
[0058] Based on the same inventive concept, such as Figure 2 As shown in the embodiment, an open-domain scientific knowledge discovery device based on a pre-trained language model is also provided, including:
[0059] The external knowledge acquisition module is used to extract triples from the scientific knowledge graph, classify the triples according to the relationship, and construct a first prompt and a second prompt for each relationship using external scientific knowledge.
[0060] An input template construction module is used to construct an input template for a pre-trained language model. The input template includes a head entity, a first prompt, a second prompt, and a tail entity mask.
[0061] The input sample data construction module is used to take all triples containing the target relation as sample data, fill the input template with the head entity of each triple, the first prompt and the second prompt corresponding to the target relation, and process the tail entity mask to form the input sample data.
[0062] The training module is used to build a single pre-trained language model for each target relation, and to train the pre-trained language model for the masking task using the input sample data corresponding to the target relation, thereby optimizing the embedding representation of the first and second prompts.
[0063] The application module is used to predict missing entities in triples containing target relations by utilizing optimized embedding representations of the first and second prompts and a pre-trained language model, thereby enabling the discovery of open-domain scientific knowledge.
[0064] It should be noted that the open-domain scientific knowledge discovery device based on pre-trained language models provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing open-domain scientific knowledge discovery based on pre-trained language models. The functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the open-domain scientific knowledge discovery device based on pre-trained language models provided in the above embodiments and the open-domain scientific knowledge discovery method embodiments based on pre-trained language models belong to the same concept. For details of its implementation process, please refer to the open-domain scientific knowledge discovery method embodiments based on pre-trained language models, which will not be repeated here.
[0065] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for discovering open-domain scientific knowledge based on a pre-trained language model, characterized in that, Includes the following steps: Extract triples (head entity, relation, tail entity) from the scientific knowledge graph. For each relation, construct a first prompt and a second prompt using external scientific knowledge. The first prompt and the second prompt are related to the relation in the triple. The first prompt is used as discrete tokens in the input template, and the second prompt is used as initialization tokens for a continuous space vector in the input template. Construct an input template for a pre-trained language model, the input template including a head entity, a first prompt, a second prompt, and a tail entity mask; All triples containing the target relation are used as sample data. The head entity of each triple, the first prompt and the second prompt corresponding to the target relation are filled into the input template. The tokens in the second prompt corresponding to the target relation are embedded using pre-trained embedding. This embedding is used as the initialization of the continuous vector of the target relation. The tail entity is masked to form the input sample data, so that the input sample data contains more scientifically relevant semantic information. A single pre-trained language model is constructed for each target relation. The pre-trained language model is trained using the input sample data corresponding to the target relation for a masking task. Semantic information from the continuous space is incorporated to explore the knowledge information of the pre-trained language model and optimize the embedding representation of the first and second prompts. This embedding representation is more conducive to the prediction of missing entities in triples. By utilizing optimized embeddings of the first and second prompts and a pre-trained language model, we can predict missing entities in triples containing target relations, thereby enabling the discovery of open-domain scientific knowledge.
2. The open-domain scientific knowledge discovery method based on a pre-trained language model according to claim 1, characterized in that, The first prompt corresponding to the target relation is filled in the input template in a discrete form, serving as discrete tokens of the target relation.
3. The open-domain scientific knowledge discovery method based on a pre-trained language model according to claim 1, characterized in that, The second prompt corresponding to the target relation is filled in the input template in the form of a continuous vector, serving as the initialization tokens for the continuous vector of the target relation.
4. The open-domain scientific knowledge discovery method based on a pre-trained language model according to claim 1, characterized in that, The pre-trained language model is trained for the masking task using the input sample data corresponding to the target relationship. The parameters of the pre-trained language model are kept constant. The following loss function is used and the gradient correction method is used to minimize the negative log probability of the input sample data in order to update the embedding representation of the first and second prompts. in, Represents the loss function. Representing target relationship Head and tail entity pairs The set, Indicates the inclusion of a target relationship Head and tail entity pairs The input sample data is filled into the input template. express Mask input to the output of the pre-trained language model equal to tail entity The probability value.
5. The open-domain scientific knowledge discovery method based on a pre-trained language model according to claim 1, characterized in that, The prediction of missing entities in triples containing target relations is performed using optimized embeddings of the first and second cues and a pre-trained language model, including: The known entities in the triples are used as the head entities, and the embedded representations of the head entities, the optimized first prompt, and the second prompt are filled into the input template to form the input sample data. The input sample data is fed into a pre-trained language model, and the predicted probability of the missing entity in the triple is calculated and output.
6. An open-domain scientific knowledge discovery device based on a pre-trained language model, characterized in that, include: The external knowledge acquisition module is used to extract triples (head entity, relation, tail entity) from the scientific knowledge graph. For each relation, a first prompt and a second prompt are constructed using external scientific knowledge. The first prompt and the second prompt are related to the relation in the triple. The first prompt is used as discrete tokens in the input template, and the second prompt is used as initialization tokens for a continuous space vector in the input template. An input template construction module is used to construct an input template for a pre-trained language model. The input template includes a head entity, a first prompt, a second prompt, and a tail entity mask. The input sample data construction module is used to take all triples containing the target relation as sample data, fill the input template with the head entity of each triple, the first prompt and the second prompt corresponding to the target relation, and use the pre-trained embedding to embed the tokens in the second prompt corresponding to the target relation. This embedding is used as the initialization of the continuous vector of the target relation, and the tail entity is masked to form the input sample data, so that the input sample data contains more scientifically relevant semantic information. The training module is used to build a single pre-trained language model for each target relation. It uses the input sample data corresponding to the target relation to train the pre-trained language model for the masking task, incorporates semantic information from the continuous space, explores the knowledge information of the pre-trained language model, and optimizes the embedding representation of the first and second prompts. This embedding representation is more conducive to the prediction of missing entities in triples. The application module is used to predict missing entities in triples containing target relations by utilizing optimized embedding representations of the first and second prompts and a pre-trained language model, thereby enabling the discovery of open-domain scientific knowledge.