Knowledge injection method for pre-trained language model and corresponding interactive system

Abstract

Disclosed are a method for knowledge injection into a pre-trained language model and a corresponding interactive system. The method comprises: recalling triple information related to long-tail entities contained in training samples from a knowledge graph; constructing a knowledge representation embedding for replacing the long-tail entities based on the triple information; using training samples in which long-tail entities are replaced with the knowledge representation embedding for pre-training the pre-trained language model PLM; and during the pre-training process, adjusting the neural network of the PLM using a first loss function that predicts that the long-tail entities do not introduce losses. The present invention can complete knowledge injection in the pre-training stage without using an additional knowledge encoder by injecting pseudo-labeled embeddings of triple information recalled from the knowledge graph into the pre-trained model and requiring the model to predict the original long-tail entity to be replaced, thereby facilitating the processing of subsequent fine-tuning and reasoning stages.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and more particularly to a method for injecting knowledge into a pre-trained language model, and an interactive system equipped with a pre-trained model injected with knowledge acquired by the method. Background Art

[0002] As the basic model of natural language processing (NLP), pre-trained language models (PLM, including BERT, RoBERTa, XLNET, etc.) have achieved excellent results in various downstream natural language understanding (NLU) tasks and have strong versatility. However, mainstream pre-trained language models are based on public documents, learning general language knowledge from unstructured documents, and ignoring the learning of a large amount of knowledge information, especially the learning of structured knowledge graph (KG) information. This will result in the model being unable to truly understand a series of knowledge such as entities in reality and the relationships between them, and it is difficult to have real NLU capabilities. The lack of knowledge will produce some counterfactual outputs (for example, the GPT model will output assertions such as "the sun has two eyes"), and will also greatly weaken the model's small sample learning ability, domain knowledge transfer ability, general knowledge induction ability, etc.

[0003] Therefore, there is a need for PLM to learn knowledge graph information to improve downstream NLU capabilities. However, since PLM learns language knowledge from unstructured documents and knowledge graphs store information in structured triples, how to enable PLM to efficiently and accurately learn knowledge from information in the form of triples has become a problem that technicians in this field need to solve. Summary of the invention

[0004] A technical problem to be solved by the present disclosure is to provide a method for knowledge injection into a pre-trained language model. The method embeds pseudo-labels of triple information recalled from a knowledge graph into the pre-trained model, and can complete the knowledge injection in the pre-training stage without using an additional knowledge encoder, thereby avoiding the use of additional knowledge encoders in subsequent fine-tuning and reasoning stages, and facilitating the reasoning application of the knowledge-injected pre-trained model.

[0005] According to a first aspect of the present disclosure, a method for knowledge injection into a pre-trained language model is provided, comprising: recalling triple information related to long-tail entities contained in training samples from a knowledge graph; constructing a knowledge representation embedding for replacing the long-tail entities based on the triple information; using training samples in which long-tail entities are replaced with the knowledge representation embedding for pre-training the pre-trained language model PLM; and during the pre-training process, adjusting a neural network of the PLM using a first loss function that predicts that the long-tail entities do not introduce losses.

[0006] Optionally, the method further includes: determining whether each sentence in a corpus serving as a pre-training data set contains a long-tail entity, wherein the determination is performed based on a first factor, and the first factor is related to the frequency of occurrence of the entity in the corpus.

[0007] Optionally, the method also includes: determining whether each sentence in the corpus serving as a pre-training data set contains a long-tail entity based on at least one of the following factors: a second factor, the second factor being related to the semantic importance of the entity in the sentence; and a third factor, the third factor being related to the connectivity of the entity in the knowledge graph.

[0008] Optionally, the method also includes: recalling similar triple information and invalid triple information of the triple information from the knowledge graph; and performing comparative learning in a pre-training task for the triple information, the similar triple information and the invalid triple information.

[0009] Optionally, the method also includes: constructing a relationship tree based on the knowledge graph, and recalling similar triple information and invalid triple information of the triple information from the knowledge graph includes: recalling the similar triple information and multiple invalid triple information with different comparative learning difficulties based on the distance of the relationship on the relationship tree.

[0010] Optionally, constructing a knowledge representation embedding for replacing the long-tail entity based on the triple information includes: when the long-tail entity is the head entity of the triple information, the knowledge representation embedding includes the tail entity embedding minus the relational word embedding; when the long-tail entity is the tail entity of the triple information, the knowledge representation embedding includes the head entity embedding plus the relational word embedding.

[0011] Optionally, the knowledge representation embedding further includes: a concatenated explanation text of the long-tail entity, wherein the explanation text is selected from a corpus serving as a pre-training data set.

[0012] Optionally, the method also includes: masking non-entities in the training samples; inputting the masked training samples into the pre-trained language model PLM; obtaining a second loss value output by the PLM for the masked words, and adjusting the parameters of the neural network model in the PLM accordingly, wherein the second loss value function is defined as the cross entropy between the one-hot vector and the model predicted probability distribution.

[0013] Optionally, the method further includes: before pre-training, analyzing the corpus and knowledge graph serving as the pre-training data set to mark the long-tail entities contained in each sentence in the corpus and a specific triplet information related to each long-tail entity recalled from the knowledge graph.

[0014] According to a second aspect of the present disclosure, an interactive system based on knowledge injection is provided, comprising: a user input receiving unit, used to obtain a specific domain related query input by a user; a question matching unit, comprising a pre-trained model injected with specific domain knowledge obtained using a specific domain corpus and a specific domain knowledge graph according to the method described in the first aspect, wherein the model identifies relevant long-tail entities in the specific domain related query and generates feedback accordingly; and a feedback providing unit, used to provide the generated feedback to the user.

[0015] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first aspect above.

[0016] According to a fourth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0017] Therefore, based on the relational tree pseudo-embedding method of the shared encoder, the present invention proposes a method that combines knowledge encoding, knowledge injection and pre-training process, so that the model can achieve good results without adding additional parameters and without relying on the knowledge graph during the use of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0019] Figure 1 A schematic diagram showing the comparison of the DKPLM used in the present invention and other models in three operation stages.

[0020] Figure 2 A schematic flowchart of a method for injecting knowledge into a pre-trained language model according to an embodiment of the present invention is shown.

[0021] Figure 3 An example of a knowledge graph is shown.

[0022] Figure 4 A schematic diagram of long-tail entity detection according to an embodiment of the present invention is shown.

[0023] Figure 5 A schematic diagram of pre-training of DKPLM according to an embodiment of the present invention is shown.

[0024] Figure 6 An example of using the PLM trained by the present invention for actual interaction is shown.

[0025] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the knowledge injection method for the pre-trained language model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] Pre-trained language model (PLM) is a general term for a class of natural language processing models that are used to learn low-dimensional dense real-valued vector representations of text. Early pre-trained language models were designed to use shallow neural networks to learn word embedding representations, which are used for various downstream natural language processing tasks; the latest pre-trained language models learn context-based word embedding representations, and the learned models are used for fine-tuning in downstream tasks. Pre-trained language models have achieved excellent results in various downstream natural language understanding (NLU) tasks and have strong versatility.

[0028] Mainstream pre-trained language models are based on public documents and learn general language knowledge from unstructured documents, ignoring the learning of a large amount of knowledge information, especially the learning of structured knowledge graph (KG) information. Here, unstructured and structured are intended to indicate the presentation of language knowledge. In natural language processing, the presentation of language knowledge usually includes the following three forms: unstructured text, semi-structured tables, etc. and structured triples. Specifically, triple knowledge is stored in artificially constructed large-scale knowledge graph data, consisting of <head entity, relationship, tail entity>. The head and tail entities represent a specific thing that exists in the real world (for example: Hangzhou), and the relationship expresses a certain semantic association between entities (for example: birthplace).

[0029] In PLM, the two-stage strategy (i.e., pre-training and fine-tuning) inherits the knowledge learned during pre-training and applies it to downstream tasks. Although PLM stores a large amount of internal knowledge, it is difficult for it to understand external background knowledge, such as facts and common sense knowledge, because PLM learns general language knowledge from unstructured documents and lacks systematic learning of structured knowledge. The lack of knowledge will produce some counterfactual content (for example, the GPT model will output obviously wrong statements such as "the sun has two eyes"), and will also greatly weaken the model's small sample learning ability, domain knowledge transfer ability, and general knowledge induction ability, etc.

[0030] Therefore, the performance of PLM can be improved by injecting external knowledge triples, which is called knowledge-enhanced PLM (KEPLM). In the prior art, the methods of injecting knowledge mainly include:

[0031] Fusion methods based on knowledge representation embedding: There are two main problems to be solved in this type of method: (1) How to represent the triples in the existing knowledge graph in text form and obtain low-dimensional dense real-valued vectors of the head and tail entities and semantic relations respectively. (2) How to fuse and align the context representation space encoded by the pre-trained language model with the knowledge feature space. For problem (1), existing models (such as ERNIE-THU) use existing knowledge graph representation models (such as TransE) for embedding learning. For problem (2), existing models (such as ERNIE-THU, KnowBERT) align the two parts of the representation space by designing a separate neural network parameter module. However, their main disadvantages are: (1) This type of method requires the design of additional modules other than the pre-trained model to align the two feature spaces, which requires more machine resources. (2) In the process of real-world application, the three processes in the training paradigm require the injection of structured knowledge, but it is difficult to independently construct a large-scale general knowledge graph or domain knowledge graph on the user side. Therefore, this type of method is not user-friendly when used in practice.

[0032] Fusion method based on knowledge sentence conversion (Knowledge Converting): This type of method is an improvement on the above-mentioned knowledge representation fusion method, and basically does not change the original pre-trained language model framework. Usually, this type of method includes two modules: (1) Knowledge sentence conversion: Convert the representation of the triple structure into a natural sentence expression according to the template, for example, <China, capital, Beijing> is converted to "China's capital is Beijing", and this knowledge text is linked to the original training text, and directly encoded using the existing pre-trained model without the need for a traditional knowledge encoding model. (2) Knowledge text fusion process: Train together with the knowledge triple text linked to the original training sentence, and use the Visible Matrix in the input process to allow the model to fuse different input text information separately, thereby fusing the two. However, this type of method still needs to use the constructed knowledge graph information in the three training processes, which is still not user-friendly.

[0033] The knowledge-enhanced pre-trained model (KEPLM) described above can further improve the performance of downstream tasks by understanding the semantics of key entity information in the text based on the pre-trained model modeling. However, the biggest problem with the above methods is that when training and reasoning downstream tasks, the knowledge-enhanced pre-trained model still needs to construct a large-scale available knowledge graph in advance, and the additional network parameters will increase the burden of computing resources. Therefore, it is cumbersome in actual application and the effect is not stable enough.

[0034] Therefore, it is hoped that the knowledge-enhanced model can achieve good performance in downstream tasks without using the information of the knowledge graph in the fine-tuning and reasoning stages. To this end, the present invention proposes a method that combines knowledge encoding, knowledge injection and pre-training process based on the relationship tree pseudo-embedding method of the shared encoder, so that the model does not add additional parameters and does not rely on the knowledge graph during the use of downstream tasks. It can also achieve good results. Specifically, the present invention proposes a new KEPLM called DKPLM (DecomposableKnowledge-enhancedPLM), which decomposes the three-stage knowledge injection process of KEPLM. Figure 1 The figure shows a comparison diagram of the DKPLM used in the present invention and other models in three stages of operation. In real scenarios, the operation of a pre-trained language model usually includes three processes (stages): pre-training, fine-tuning, and inference, which is also true for the training paradigm of the knowledge-enhanced pre-training model shown in the figure.

[0035] As shown in the first row of the left half of the figure, ordinary PLM does not utilize external knowledge in all three stages and includes the most basic operations, namely, pre-training using a corpus; fine-tuning using a dataset, and performing inference on input samples in the inference stage.

[0036] As shown in the second row of the left half of the figure, the existing KEPLM needs to construct a knowledge source, such as the KG, dictionary (dic.) and text shown on the right side of the figure, as well as a knowledge encoder that can process the input triples (as shown in the figure, it can be implemented by various neural networks (NN) and graph neural networks (GNN)). Since the knowledge encoder is used in the pre-training stage, the knowledge source and the knowledge encoder that can process the knowledge source are still needed to realize the knowledge understanding ability in the subsequent fine-tuning and reasoning stages.

[0037] As shown in the third row on the left half of the figure, the DKPLM of the present invention can use the same data source as the existing KEPLM during pre-training, but does not require a knowledge encoder (e.g., neural network and graph neural network). In the subsequent fine-tuning and reasoning process, the model of the present invention no longer requires KG, so it is very flexible and efficient.

[0038] Since for the DKPLM of the present invention, knowledge injection is only applied during pre-training without using an additional knowledge encoder. Therefore, the model can be used in the same way as BERT and other ordinary PLMs during fine-tuning and inference, which facilitates the application of DKPLM in practical scenarios. Specifically, three new techniques for pre-training DKPLM are introduced in the present invention:

[0039] Pseudo-token representation injection: The embeddings of detected long-tail entities are replaced with the representations of the corresponding knowledge triples generated by the shared PLM encoder, called “pseudo-token representations”. Therefore, knowledge can be injected without introducing any additional parameters to the model.

[0040] Relational knowledge decoding: For a relation triple, the representation of one entity and the relational predicate (attribute) are used to decode each token of the other entity. This pre-training task acts as a supervisory signal for KEPLM, forcing the model to understand what knowledge is injected into KEPLM.

[0041] Knowledge-aware long-tail entity detection: The DKPLM model can detect only long-tail entities for knowledge injection, thus avoiding learning too much redundant and irrelevant information.

[0042] The specific implementation of the DKPLM of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] Figure 2The figure shows a schematic flowchart of a knowledge injection method for a pre-trained language model according to an embodiment of the present invention. Figure 2 The method shown is as Figure 1 shown, and it is only carried out in the pre-training stage of the pre-trained model, and no additional knowledge injection is required in the subsequent fine-tuning and inference stages.

[0044] In step S210, triple information related to the long-tail entities included in the training samples is recalled from the knowledge graph.

[0045] Here, it should be clear that existing pre-trained language models use a corpus as the training sample set. A corpus refers to a large-scale electronic text library that has been scientifically sampled and processed, and stores language materials that have actually appeared in the actual use of language. With the help of computer analysis tools, researchers can carry out relevant language theory and application research. The pre-trained language model uses each sentence included in the corpus as a training sample, and the corpus containing these sentences can be called a "training sample set".

[0046] A knowledge graph is a knowledge base, and the data in it is integrated through a graph structure data model or topology. A knowledge graph is usually used to store entities that are related to each other. The knowledge graph effectively processes, processes, and integrates the data of complex documents, transforms them into simple and clear <head entity, relationship, tail entity> triples, and finally aggregates a large amount of knowledge, so as to achieve rapid response and inference of knowledge. Figure 3 An example of a knowledge graph is shown. As shown in the figure, the knowledge graph includes a large number of <head entity, relationship, tail entity> triples for the structured representation of knowledge. The head and tail entities represent a specific thing that exists in the real world, and the relationship expresses a certain semantic association between the entities. For example, Figure 3 in a triple <China, capital, Beijing> framed by a dotted line box, China is the head entity, capital is the relationship word, and Beijing is the tail entity.

[0047] The "long-tail entities" in step S210 can be understood as entities that show a long-tail distribution in the training sample set. Since these entities lack sufficient information description in the training sample set, additional knowledge injection is required to increase the PLM's understanding of these long-tail entities. A triple that best represents the long-tail entity can be recalled from the knowledge graph.

[0048] Subsequently, in step S220, a knowledge representation embedding for replacing the long-tail entity can be constructed based on the recalled triple information. Specifically, the other two sets of information in the triple information can be used to replace the long-tail entity. For example, in the example of the triple <China, capital, Beijing>, the long-tail entity <Beijing> in the training sample (i.e., a sentence in the corpus) is replaced with <China's capital>, and the text <China's capital> is converted into an embedding representation of the text through the same set of pre-trained models. And in step S230, the training sample in which the long-tail entity is replaced with the knowledge representation embedding is used for pre-training of the pre-trained language model PLM. Therefore, the pseudo-embedding method can effectively solve the problem in the prior art of adding additional network parameters in the knowledge injection process to align the feature space.

[0049] Therefore, in step S240, during the pre-training process, the neural network of the PLM can be adjusted using the first loss function that predicts that the long-tail entity does not introduce loss. In order to improve the effect of knowledge injection, the setting of the loss function enables the model to reconstruct the corresponding specific text by decoding the injected knowledge triple information. In other words, when injecting the embedding of <China's capital> to replace the original long-tail entity <Beijing> in the training sample, the model needs to be able to correctly predict the original text <Beijing>.

[0050] Therefore, the present invention uses the knowledge representation embedding of the long-tail entity derived from the knowledge graph to replace the long-tail entity for prediction in PLM, which can realize knowledge injection without the need for an additional knowledge encoder, and by only injecting the long-tail entity, the model avoids learning redundant and unnecessary knowledge information.

[0051] Although the above example (and the following text) uses "Beijing" as the long-tail entity, <China, capital, Beijing> as the recalled triple information, and <China's capital> as the prototype text for the knowledge representation embedding to replace the long-tail entity for the sake of convenience of explanation, it should be understood that "Beijing" is usually not regarded as a long-tail entity in practical applications because it corresponds to a single English word and appears frequently.

[0052] Specifically, the determination of long-tail entities can be performed in the preparation stage before pre-training. Specifically, all sentences in the corpus can be processed in advance to determine whether the entity in the sentence is a long-tail entity. To this end, the knowledge injection method of the present invention may include: determining whether each sentence in the corpus serving as a pre-training data set contains a long-tail entity. In different embodiments, the determination of long-tail entities can be performed based on the first factor, or even the second factor, the third factor, or any combination thereof.

[0053] Here, the first factor may be related to the frequency of occurrence of an entity in a corpus. The second factor may be related to the semantic importance of an entity in a sentence. The third factor may be related to the degree of connection of an entity in the knowledge graph. For example, an entity with a relatively low frequency of occurrence in a corpus (e.g., below a certain threshold) may be regarded as a long-tail entity. An entity with a relatively low frequency of occurrence in a corpus and relatively important in the semantic expression of a certain sentence may be regarded as a long-tail entity. An entity that is relatively important in the knowledge graph may also be regarded as a long-tail entity. In addition, in the implementation of an English corpus, a long-tail entity needs to be an entity composed of two or more words or initials.

[0054] The following will be combined with Figure 4 describe the specific implementation of knowledge-aware long-tail entity detection in the present invention.

[0055] Figure 4 FIG. shows a schematic diagram of long-tail entity detection according to an embodiment of the present invention.

[0056] Here, taking a sentence in an English corpus as an example: " Kevin Durant , Brooklyn Nets playerin NBA , led the US men’s basketball team to the 2021 XXX Games in Japan.” (The reference translation can be: Kevin Durant , NBA Brooklyn Nets player, led USA the men’s basketball team to participate in the 2021 XXX Sports Meeting .)

[0057] The entities contained in the sentence are underlined in the text, specifically including: "Kevin Durant", "Brooklyn Nets", "NBA", "U.S", "XXX Games". Here, although "Japan (日本)" is an entity, it is not considered in the determination of long-tail entities because it is a single word (in other words, long-tail entities only consider entities composed of two or more words, or capital letter combinations such as "NBA", "U.S"). It is necessary to identify long-tail entities from these entities such as "Kevin Durant", "Brooklyn Nets", "NBA", "U.S", "XXX Games".

[0058] In specific operations, structured knowledge triples can be first extracted from a large-scale KG, and the entities in the KG can be linked to the target objects in the pre-trained samples through an entity linking tool (e.g., TAGME). In a large corpus, the distribution of entities in the corpus closely follows the power-law distribution, and the formula is as follows:

[0059]

[0060] Where C and α are hyperparameters, rank is the entity frequency rank, and Freq(e) is the frequency of entity e. It can be found that although a few entities appear frequently, most entities rarely appear in the pre-training corpus (i.e., present a long-tail distribution), which makes it difficult for PLM to learn better context representations.

[0061] Since the model can learn enough knowledge about high-frequency entities from the sentences where the high-frequency entities are located, injecting high-frequency relationship triples into PLM is not always beneficial to downstream tasks. This practice is more likely to cause negative knowledge injection. Therefore, injecting knowledge about long-tail entities instead of all entities appearing in the corpus can further improve the understanding ability of PLM. It should also be noted that the above entity analysis only considers the frequency in the pre-training corpus, ignoring the information of each entity in the KG and the importance of these entities in the sentence. To this end, the present invention proposes a knowledge-aware long-tail entity detection mechanism to select target entities for knowledge injection.

[0062] In a preferred embodiment of the present invention, three adjacent information of an entity are considered to characterize the "long tail" attribute of the entity, namely, the entire pre-trained corpus, the current input sentence, and the KG. For a specific entity e, the following three factors can be considered:

[0063] Entity frequency: the frequency of entities in the entire pre-training corpus, denoted as Freq(e);

[0064] Semantic importance: the importance of entity e in the sentence, denoted as SI(e);

[0065] Knowledge connectivity: The number of multi-hop neighboring nodes of entity e in the KG (i.e., the total number of times the entity appears as a head entity and a tail entity in the KG), denoted as KC(e).

[0066] The calculation example of Freq(e) has been shown above in conjunction with equation (1).

[0067] SI(e) refers to the semantic similarity between the sentence representation containing entity e and the sentence representation replacing e. The greater the similarity between sentences, the smaller the impact on the semantics of the sentence when the entity is replaced. o and h rep It is represented as the representation of the original sentence and the sentence after entity replacement. For simplicity, the inverse of the cosine similarity can be used to measure SI(e):

[0068]

[0069] In implementation, a special token "[UHN]" ​​can be used to replace entity e in a sentence. For example, Figure 4 As shown, use "[UHN][UHN]" ​​to replace "Brooklyn Nets" in the sentence.

[0070] KC(e) represents the importance of entity e in the triple neighboring structure, and can be calculated using the number of multi-hop neighboring entities:

[0071]

[0072]

[0073] Where R min and R max is a predefined threshold. Specifically, we limit the number of hops to calculate KC(e) to R min to R max |·| represents the number of adjacent entities in the set. The Hop function represents the number of multi-hops between entity e and entity e' in the KG structure. Then the degree of the "knowledge-aware long tail" KLT(e) of entity e is calculated as:

[0074]

[0075] II {x} is the indicator function, where x is a Boolean expression. freq is a predefined threshold. Given a sentence, we need to detect all entities and consider entities with KLT(e) scores below the average as knowledge-aware long-tail entities.

[0076] Given a corpus and KG, for example, “Brooklyn Nets” and “XXXGames” in the above example can be considered as long-tail entities and subsequent knowledge injection can be performed.

[0077] After the long-tail entities are determined, the triple information related to the long-tail entities included in the training samples can be recalled from the knowledge graph. Since there is usually more than one triple related to a long-tail entity, for example, the long-tail entity in the sentence is "Beijing", the triples in the knowledge graph include not only "China-Capital-Beijing", but also "China-Core City-Beijing" and "Moscow-Sister City-Beijing", so it is necessary to select from them. In one embodiment, all tail entities can be retrieved based on the head entity, and the triple to be selected is determined based on the maximum frequency of the tail entity appearing in the pre-trained text.

[0078] Subsequently, constructing a knowledge representation embedding for replacing the long-tail entity based on the triple information includes: when the long-tail entity is the head entity of the triple information, the knowledge representation embedding includes the tail entity embedding minus the relational word embedding; when the long-tail entity is the tail entity of the triple information, the knowledge representation embedding includes the head entity embedding plus the relational word embedding. For example, if <China, capital, Beijing> is taken as an example, "China" in the sentence will be replaced by "Beijing-capital", and "Beijing" in the sentence will be replaced by "China+capital". The "-" and "+" here are for the respective embeddings of "China", "capital", and "Beijing".

[0079] In a preferred embodiment, the knowledge representation embedding further includes: a concatenated explanation text of the long-tail entity, wherein the explanation text is selected from a corpus serving as a pre-training dataset.

[0080] Since the original corpus and the knowledge graph can be bound according to a certain mechanism, the corpus and the knowledge graph used as the pre-training dataset can be analyzed before pre-training to mark the long-tail entities contained in each sentence in the corpus and a specific triplet of information related to each long-tail entity recalled from the knowledge graph, and the entity explanation text can be extracted from the corpus, with each entity corresponding to an explanation sentence. For example, after "Beijing" is replaced by "China + Capital", the embedded representation of the entity explanation problem of "Beijing is a core city in China" is concatenated.

[0081] Specifically, in order to enhance the PLM's ability to understand long-tail entities, the present invention injects knowledge triples into the locations of such entities without introducing any other parameters. Here, a pseudo-token embedding injection technique is used. If the entity in the pre-trained sentence is the head entity of the knowledge triple e h , then e h The representation of is modeled by the following function:

[0082]

[0083] where h eh 、h et and h r They are the head entity e h , tail entity e and relation predicate (i.e., relation word) r. Similarly, if an entity is a tail entity e t and relation predicate r. Similarly, if an entity is a tail entity e in the KG t , we can get: et =h eh +h r .

[0084] Specifically, the underlying PLM can be used as a shared encoder to obtain knowledge representation. Here, for example, consider that the entity is the head entity e in the KG h The tail entity e t The tokens of are connected with the relational predicate r and provided to PLM. The token representation of the last layer of PLM is represented as F(e t ) and F(r).

[0085] Therefore, the pseudo-label represents h et and h r It can be calculated as follows:

[0086]

[0087]

[0088] Where LN is the LayerNorm function, f sp is a self-attention pooling operator used to generate span representation. et and W r is a trainable parameter.

[0089] Since the length of entity and relationship predicates is usually short, the representation generated by PLM may not be expressive. To this end, in one embodiment, the description text of the target entity can be further considered, represented as make Generated by PLM The head entity e h The pseudo-token embedding h eh It is expressed as follows:

[0090]

[0091] in refers to the concatenation of two representations, W eh is a trainable parameter.

[0092] The representation of the detected long-tail entities can be replaced with the pseudo-labeled representation in the PLM embedding layer (h eh or et , depending on whether the target entity is the head or tail entity in the KG). This follows consecutive multiple transformer-encoder layers to incorporate knowledge into the context representation without introducing any other new parameters for knowledge encoding.

[0093] After injecting the information of the relation triple into the model, it is unclear whether the model understands the injected knowledge. For this reason, it is necessary to adjust the neural network of the PLM during the pre-training process using the first loss function that predicts that the long-tail entity does not introduce loss. Specifically, a relational knowledge edge decoder can be designed to force the model to explicitly understand the injected knowledge. Specifically, a self-attention pooling mechanism can be used to obtain the mask entity span representation of the last layer:

[0094]

[0095] in is a learnable parameter. Given a head entity and the relational predicate h r The output of is represented by, the goal is to decode the tail entity. is the representation of the i-th token of the predicted tail entity, then:

[0096]

[0097] where δ d is the scaling factor, Equal to the initialization heuristic

[0098] Since the vocabulary is large, we can use the Sampled SoftMax function to compare the prediction results with the ground truth. Token-level loss function The definition is as follows:

[0099]

[0100]

[0101] where y i is the true mark, y n is Q(t n |t i) is the negative label sampled in the DKPLM model. Q(·|·) is the negative sampling function. N is the number of negative samples. In the DKPLM model, the training objectives may include the relational knowledge decoding pre-training task as above. Preferably, it may also include labeled level masked language modeling (MLM). Using MLM as an auxiliary task can alleviate the overfitting problem in small sample learning to a certain extent. To this end, the knowledge injection method of the present invention may also include: masking the non-entities in the training samples; inputting the masked training samples into the pre-trained language model PLM; obtaining a second loss value output by the PLM for the masked words, and adjusting the parameters of the neural network model in the PLM accordingly, wherein the second loss value function is defined as the cross entropy between the one-hot vector and the model predicted probability distribution.

[0102] Therefore, the total loss function of DKPLM can be expressed as follows:

[0103]

[0104] where λ1 is a hyperparameter, is the total decoder loss of the target entity consisting of multiple tokens. In an alternative embodiment, a simpler As a loss function, λ can also be a hyperparameter here.

[0105] In one embodiment, the model's ability to understand valid knowledge can also be improved by comparative document learning. To this end, the knowledge injection method of the present invention can also include: recalling similar triple information and invalid triple information similar to the triple information from the knowledge graph; performing comparative learning in the pre-training task for the triple information, the similar triple information, and the invalid triple information. Thus, by performing comparative learning on valid and invalid knowledge, the model can maximize the perception that the improvement of sentence comprehension ability by valid knowledge is far greater than that of invalid knowledge.

[0106] In one embodiment, a relationship tree can be constructed based on KG. The relationship tree is a tree extracted from KG and used to specifically represent relationships. The nodes in the relationship tree represent relationships. The nodes in the first layer may not include any relationships, and the second layer includes a large classification of relationships, followed by subsequent small classifications.

[0107] After constructing the relationship tree, similar triple information and invalid triple information can be recalled from the knowledge graph according to the distribution of the relationship tree. Specifically, the similar triple information and multiple invalid triple information with different comparison learning difficulties can be recalled according to the distance of the relationship on the relationship tree. For example, the recalled valid triple is <China, capital, Beijing>, and the similar triple can be <China, core city, Beijing>. The invalid triple information used for comparison can include three different invalid triples (also called "dissimilar triples" or "irrelevant triples") with low, medium and high comparison learning difficulties. Therefore, by comparing and learning valid and invalid knowledge, the model can maximize the perception that the improvement of sentence comprehension ability by valid knowledge is far greater than that of invalid knowledge.

[0108] Figure 5 The figure shows a pre-training schematic diagram of a DKPLM according to an embodiment of the present invention. The DKPLM of the present invention has two modifications based on the basic pre-training model: (1) the selection and injection method of entity types in pre-training sentences (2) the comparative learning of effective and invalid knowledge (3) the design of a new type of pre-training self-supervision task.

[0109] Take the pre-trained sentence input in the figure as an example. First, in the data preparation stage, the data source can be processed. The data source may include a corpus and a knowledge graph. As described above, it can be determined whether each sentence in the pre-training corpus (which is then used as a training sample) contains a long-tail entity based on the frequency of the entity in the pre-training data set, the connectivity of the entity in the knowledge graph, and the semantic importance of the sentence. As shown in the lower right part of the figure, in the sentence "Kevin Durant, Brooklyn Nets player in NBA, led the USmen's basketball team to the 2021 XXX Games in Japan.", according to the long-tail entity detection method described above, untrivial entities can be detected, including "Brooklyn Nets" and "XXX Games" in this sentence, that is, these rare long-tail entities need to be masked and pseudo-embedded. Injection operations are not performed on trivial entities, and some trivial tokens are masked and predicted for MLM auxiliary tasks.

[0110] In addition, as shown in the lower left part of the figure, a relationship tree can be constructed from the KG, and based on the relationship tree, the long-tail entities appearing in the sentence are recalled into valid relationship triples and invalid relationship triples, and the two are compared and learned in the pre-training task. Specifically, the valid relationship triples are the triples used for the subsequent pseudo-embedding injection operation as described above. Further, positive and negative example triples can be constructed, including similar triples similar to the valid relationship triples, and three invalid relationship triples (corresponding to the difficult, medium, and easy comparative learning difficulties).

[0111] Subsequently, the corresponding valid triple information after transformation is retrieved from the identified long-tail entity, and the same pre-trained model is used to encode it to obtain the corresponding knowledge embedding representation result. The injection process is to replace the corresponding input embedding according to whether the entity is the head entity or the tail entity using the idea of ​​the TransE algorithm. The text information and the injected embedding information will simultaneously pass through the same set of shared pre-trained models (such as BERT) to obtain the final feature representation enhanced by hidden context knowledge. Finally, in order to further improve the effect of knowledge injection, the model can be enabled to reconstruct the corresponding specific text of the injected knowledge triple information by decoding (corresponding to the entity-level decoder pre-training task on the left side of the top layer), and at the same time, different triple information is also compared and learned (corresponding to the infoNCE pre-training task of the triple in the middle of the top layer plus the morning), so that the model can perceive that the effective knowledge that helps sentence understanding is far greater than the invalid knowledge. Specifically, the contrastive learning input is a current triple (for example, China-capital-Beijing in the above example) and a similar triple (for example, China-core city-Beijing), as well as a group of dissimilar triples (divided into three levels of difficulty: difficult, medium, and easy). Similar and dissimilar triplets are retrieved through the relationship tree. The goal of contrastive learning is to bring the current triple (China-capital-Beijing) closer to similar triplets and further away from dissimilar triplets.

[0112] The specific loss function consists of two parts. The first is the knowledge decoding loss (i.e., ) The original black-box knowledge-enhanced pre-trained model is decoded to improve the model's interpretability by decoding the real injected knowledge text. It also includes triple contrastive learning loss to improve the model's effective knowledge perception ability. Finally, in order to avoid overfitting and retain the word prediction ability of the MLM head (corresponding to the token-level MLM pre-training task on the right side of the top layer), the algorithm sets a word masking task as an auxiliary loss.

[0113] The pre-trained knowledge enhancement model can be used in the fine-tuning and reasoning stages in exactly the same way as the basic pre-trained model. There is no need to introduce knowledge graph data, and the user only needs to input data for specific tasks.

[0114] This paper proposes a knowledge-enhanced pre-training model algorithm based on long-tail entity pseudo-embedding, which is used to decouple the model from the need for knowledge graphs in the three training processes and can be applied to various text understanding tasks. This algorithm does not require manual construction of large-scale available knowledge graphs in downstream tasks, and can achieve good results for different natural language understanding tasks.

[0115] The DKPLM model is not restricted by specific fields. The current pre-trained texts and knowledge graphs are open domains, not texts and graphs in specific fields (for example, the medical field, the financial field). Therefore, based on the principles of the present invention, a DKPLM model with enhanced functions in a specific field can be trained on the basis of the existing PLM using a corpus and knowledge graph in a specific field (for example, medical text and medical knowledge graph). Since the DKPLM in a specific field has been trained on data in a specific field, it is much better than the DKPLM that has not been trained on data in a specific field in terms of the text representation learned for tasks in the specific field (text classification, text training annotation, etc.). For example, in terms of the performance of specific medical tasks, it will be better than the performance of the DKPLM in the general field. And the medical DKPLM adapted to the service scenarios in the medical field can be used directly as BERT, and it can be called for any medical text tasks.

[0116] To this end, the present invention can also be implemented as an interactive system based on knowledge injection. Figure 6 An example of using the PLM trained by the present invention for actual interaction is shown.

[0117] Specifically, an interactive system based on knowledge injection includes: a user input receiving unit, used to obtain a specific domain-related query input by a user;

[0118] A question matching unit, comprising a pre-trained model (e.g., medical DKPLM, financial DKPLM, e-commerce DKPLM) injected with specific domain knowledge obtained by the method described above using a specific domain corpus and a specific domain knowledge graph, wherein the model identifies relevant long-tail entities in the specific domain-related inquiries and generates feedback accordingly; a feedback providing unit, used to provide the generated feedback to the user.

[0119] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the knowledge injection method for the pre-trained language model according to an embodiment of the present invention is shown.

[0120] See also Figure 7 , the computing device 700 includes a memory 710 and a processor 720 .

[0121] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 720 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0122] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0123] The memory 710 stores executable codes. When the executable codes are processed by the processor 720, the processor 720 can execute the knowledge injection method of the pre-trained language model mentioned above.

[0124] The knowledge injection method of the pre-trained language model according to the present invention and the interactive system equipped with the pre-trained model injected with knowledge acquired by the method have been described in detail above with reference to the accompanying drawings. The DKPLM of this invention has the following advantages:

[0125] By using the same set of shared parameters in the pre-trained model, the existing model can be used to maximize the representation of knowledge triples; the pseudo-embedding method can effectively reduce the problem of adding additional network parameters to align feature space during the knowledge injection process; parsing the injected triple knowledge during the pre-training process not only enhances the modeling of entity granularity during the injection process, but also greatly improves the interpretability of the model compared to previous work; by comparing and learning valid and invalid knowledge, the model can maximize the perception that the improvement of sentence comprehension ability by valid knowledge is far greater than that of invalid knowledge; and using MLM as an auxiliary task can alleviate the overfitting problem in small sample learning to a certain extent.

[0126] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0127] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0128] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0129] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A knowledge injection method for a pre-trained language model, comprising: Recall triples from the knowledge graph related to the long-tail entities contained in the training samples; Constructing a knowledge representation embedding for replacing the long-tail entity based on the triple information; Using training samples in which long-tail entities are replaced with the knowledge representation embedding for pre-training of the pre-trained language model; as well as During the pre-training process, the neural network of the pre-trained language model is adjusted using a first loss function that predicts that the long-tail entity does not introduce a loss; Also includes: Performing mask processing on non-entities in the training samples; Inputting the masked training samples into the pre-trained language model PLM; A second loss value output by the PLM for the masked word is obtained, and parameters of the neural network model in the PLM are adjusted accordingly, wherein the second loss value function is defined as a cross entropy between a one-hot vector and a model predicted probability distribution.

2. The method of claim 1, further comprising: It is determined whether each sentence in a corpus serving as a pre-training data set contains a long-tail entity, wherein the determination is performed based on a first factor, and the first factor is related to a frequency of occurrence of the entity in the corpus.

3. The method of claim 2 further comprises determining whether each sentence in the corpus serving as the pre-training data set contains a long-tail entity based on at least one of the following factors: a second factor related to the semantic importance of the entity in the sentence; and The third factor is related to the connectivity of the entity in the knowledge graph.

4. The method of claim 1, further comprising: Recalling similar triple information and invalid triple information of the triple information from the knowledge graph; For the triple information, the similar triple information and the invalid triple information, comparative learning is performed in a pre-training task.

5. The method of claim 4, further comprising: constructing a relationship tree according to the knowledge graph, and Recalling the triple information similar to the triple information and the invalid triple information from the knowledge graph includes: According to the distance of the relationship on the relationship tree, the similar triplet information and multiple invalid triplet information with different learning difficulties are recalled.

6. The method of claim 1, wherein: Constructing a knowledge representation embedding for replacing the long-tail entity based on the triple information includes: When the long-tail entity is the head entity of the triple information, the knowledge representation embedding includes the tail entity embedding minus the relation word embedding; When the long-tail entity is the tail entity of the triple information, the knowledge representation embedding includes the head entity embedding plus the relation word embedding.

7. The method of claim 1, wherein: The knowledge representation embedding also includes: The concatenated explanation texts of the long-tail entities are selected from a corpus serving as a pre-training dataset.

8. The method of claim 1, further comprising: Before pre-training, the corpus and knowledge graph used as pre-training datasets are analyzed to mark the long-tail entities contained in each sentence in the corpus and a specific triplet information related to each long-tail entity recalled from the knowledge graph.

9. An interactive system based on knowledge injection, comprising: A user input receiving unit, used to obtain a specific domain-related query input by a user; A question matching unit, comprising a pre-trained model injected with specific domain knowledge obtained by using a specific domain corpus and a specific domain knowledge graph according to any one of claims 1 to 8, wherein the model identifies relevant long-tail entities in the specific domain-related queries and generates feedback accordingly; The feedback providing unit is used to provide the generated feedback to the user.

10. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

Images