Method and interactive system for knowledge injection of a chinese pre-trained language model

Abstract

A knowledge injection method of a Chinese pre-training language model and a corresponding interactive system are disclosed. The method comprises: using a special identifier to annotate key semantic components in a pre-training sentence to construct a reconstructed pre-training sentence; performing mask processing on the reconstructed pre-training sentence; and inputting the mask-processed pre-training sentence into a PLM and adjusting parameters of a neural network model based on a first loss value output by the PLM for a masked word. The present application realizes knowledge injection for a Chinese pre-training language model through internal linguistic knowledge annotation and external knowledge graph injection, so that the model learns semantic and factual knowledge while the architecture remains unchanged. The model pre-trained in this way can greatly reduce the required parameter size and can complete various downstream tasks without external data support, and is suitable for providing various real-time services for users in a cloud environment.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and more particularly to a knowledge injection method for a Chinese pre-trained language model, and a Chinese interactive system equipped with a pre-trained model with knowledge injection obtained by this method. Background Technology

[0002] As a foundational model for Natural Language Processing (NLP), pre-trained language models (PLMs, including BERT, RoBERTa, XLNET, etc.) have achieved excellent results in various downstream Natural Language Understanding (NLU) tasks, demonstrating strong versatility. Mainstream pre-trained language models use English for pre-training, masking and predicting the basic building blocks of sentences, i.e., words. Since a single word in English is typically a complete semantic unit, a model that correctly predicts masked words can learn the semantics of that word within the sentence. However, in Chinese pre-trained language models, if the basic building blocks of sentences, i.e., individual Chinese characters, are also masked and predicted during pre-training, the model makes predictions based on local judgments of the characters flanking the masked character. That is, it can only learn the composition of the word, but cannot understand its semantics within the pre-trained sentence. This results in less than ideal performance of Chinese pre-trained language models when performing downstream tasks.

[0003] Therefore, a pre-trained model that can better learn Chinese knowledge is needed. Summary of the Invention

[0004] One technical problem this disclosure aims to solve is to provide a knowledge injection method for Chinese pre-trained language models. This method performs linguistic analysis and key semantic component labeling on pre-trained sentences, enabling the pre-trained language model to better learn the linguistic knowledge contained within the sentences. Furthermore, positive and negative examples of external triples can be constructed for the entities contained in the pre-trained sentences, and through contrastive learning, the pre-trained language model can better learn the factual knowledge contained in the external knowledge graph. This significantly reduces the parameter size of the Chinese pre-trained language model while improving the accuracy of various downstream tasks performed based on the Chinese pre-trained language model.

[0005] According to a first aspect of this disclosure, a knowledge injection method for a Chinese pre-trained language model is provided, comprising: labeling key semantic components in a pre-trained sentence with special identifiers to construct a reconstructed pre-trained sentence; masking the reconstructed pre-trained sentence; inputting the masked pre-trained sentence into the pre-trained language model PLM, and adjusting the parameters of the neural network model in the PLM based on a first loss value output by the PLM for the masked characters.

[0006] Optionally, the method further includes: recalling positive and negative triples corresponding to entities contained in the pre-trained sentence from the knowledge graph; inputting the words corresponding to the entities in the pre-trained sentence, the positive and negative triples into the encoder of the PLM; and constructing a second loss value for the hidden representations of the words of the entities output by the encoder, the representations of the positive and negative triples, and the representations of the negative triples to adjust the parameters of the neural network model in the PLM based on contrastive learning.

[0007] Optionally, the positive example triple is a single-hop triple that includes the entity, and the negative example triple is a multi-hop triple in the knowledge graph that is more than hops away from the entity.

[0008] Optionally, the number of hops between the multi-hop triple and the entity is not greater than a predetermined threshold.

[0009] Optionally, the second loss value is used to characterize the difference between positive example similarity and negative example similarity, wherein the positive example similarity characterizes the similarity between the hidden representation of the word of the entity and the representation of the positive example triple, and the negative example similarity characterizes the similarity between the hidden representation of the word of the entity and the representation of the negative example triple.

[0010] Optionally, masking the reconstructed pre-trained sentence may include masking at least some of the key semantic components.

[0011] Optionally, using special identifiers to annotate key semantic components in pre-trained sentences to construct reconstructed pre-trained sentences includes: adding semantic dependency tags before and after dependency grammar relation words in the pre-trained sentences; and adding dependency grammar relation tags before and after dependency syntactic relation words in the pre-trained sentences.

[0012] Optionally, masking the reconstructed pre-trained sentence includes: masking words or special identifiers in the reconstructed pre-trained sentence according to a predetermined ratio, and in the predetermined ratio, allocating a first ratio to random masking, a second ratio to dependency grammar relation words, and a third ratio to dependency syntax relation words.

[0013] According to a second aspect of this disclosure, a knowledge-injection-based interactive system is provided, comprising: a user input receiving unit for acquiring a domain-specific query input by a user; a question matching unit, comprising a Chinese pre-trained model using knowledge injection obtained from a Chinese corpus and a Chinese knowledge graph as described in the first aspect, wherein the model identifies relevant entities and semantics in the Chinese query and generates feedback accordingly; and a feedback providing unit for providing the generated feedback to the user. Multiple Chinese pre-trained models with different parameter sets can be trained to generate the feedback with different accuracy and speed in different interactive scenarios.

[0014] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0015] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0016] Therefore, this invention, based on the design of input data and pre-training tasks, achieves knowledge injection into the Chinese pre-trained language model through internal linguistic knowledge annotation and external knowledge graph injection. This allows the model to learn the linguistic knowledge of the pre-trained sentences themselves without changing the architecture, and to learn factual knowledge about entities contained in the pre-trained sentences from the external knowledge graph. The resulting pre-trained model can complete various downstream tasks with fewer parameters and without external data support, making it suitable for providing various real-time services to users in a cloud environment. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0018] Figure 1 Examples of masking and prediction for individual Chinese characters and individual words are shown in a Chinese pre-trained language model.

[0019] Figure 2 A schematic flowchart illustrating a knowledge injection method for a Chinese pre-trained language model according to an embodiment of the present invention is shown.

[0020] Figure 3 An example of linguistic tagging and reconstruction of pre-trained sentences according to the present invention is shown.

[0021] Figure 4 An example of a knowledge subgraph included in a knowledge graph is shown.

[0022] Figure 5 A pre-training schematic diagram of CKBERT according to an embodiment of the present invention is shown.

[0023] Figure 6 An example of the PLM trained by the present invention used for actual interaction is shown.

[0024] Figure 7 A schematic structural diagram of a computing device that can be used to implement the above knowledge injection method for a Chinese pre-trained language model according to an embodiment of the present invention is shown. Detailed Description of the Preferred Embodiments

[0025] Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] Pre-trained Language Model (PLM) is a general term for a class of natural language processing models, which are used to learn the representation of low-dimensional dense real-value vectors of text. Early pre-trained language models aimed to use shallow neural networks to learn word embedding representations, and these word embedding representations were used for various downstream natural language processing tasks; the latest pre-trained language models learn context-based word embedding representations, and the learned models are used for fine-tuning on downstream tasks. Pre-trained language models have achieved excellent results in various downstream natural language understanding (NLU) tasks and have strong versatility.

[0027] However, mainstream pre-trained language models use English for pre-training. During the pre-training process, the constituent unit of a sentence, that is, a single word ("word"), is masked, and the model is required to predict the masked word. Because a word in English is usually a complete semantic unit, for example, the word "Harbin" corresponds to a complete semantic unit "Harbin". Therefore, the model needs to use a wider range of information in the sentence to achieve correct prediction of the word. In other words, the masking and prediction of words enable English pre-trained language models to naturally learn the semantics of the word in the sentence.

[0028] However, in Chinese pre-trained language models, if the basic unit of a sentence, i.e., a single Chinese character, is masked and predicted during pre-training, the model predicts the masked character through local judgment based on the characters on both sides of the masked character. That is, the model can only learn the composition of words, but cannot acquire the semantics of the word in the pre-trained sentence. This results in unsatisfactory performance of Chinese pre-trained language models when performing downstream tasks.

[0029] Figure 1 An example of masking and predicting single Chinese characters and single words in a Chinese pre-trained language model is shown.

[0030] As shown in the figure, the pre-trained sentence is "Ha'erbin is the capital city of Heilongjiang, an international famous city of ice and snow culture". Figure 1 On the left is the masking method used by BERT during pre-training, for example. The data used by BERT only masks single characters. When training BERT, the character "Er" is inferred through the local co-occurrence of "Ha" and "Bin", but the model (shown as Transformer in the figure) does not actually learn knowledge related to "Ha'erbin". That is, it only learns the word "Ha'erbin" but does not know the meaning represented by "Ha'erbin". On the right, an example is shown where the data used in pre-training masks the whole word. Since the whole word is masked—for example, masking words such as "Ha'erbin" and "ice and snow"—the model (shown as Transformer in the figure) must learn the representation of words and entities to model the relationship between "Ha'erbin" and "Heilongjiang", learn the meaning that "Ha'erbin" is the capital of "Heilongjiang" and "Ha'erbin" is an ice and snow city, so as to correctly predict the masked word.

[0031] However, as Figure 1 shown on the right, masking and predicting words requires the model to learn the semantics in sentences by itself, and the masked words are often not important words in sentences, which leads to low learning efficiency of the model and huge parameter scale. To this end, the present invention proposes an improved knowledge injection method for Chinese pre-trained language models, which reconstructs pre-training samples by identifying and prompting key semantic components in pre-training sentences, enabling the model to quickly learn linguistic knowledge inside sentences, thereby improving the model's performance in performing downstream natural language understanding tasks while greatly reducing the parameter scale.

[0032] Figure 2 A schematic flowchart of a knowledge injection method for a Chinese pre-trained language model according to an embodiment of the present invention is shown.

[0033] In step S210, key semantic components in the pre-training sentences are labeled using special identifiers to construct reconstructed pre-training sentences. Here, key semantic components in a sentence refer to semantic components that are considered "key" from a linguistic perspective after linguistic analysis of the sentence. Because they need to have linguistic meaning, key semantic components are usually complete words.

[0034] Specifically, it can identify dependency grammar relation words and dependency syntax relation words in pre-trained sentences. Both "dependency grammar relation words" and "dependency syntax relation words" correspond to concepts in linguistics.

[0035] Syntactic parsing is a key technique in natural language processing. It involves analyzing input text sentences to obtain their syntactic structure. Dependency parsing analyzes a sentence into a dependency tree, describing the dependency relationships between words. In other words, it identifies the syntactic collocations between words, which are semantically related.

[0036] In dependency syntax theory, "dependency" refers to the dominant-subordinate relationship between words. This relationship is not equivalent and is directional. More precisely, the dominant element is called the agent (AGT). Dependency syntax considers the verb in the predicate to be the center of a sentence, with other elements directly or indirectly connected to the verb and being the objects to which the agent (AGT) points. In natural language processing, the framework for describing language structure using dependency relationships between words is called dependency grammar. Therefore, "dependency grammar relation words" can be used here to refer to the agent in a sentence, usually the predicate. After identifying dependency grammar relation words, subject-verb-object, modifier-head (modifier-headword) relationships, and coordinate relationships in a sentence can be processed according to dependency syntactic relations to find "dependency syntactic relation lexicalities."

[0037] After identifying the "dependency grammar relation words" and "dependency syntax relation words," special markers can be used to label key components in the sentence. Therefore, labeling key semantic components in a pre-training sentence using special identifiers to construct a reconstructed pre-training sentence can include: adding semantic dependency markers (e.g., adding [SDP] before the predicate word and [ / SDP] after it) before and after the dependency grammar relation words (e.g., typically the "predicate"); and adding dependency syntax relation markers (e.g., adding [DEP] before the headword and [ / DEP] after it) before and after the dependency syntax relation words (e.g., the "headword" pointed to by the modifier).

[0038] Figure 3 An example of linguistic tagging and reconstruction of a pre-training sentence according to the present invention is shown. The original pre-training sentence is "Everyone knows that more practice is the key to truly improving spoken pronunciation." Here, any existing or future-developed tools can be used to perform dependency parsing on the sentence, thereby identifying the "dependency relation words" and "dependency relation lexicalities" contained within the sentence. Figure 3 In the sentence shown, the agent "everyone" refers to the predicate "know," so "know" is used as a dependency relation word, and the identifiers [SDP] and [ / SDP] are added before and after it, respectively. The modifier "truly" refers to the verb "improve," so "improve" is used as a dependency relation word, and the identifiers [DEP] and [ / DEP] are added before and after it, respectively. Further, as shown in the figure, a sentence start identifier [CLS] can be added before the sentence and a sentence end identifier [SEP] can be added after the sentence. Thus, the original pre-training sentence "Everyone knows that more practice is needed to truly improve spoken pronunciation" is reconstructed as "[CLS]Everyone[SDP]knows[ / SDP]more practice is needed to truly[DEP]improve[ / DEP]spoken pronunciation[SEP]".

[0039] After reconstructing the pre-trained sentence as described above, a masking process can be performed on the reconstructed pre-trained sentence in step S220. In a preferred embodiment, masking the reconstructed pre-trained sentence may include masking at least a portion of the key semantic components. For example, for Figure 3 The key semantic words "know" and "improve" in the reconstructed pre-trained sentences are masked.

[0040] To simultaneously ensure prediction accuracy and learning rate, a predetermined proportion of characters or special identifiers in the pre-training samples are typically masked. In one embodiment of the present invention, this predetermined proportion may be 15%. Within this 15% predetermined proportion, a portion can be allocated to randomly masked individual Chinese characters, and another portion to key semantic words. This allows the model to learn not only the linguistic knowledge of the tags but also other knowledge from the pre-training sentences through random masking. In one embodiment, masking the reconstructed pre-training sentence may include: masking characters or special identifiers in the reconstructed pre-training sentence according to a predetermined proportion, and within the predetermined proportion (e.g., 15%), allocating a first proportion (e.g., 40% of 15%) to the random mask, a second proportion (e.g., 30% of 15%) to dependency grammar relation words, and a third proportion (e.g., 30% of 15%) to dependency grammar relation words. In a preferred embodiment, special identifiers ([DEP], [ / DEP], [SDP], and [ / SDP]) can also be regarded as general markers used for masking, so the model needs to know the boundaries of the predicted vocabulary rather than simply filling the mask according to the context.

[0041] Subsequently, in step S230, the masked pre-trained sentence can be input into the pre-trained language model PLM, and the parameters of the neural network model in the PLM can be adjusted based on the first loss value output by the PLM for the masked word.

[0042] Here, the first loss value corresponds to the first loss function, which is the loss function corresponding to the MLM task in this invention. Masked Language Modeling (MLM) is a self-supervised task that attempts to mask a word or phrase in a sentence and predict the masked word or phrase based on the rest of the sentence. However, unlike conventional MLM, the pre-training task of the pre-trained model in this invention is LMLM, i.e., linguistic-aware MLM. Through linguistic reconstruction of the pre-trained sentence as described above, masking of key semantic words and random characters, and prediction of the mask, the model can learn the linguistic knowledge contained in the sentence while performing the MLM task, thereby improving the model's semantic understanding and accelerating model convergence.

[0043] In the MLM-based training scheme, the training samples of PLM are masked text, i.e., sentences in which some characters are randomly replaced with special markers (e.g., [MASK]). For example, the pre-training sentence reconstructed by the linguistics of this invention is "[CLS]Everyone[SDP]knows[ / SDP]to practice more, only then can we truly[DEP]improve[ / DEP]spoken pronunciation[SEP]". An example of the masked text is "[CLS]Everyone[SDP] [MASK][MASK] [ / SDP][MASK]to practice more, only then can we truly[DEP][MASK][MASK][ / DEP]spoken pronunciation[SEP]" (i.e., the key semantic words "know" and "improve" are masked, and the first "more" is randomly masked). The masked text is input into PLM, and PLM needs to predict the masked content as "know", "way", "more", "change", and "good". The training samples of PLM can be called masked training samples. In a sentence, for the masked content, the unmasked content is its contextual information (and the special identifier information in this invention). PLM learns the ability to capture textual contextual information and the linguistic meaning of masked keywords by predicting the masked content. Therefore, PLM trained based on the LMLM training scheme of this invention has the ability to understand deep semantics of natural language and can be used for a series of downstream NLP-related tasks. Furthermore, because sentence reconstruction and the selection of semantic keywords enable the model to efficiently learn linguistic knowledge, the model can support the same information content with a smaller parameter scale. This results in a smaller pre-trained model size, making it more suitable for subsequent deployment in practical application scenarios.

[0044] Pre-trained models represent learning (determining) ideal values ​​for all weights and biases through labeled samples. These determined weights and biases then enable high-accuracy inference of input feature values ​​during the neural network deployment phase, such as correct prediction of masked Chinese characters based on context.

[0045] In self-supervised learning, machine learning algorithms learn parameters by examining multiple samples and trying to find a model that minimizes loss; this process is called empirical risk minimization.

[0046] Loss is a penalty for poor predictions. That is, loss can be a numerical value representing how accurately the model predicts for a single sample. If the model's predictions are perfectly accurate, the loss is zero; otherwise, the loss will be large. The goal of training a model is to find a set of weights and biases with a "small" average loss from all samples.

[0047] In the training and fine-tuning of neural networks, a loss function needs to be defined to quantify whether the current weights and biases can allow the network input to fit all network inputs. Therefore, the goal of training the network can be transformed into minimizing the loss function of the weights and biases. Typically, gradient descent (and backpropagation in multi-layer neural network training) is used to achieve this minimization.

[0048] The backpropagation algorithm involves repeated iterative processes of forward and backward propagation. Forward propagation is the process by which neurons in different layers are connected through weight matrices, allowing stimuli (feature values) to be continuously transmitted from one layer to the next via the activation function of each layer. In backpropagation, the error of the current layer needs to be derived backward from the error of the next layer. Thus, through the iterative processes of forward and backward propagation, the weights and biases are continuously adjusted, causing the loss function to gradually approach its minimum, thereby completing the training of the neural network. In this invention, the loss function for the LMLM task can be, for example, the first loss function described below. accomplish.

[0049] Most mainstream pre-trained language models are based on publicly available documents, learning general language knowledge from unstructured documents, neglecting the learning of large amounts of knowledge information, especially structured knowledge graph (KG) information. Here, unstructured and structured refer to the way language knowledge is presented. In natural language processing, the presentation of language knowledge typically includes three forms: unstructured text, semi-structured tables, and structured triples. Specifically, triple knowledge is stored in artificially constructed large-scale knowledge graph data, consisting of <head entity, relation, tail entity>. The head and tail entities represent a specific thing that exists in the real world (e.g., Hangzhou), and the relation expresses a semantic connection between entities (e.g., birthplace).

[0050] In PLM, the two-stage strategy (i.e., pre-training and fine-tuning) inherits the knowledge learned during pre-training and applies it to downstream tasks. While PLM stores a large amount of internal knowledge, it struggles to understand external contextual knowledge, such as facts and common sense, because it learns general language knowledge from unstructured documents and lacks systematic learning of structured knowledge. This lack of knowledge leads to counterfactual statements (for example, the GPT model might output obviously false assertions like "the sun has two eyes"), and significantly weakens the model's few-shot learning ability, domain knowledge transfer ability, and general knowledge induction ability.

[0051] Therefore, in one embodiment, when the pre-trained sentence includes entities, the present invention can further improve the performance of the Chinese PLM by injecting external knowledge triples associated with the entities included in the sentence. The PLM with injected external knowledge can be called a Knowledge-Enhanced Pre-trained Model (KEPLM). In the prior art, KEPLM can further improve the performance of downstream tasks by semantically understanding key entity information in the text, based on the pre-trained model modeling. However, the biggest problem with the above methods is that during downstream task training and inference, the knowledge-enhanced pre-trained model still needs to pre-construct a large-scale, usable knowledge graph, and the additional network parameters increase the burden on computational resources. Therefore, it is cumbersome and its performance is not stable enough in practical applications.

[0052] Therefore, this invention aims to enable knowledge-enhanced models to achieve good performance in downstream tasks even without using knowledge graph information during the fine-tuning and inference stages. To this end, this invention proposes a method based on a shared encoder that combines knowledge encoding, knowledge injection, and pre-training. The model is modified only at the data input and pre-training task levels, without altering the model architecture. This allows the model to perform well in downstream tasks without requiring additional parameters and without relying on external knowledge graphs (i.e., external knowledge has already been injected into the model during the pre-training stage).

[0053] Specifically, the present invention can utilize knowledge graphs to construct positive and negative triples of related entities, and inject factual knowledge contained in external knowledge graphs into the model based on contrastive learning.

[0054] A knowledge graph is a knowledge base where data is integrated through a graph-structured data model or topology. Knowledge graphs are typically used to store entities that are interconnected. By effectively processing, handling, and integrating complex document data, knowledge graphs transform it into simple, clear triples of <head entity, relation, tail entity>, ultimately aggregating vast amounts of knowledge to achieve rapid knowledge response and reasoning. Figure 4 This illustrates an example of a knowledge subgraph included in a knowledge graph. A knowledge graph is a knowledge base composed of a series of triples. Multiple triples with interconnected relationships extracted from it can construct a knowledge subgraph, such as... Figure 4 As shown in the image. Figure 4 An example of a knowledge subgraph centered on entity "Person C" is shown. Figure 4 In the diagram, the circles represent nodes in the knowledge subgraph and correspond to different entities. Arrowed lines represent edges in the knowledge subgraph, with the arrow pointing from the head entity to the tail entity. The text on the edges indicates the relationship between the head and tail entities.

[0055] For example, when the pre-training sentence is "Character C is a shining pearl in the history of 20th-century literature," the training can target the entity "Character C" identified from the sentence, based on... Figure 4 The knowledge subgraph shown constructs positive and negative triplets for external knowledge injection.

[0056] Therefore, in one embodiment, the knowledge injection method of the Chinese pre-trained language model of the present invention may further include: recalling positive triples and negative triples corresponding to entities contained in the pre-trained sentence from a knowledge graph; inputting the characters corresponding to the entities in the pre-trained sentence, the positive triples and the negative triples into the encoder of the PLM; and constructing a second loss value for the hidden representation of the characters of the entity output by the encoder (i.e., the embedding obtained by model processing), the representation of the positive triples and the representation of the negative triples, so as to adjust the parameters of the neural network model in the PLM based on contrastive learning.

[0057] Here, the positive example triple is a one-hop triple containing the entity, and the negative example triple is a multi-hop triple in the knowledge graph that is more than hops away from the entity. Figure 4 For example, when the pre-training sentence includes the entity "Person C", a single-hop triple can be a triple consisting of "Person C", the nodes reached from "Person C" in the knowledge subgraph by a single hop (e.g., "Location F", "Novelist", "PhD in Literature", "G Award", and "Work E" in the graph), and the relationships corresponding to the edges along which the single hop occurs. In other words, a single-hop triple is a triple that includes the target entity "Person C" contained in the sentence, and preferably a triple with the target entity as the head entity. This is because the features of an entity are more obvious when it is the head entity. In one embodiment, a triple can be randomly selected from multiple triples with the target entity as the head entity as the positive example triple. In another embodiment, the most representative triple can be selected from multiple triples with the target entity as the head entity. For example, the selection can be based on indicators such as the number of other entities connected to the pointed-to tail entity as the head entity, or the frequency of entity occurrences in the corpus. Figure 4 In the example shown, the most representative triplet can be selected from the five single-jump triplets as the positive triplet, namely <person C, representative work, work E>.

[0058] Multi-jump triples start from "Character C" and jump to nodes reached by skipping at least two edges. Figure 4In the example, <Work E, Female Protagonist, Character B>, <Work E, Year of Occurrence, Year a>, <Character B, Famous Figure, Character A>, and <Location F, Celebrity, Character D> can all be considered as triples obtained through multiple hops starting from "Character C". In different embodiments, the number of hops can be limited to construct negative example triples with different contrastive learning "difficulties". In one embodiment, the number of hops between the multi-hop triple and the entity can be set to no more than a predetermined threshold. δ If the hop count threshold δ The size is too large, and due to the significant semantic difference, the model can easily distinguish between positive and negative triples. For effective contrastive learning, good negative triples should be "difficult," therefore... Figure 4 In the example, you can set δ The value is 3. Therefore, <Work E, Female protagonist, Character B>, <Work E, Year of occurrence, Year a>, <Character B, Famous figure, Character A>, and <Location F, Celebrity, Character D> in the knowledge subgraph can all be regarded as negative triples.

[0059] After obtaining the positive and negative triplets, each triplet can be converted into a natural language input model, and a second loss function can be used to achieve contrastive learning. The triplets can be directly concatenated into a sentence or rewritten into a more appropriate sentence. For example, for the triplet <work E, female protagonist, character B>, it can be directly concatenated as "work E female protagonist character B", or simply rewritten as "the female protagonist of work E is character B". In one embodiment, the second loss value is used to characterize the difference between positive and negative similarity, where the positive similarity characterizes the similarity between the hidden representation of the entity's words and the representation of the positive triplet, and the negative similarity characterizes the similarity between the hidden representation of the entity's words and the representation of the negative triplet. The second loss function can be implemented, for example, as detailed below. Similar to internal linguistic knowledge injection, injecting factual knowledge from external knowledge graphs enables the model's parameters to learn knowledge more effectively, thus further reducing the size of the model parameters required to achieve the same predictive performance.

[0060] In one embodiment, the pre-trained language model can be fine-tuned based on both a linguistically aware MLM task and a contrastive learning task. Therefore, the overall loss function can be the first loss function. Second loss function The sum, or weighted sum.

[0061] Therefore, by injecting internal linguistic knowledge and external knowledge graphs through the method of this invention, the resulting Chinese pre-trained language model can learn more semantic and factual knowledge, thereby ensuring the performance of subsequent downstream tasks.

[0062] Furthermore, it should be noted that sentence reconstruction targets key semantic words within the sentence, such as those mentioned above. Figure 3 The input typically consists of verbs in the main or subordinate clause, while knowledge graph-based processing targets entities within the sentence. In other words, the simultaneous injection of internal linguistic knowledge and external knowledge graph knowledge usually targets different words in the pre-trained sentences, thereby maximizing knowledge learning from a limited set of pre-trained sentences. This further reduces the model's required parameters and improves the performance of subsequent downstream tasks.

[0063] The following will combine Figure 5 This describes the specific implementation of the knowledge injection scheme of the present invention. Figure 5 A schematic diagram of CKBERT pre-training according to an embodiment of the present invention is shown. CKBERT (Chinese Knowledge-Enhanced BERT) can be regarded as an implementation example of the knowledge injection scheme of the Chinese pre-trained language model according to the present invention.

[0064] Figure 5 The implemented model uses existing Chinese PLM models, such as BERT, and only modifies the data input and pre-training task levels, without altering the model architecture, thus facilitating parameter expansion. At the data input layer, two parts of knowledge are processed: external knowledge graph triples and sentence-level internal linguistic knowledge. For linguistic knowledge, existing or future open-source tools can be used to process pre-trained sentences for semantic role labeling and dependency parsing, and important components in the recognition results are labeled according to rules. For external triple knowledge, positive and negative triple samples are constructed based on entities appearing in the sentence. Positive samples are sampled from single-hop entities in the graph, and negative samples are sampled from multi-hop entities in the graph. However, the sampling process for negative samples is limited to a specified multi-hop range and cannot be too far away in the graph.

[0065] Specifically, the CKBERT of this invention shares the same model backbone as BERT. It accepts a sequence of N WordPiece tags ( x 1, x 2, ..., x N As input, a D-dimensional context representation is computed by stacking N transformer encoder layers consecutively. H i∈ Here, the architecture is not modified to ensure that CKBERT can be seamlessly integrated into any industrial application supported by BERT with better performance. In other embodiments, the model architecture can also be extended.

[0066] CKBERT includes the following two pre-training tasks: • Linguistic Aware Masked Language Modeling (LMLM): LMLM extends Masked Language Modeling (MLM) by introducing two key linguistic tokens derived from dependency parsing and semantic role tagging. A unique token can be inserted for each linguistic component in a continuous set of tokens. The goal of LMLM is to predict randomly selected tokens and linguistic tokens masked in pre-trained sentences.

[0067] • Contrastive Multi-hop Relation Modeling (CMRM): This approach samples fine-grained subgraphs from a large-scale Chinese knowledge base (KG) using multi-hop relations to supplement the understanding of contextual knowledge about target entities. Specifically, positive triples are constructed for matching target entities by retrieving single-hop entities from the corresponding subgraphs. Negative triples are sampled from unrelated multi-hop entities through relational paths within the KG. The CMRM task is proposed to bring similar entities closer together semantically and push away those that are unrelated. Aggregating heterogeneous knowledge information can further benefit the context-aware representation of PLM.

[0068] In BERT pre-training, 15% of all labeled locations are randomly masked for prediction. However, the randomly masked labels may be unimportant units, such as conjunctions and prepositions. Therefore, this invention reconstructs the input pre-training sentence and masks more labels based on linguistic knowledge, so that CKBERT can better understand the semantics of important labels in the pre-training sentence. The following three steps can be used to mask the linguistic input units: • Identifying Linguistic Markers: Existing tools can be used to identify important units in the pre-trained sentences, including dependency grammar and semantic dependency parsing. The extracted relations here serve as an important source of linguistic knowledge. As shown in the lower right of the figure, the original pre-trained sentence is extracted as the object that the agent "everyone" refers to in the sentence "Everyone knows that more practice is the only way to truly improve spoken pronunciation," i.e., the predicate "knows," which is used as a dependency grammar relation word. The head noun "improve" modified by the modifier "truly" is extracted as a dependency syntax relation word.

[0069] • Reconstructing the input sentence: Based on the original input form, special identifiers are inserted between lexical spans based on the key semantic components of the extracted linguistic relations, providing clear boundary information for model pre-training. For example, we add [DEP] and [ / DEP] to dependency grammar relation words, and [SDP] and [ / SDP] to dependency syntax relation words.

[0070] • Masking Tag Selection: 15% of the tag positions from the reconstructed input sentence are masked using special tags [MASK]. Of these tags, 40% are assigned to randomly selected tags, and the remainder to linguistic tags. Here, special identifiers ([DEP], [ / DEP], [SDP], and [ / SDP]) are also treated as ordinary tags for masking, so the model needs to be aware of predicted word boundaries rather than simply filling in the mask based on context. This results in the reconstructed pre-trained sentence shown in the middle of the right side of the figure, with the gray-marked portions masked.

[0071] After the input sentence is processed as described above, the resulting linguistic mask token is fed into a model consisting of N stacked multi-head self-attention layers and FFN units. For the LMLM task, let Ω = ( m 1, m 2, m 3,..., γ K 1, γ K ) represents a sentence X The index marked by the mask in the middle, where m i It is an index of a random mask marker. γ i It is the index of the selected linguistically significant mask mark. K This is the total number of mask markers. Let... X Ω express X The set of mask markers in the middle, X Ω This represents the set of observed (unmasked) tokens. An example implementation of LMLM could be as follows:

[0072] in Indicates a randomly selected mark or linguistic mark. θ This represents the set of parameters for the model.

[0073] Beyond the LMLM task, when pre-trained sentences include entities, relation triples can be further injected into CKBERT to enable it to understand the contextual factual knowledge of the entities. For entities in the pre-trained sentences, positive and negative relation triples are constructed as follows: • Positive triples: Entity links are used to connect entities in the pre-trained sentences to target entities in the knowledge graph. Relation triples, i.e., single-hop entities, are considered candidate positive triples. Next, a relation triple is randomly selected from the candidates as a positive sample, denoted as . t p .

[0074] • Negative triples: Because of positive triples t p The semantic similarity between relation triples decreases along the KG path, therefore it can be obtained from the target entity. e t Start building with multiple jumps L One candidate negative triplet For example, in Figure 5 In the middle, with the target entity e Starting at node 0, nodes are retrieved along each edge. This results in a multi-hop relationship. H op ( , e 0, e end , r End node of ) e end ,in H op (·) represents a knowledge graph middle e 0 and e end The shortest distance between them. Here, if... H op (·)>1 and not greater than a small threshold δ (In the example in the picture) δ If the sum is 3), then a triplet is considered a negative triplet. Here, we can set... δ =3. Therefore, Figure 5 middle e 0 has four negative triplets. A sample's three-hop path is e 0→ e 2→ e 6→ e 9.

[0075] Taking the pre-trained sentence "Character C is a shining pearl in the history of 20th-century literature" as an example, the entity "Character C" contained in the sentence can be linked to the corresponding entity in the knowledge graph as the target entity.e 0, and targeting Figure 4 The knowledge subgraph shown constructs positive and negative triples. For example, a single-hop triple <person C, representative work, work E> is used as a positive sample. t p (Where, entity "work E" can be considered to correspond to...) Figure 5 Entities in e 2). e The four negative triples of 0 can correspond to <work E, female protagonist, character B ( e 6) > < Work E, Year of Occurrence, Year a ( e 7) >, < Person B, Famous Figure, Person A ( e 9) > and < Location F ( e 5), Celebrities, People D ( e 8)>.

[0076] The CMRM task aims to bring similarity triples of target entities closer together and push away irrelevant multi-hop relation triples, thereby enhancing the external contextual knowledge of target entities from the knowledge graph. Specifically, when retrieving target entities... e t Positive samples t p and negative samples After that, the target entity can be obtained. e t The context-aware representation is as follows:

[0077] in h et The target entity e t The hidden representation is derived from the entity's tags, since an entity can have multiple tags in a pre-trained sentence. Build. f sp It is a self-attention pooling operator. σ (·) is the non-linear activation function GELU. (·) is the LayerNorm function. W1 is the learnable weight matrix.

[0078] Furthermore, since relation triples can be viewed as natural sentences by concatenating their tags, triples can be transformed into sentences to generate sentences generated by the shared encoder. θ The representation obtained (which could be the Transformer encoder of the CKBERT model). Therefore, positive triples can also be derived. and negative triplet The representation is as follows. For the CMRM task, InfoNCE can be used as the loss function to calculate similarity, as shown below:

[0079] Where cos(·,·) represents the cosine function used to calculate the similarity between entity and relation representations. τ These are predefined hyperparameters.

[0080] For model training optimization, the total loss function for pre-trained CKBERT can be given based on the two pre-training tasks above, as shown below: (4) The present invention can also be implemented as a Chinese interactive system based on knowledge injection. Figure 6 An example of using the PLM trained by this invention for actual interaction is shown. Specifically, a knowledge-injected interaction system includes: a user input receiving unit for acquiring Chinese queries input by the user; a question matching unit, including a pre-trained Chinese model (e.g., CKBERT) with knowledge injection obtained using the method described above with a Chinese corpus and a Chinese knowledge graph, the model identifying relevant entities and semantics in the Chinese query and generating feedback accordingly; and a feedback providing unit for providing the generated feedback to the user.

[0081] A series of CKBERT models can be pre-trained on a distributed GPU cluster. For example, multiple pre-trained Chinese models with different parameter sets can be trained to generate the feedback with varying accuracy and speed in different interactive scenarios. Due to the infusion of linguistic and factual knowledge during the pre-training phase, the parameter sets of the models with different parameter sets are significantly reduced compared to the other models.

[0082] Figure 7 A schematic diagram of a computing device is shown, which can be used to implement the knowledge injection method of the above-described Chinese pre-trained language model according to an embodiment of the present invention.

[0083] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0084] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0085] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0086] The memory 710 stores executable code. When the executable code is processed by the processor 720, the processor 720 can execute the knowledge injection method of the pre-trained language model described above.

[0087] The knowledge injection method for a Chinese pre-trained language model according to the present invention, and the interactive system equipped with the Chinese pre-trained model with knowledge injection obtained by the method, have been described in detail above with reference to the accompanying drawings.

[0088] This invention, based on the design of input data and pre-training tasks, achieves knowledge injection into the Chinese pre-trained language model through internal linguistic knowledge annotation and external knowledge graph injection. This allows the model to learn the linguistic knowledge of the pre-trained sentences themselves without changing the architecture, and to learn factual knowledge about entities contained in the pre-trained sentences from the external knowledge graph. The resulting pre-trained model can complete various downstream tasks with a significantly reduced parameter size and without external data support, making it suitable for providing various real-time services to users in a cloud environment.

[0089] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0090] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0091] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A knowledge injection method for a Chinese pre-trained language model, comprising: Add semantic dependency tags before and after dependency grammar relation words in pre-trained sentences; And add dependency syntax markers before and after the dependency syntax relation words in the pre-trained sentences; The words or special identifiers in the pre-trained sentence are reconstructed according to a predetermined ratio mask, and in the predetermined ratio, a first ratio is allocated to a random mask, a second ratio is allocated to dependency grammar relation words, and a third ratio is allocated to dependency syntax relation words. as well as The pre-trained sentence after masking is input into the pre-trained language model PLM, and the parameters of the neural network model in the PLM are adjusted based on the first loss value output by the PLM for the masked word. Also includes: Retrieve positive and negative triplets from the knowledge graph that correspond to entities contained in the pre-trained sentences; The words corresponding to the entities in the pre-trained sentences, the positive triples, and the negative triples are input into the encoder of the PLM; and A second loss value is constructed for the hidden representation of the entity's word output by the encoder, the representation of the positive triplet, and the representation of the negative triplet to adjust the parameters of the neural network model in the PLM based on contrastive learning.

2. The method as described in claim 1, wherein, The positive example triple is a single-hop triple that includes the entity, and the negative example triple is a multi-hop triple in the knowledge graph that is more than hops away from the entity.

3. The method as described in claim 2, wherein, The number of hops between the multi-hop triplet and the entity is not greater than a predetermined threshold.

4. The method of claim 1, wherein, The second loss value is used to characterize the difference between positive example similarity and negative example similarity, wherein the positive example similarity characterizes the similarity between the hidden representation of the word of the entity and the representation of the positive example triple, and the negative example similarity characterizes the similarity between the hidden representation of the word of the entity and the representation of the negative example triple.

5. The method of claim 1, wherein, Masking the reconstructed pre-trained sentences includes: Masking is applied to at least some key semantic components.

6. A Chinese interactive system based on knowledge injection, comprising: The user input receiving unit is used to acquire Chinese queries input by the user. The question matching unit includes a Chinese pre-trained model that uses knowledge injection from a Chinese corpus and a Chinese knowledge graph obtained by the method of any one of claims 1-5, wherein the model identifies relevant entities and semantics in the Chinese query and generates feedback accordingly; A feedback providing unit is used to provide the generated feedback to the user.

7. The system of claim 6, wherein, Multiple Chinese pre-trained models with different parameter amounts are trained using the method described in any one of claims 1-5, and are used to generate the feedback with different accuracy and speed in different interactive scenarios.

8. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-5.

9. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-5.

Images