Knowledge enhanced pre-training language model training method, application method and device

Abstract

The application discloses a kind of training method, application method and device of knowledge enhanced pre-training language model.The method comprises: hyperbolic space learning is carried out to domain knowledge graph, and the hyperbolic entity embedding representation of the domain knowledge graph is obtained;The hyperbolic entity embedding representation is fused with the pre-constructed positive sample data, to obtain the fused positive sample data;The fused positive sample data is injected into the existing text training data of the pre-training language model, to obtain knowledge enhanced text training data and knowledge enhanced positive sample data;Knowledge enhanced text training data and the knowledge enhanced positive sample data are used to train pre-training language model.The application can effectively improve the performance of pre-training language model to execute corresponding domain text task.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a training method, application method, and apparatus for a knowledge-enhanced pre-trained language model. Background Technology

[0002] Deep learning-based pre-trained models can learn lexical, syntactic, and semantic information embedded in text from large-scale unsupervised data, achieving significant breakthroughs in downstream tasks in the natural language processing domain. However, pre-trained models also face the following challenges: They are based on statistical methods, learning implicit relationships between entities in text based on co-occurrence information. This results in a lack of deep understanding and logical reasoning capabilities. Knowledge, on the other hand, can provide pre-trained models with more comprehensive and richer semantic and relational information about entities, enhancing their robustness by providing common sense and domain knowledge. Against this backdrop, researchers in this field have continuously attempted to infuse knowledge into pre-trained models to better apply them to knowledge-driven and semantic understanding tasks, leading to the proposal of Knowledge-enhanced Pre-trained Language Models (KEPLM) to address this issue.

[0003] Knowledge-enhanced pre-trained language models have garnered widespread attention from academia and industry since the introduction of the next-generation pre-training-fine-tuning algorithmic paradigm. In recent years, the rapid development of open-domain pre-trained language models has significantly improved the performance of various downstream natural language processing tasks.

[0004] However, in certain vertical domains, such as finance, healthcare, and education, the inconsistency between the distribution of basic pre-training data and domain data means that knowledge in closed domains (such as the aforementioned vertical domains) exhibits global sparsity and local density compared to open text corpora or general knowledge graph data. This results in the original basic model being unable to perform good representation learning for domain text tasks. At the same time, domain pre-training data is also extremely scarce, making it impossible to conduct sufficient pre-training again to obtain a suitable domain pre-trained model. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a training method, application method and apparatus for a knowledge-enhanced pre-trained language model that overcomes or at least partially solves the above problems.

[0006] In a first aspect, embodiments of the present invention provide a training method for a knowledge-enhanced pre-trained language model, characterized in that it includes:

[0007] Hyperbolic space learning is performed on the domain knowledge graph to obtain the hyperbolic entity embedding representation of the domain knowledge graph;

[0008] The hyperbolic entity embedding representation is fused with pre-constructed positive sample data to obtain fused positive sample data;

[0009] The fused positive sample data is injected into the existing text training data of the pre-trained language model to obtain knowledge-enhanced text training data and knowledge-enhanced positive sample data.

[0010] The pre-trained language model is trained using knowledge-enhanced text training data and the knowledge-enhanced positive sample data.

[0011] In one embodiment, hyperbolic space learning is performed on the domain knowledge graph to obtain hyperbolic entity embedding representations, including:

[0012] By using the Poincaré sphere model, the tree-like hierarchical structure of the domain knowledge graph is learned, and the hyperbolic entity embedding representation of the domain knowledge graph is obtained.

[0013] In one embodiment, the method further includes: pre-constructing negative sample data;

[0014] Accordingly, training the pre-trained language model using the knowledge-enhanced text training data and the knowledge-enhanced positive sample embedding representation includes:

[0015] Using knowledge-enhanced text training data, a pre-defined masked language model is trained to obtain the masked language model loss.

[0016] The positive sample data enhanced by the knowledge are compared with the negative sample data to obtain the contrastive learning loss;

[0017] The total loss of the pre-trained language model is obtained based on the masked language model loss and the contrastive learning loss, and the pre-trained language model is trained based on the total loss.

[0018] In one embodiment, the total loss of the pre-trained language model is obtained based on the masked language model loss and the contrastive learning loss, including:

[0019] The total loss of the pre-trained language model is obtained by calculating the weighted sum of the masked language model loss and the contrastive learning loss.

[0020] In one embodiment, the positive sample data is pre-constructed in the following manner:

[0021] Random sampling is performed on entity pairs with first-order relations in the domain knowledge graph, wherein the entity pairs include a head entity and a tail entity;

[0022] Based on the sampled head entity, tail entity, and the relationship between the head and tail entities, a triplet text is constructed.

[0023] The positive sample data is generated by combining the embedded representations of the triplet texts of different entity pairs.

[0024] In one embodiment, the pre-built negative sample data includes:

[0025] Use entity nodes in the domain knowledge graph as the center nodes of concentric circles;

[0026] Starting from the central node, and according to a preset relationship jump distance, randomly search outwards along different relationships to obtain the ending node; construct text for different paths from the central node to the ending node;

[0027] Negative sample data is generated based on the embedding representation of the text from the different paths.

[0028] In one embodiment, the preset jump distance includes multiple different jump distances;

[0029] Accordingly, negative sample data is generated based on the embedded representations of the text from the different paths, including:

[0030] For different jump distances, the embedded representations of texts from different paths at the same jump distance are combined to generate negative sample data at different levels.

[0031] In one embodiment, the method further includes: if there are at least two paths between the center node and the end node, selecting the path with the shortest jump distance from the at least two paths to generate the negative sample data.

[0032] Secondly, embodiments of the present invention provide an application method in the field of natural language, wherein the application method uses a knowledge-enhanced pre-trained language model to perform tasks in the corresponding field.

[0033] The knowledge-enhanced pre-trained language model is obtained through the training method described above.

[0034] Thirdly, embodiments of the present invention provide a training apparatus for a knowledge-enhanced pre-trained language model, comprising:

[0035] The hyperbolic space learning module is used to perform hyperbolic space learning on the domain knowledge graph to obtain the hyperbolic entity embedding representation of the domain knowledge graph;

[0036] The entity space fusion module is used to embed the hyperbolic entity into a representation and fuse it with pre-constructed positive sample data to obtain fused positive sample data.

[0037] The entity knowledge injection module is used to inject the fused positive sample data into the existing text training data of the pre-trained language model to obtain knowledge-enhanced text training data and knowledge-enhanced positive sample data.

[0038] The training module is used to train the pre-trained language model using knowledge-enhanced text training data and the knowledge-enhanced positive sample data.

[0039] Fourthly, embodiments of the present invention provide a computing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned training method for a knowledge-enhanced pre-trained language model.

[0040] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned training method for a knowledge-enhanced pre-trained language model.

[0041] Sixthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the aforementioned training method for a knowledge-enhanced pre-trained language model.

[0042] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0043] The training method, application method, and apparatus for knowledge-enhanced pre-trained language models provided in this invention address the global sparsity and local density characteristics of closed-domain knowledge graphs compared to open-domain data. By learning the structural hierarchy of the domain knowledge graph and fusing the structural learning results with positive samples of the pre-trained model, the semantic gaps caused by the global sparsity of the domain knowledge graph are effectively supplemented, improving the semantic modeling ability of entities in positive samples. Simultaneously, knowledge enhancement is also applied to the original training data, further improving the representation capabilities of both the training data and positive samples, and effectively enhancing the performance of the pre-trained language model in performing corresponding domain text tasks.

[0044] Furthermore, data augmentation with locally dense connections is used to construct high-quality negative samples of knowledge triples. Comparative learning is then performed using negative and positive samples to better capture subtle differences between similar triples, thereby further compensating for the lack of global semantics brought about by the aforementioned domain knowledge graphs.

[0045] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a statistical comparison diagram between closed-domain and open-domain knowledge graphs in an embodiment of the present invention;

[0049] Figure 2 This is a flowchart of the training method for the knowledge-enhanced pre-trained language model in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the tree-like hierarchical structure of the knowledge graph in an embodiment of the present invention;

[0051] Figure 4A and 4B This is a path diagram for generating positive and negative samples for each entity node in the knowledge graph in this embodiment of the invention;

[0052] Figure 5 An architecture diagram of a knowledge-enhanced language representation learning framework applicable to various closed domains, provided in an embodiment of the present invention;

[0053] Figure 6 This is a structural block diagram of the training device for the knowledge-enhanced pre-trained language model in an embodiment of the present invention. Detailed Implementation

[0054] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0055] The inventors of this invention discovered that in certain vertical fields such as finance, healthcare, and education, the inconsistency between the distribution of basic pre-training data and domain data—where basic pre-training data is based on open-domain knowledge (such as open-source text corpora) and domain data is based on closed-domain knowledge (such as knowledge graphs in finance, healthcare, etc.)—leads to the original basic model being unable to perform effective representation learning for domain text tasks. Furthermore, domain pre-training data is extremely scarce, making it impossible to perform sufficient further pre-training to obtain a suitable domain pre-trained model.

[0056] Reference Figure 1 As shown, Figure 1 This is a statistical comparison graph between a closed-domain knowledge graph and an open-domain text corpus. The left side represents the closed-domain knowledge graph (e.g., a knowledge graph in the medical field), and the right side represents the open-domain text corpus. Figure 1 It can be seen that closed-domain knowledge graphs have lower entity coverage than open-domain text corpora, i.e., global sparsity. Furthermore, entities in closed-domain knowledge graphs are more densely connected locally (locally dense), and the semantics of these entities are very similar, making it difficult for the basic KEPLM to capture the finer-grained differences between these entity words.

[0057] To enhance KEPLM's ability to understand domain-specific terms in domain texts and fully consider the global sparsity and local density characteristics of domain knowledge graph data, this invention provides a training method for a knowledge-enhanced pre-trained language model, referring to... Figure 2 As shown, the method includes the following steps:

[0058] S21. Perform hyperbolic space learning on the domain knowledge graph to obtain the hyperbolic entity embedding representation of the domain knowledge graph;

[0059] The fields described in the embodiments of the present invention may be, for example, medical, financial, and educational fields, but the embodiments of the present invention do not limit the specific fields.

[0060] Knowledge graphs contain triple information, namely (entity, relation, entity). By learning hyperbolic space, the corresponding hyperbolic entity embedding representation data can be obtained.

[0061] S22. Embed the hyperbolic entity representation and fuse it with the pre-constructed positive sample data to obtain the fused positive sample data;

[0062] Integrating hyperbolic entity embeddings into positive sample data can enhance the entity representation capabilities of positive sample data.

[0063] S23. Inject the fused positive sample data into the existing text training data of the pre-trained language model to obtain knowledge-enhanced text training data and knowledge-enhanced positive sample data.

[0064] The text training data in this step S23 can be, for example, text token embedding data obtained by encoding text by the text encoder transformer of a pre-trained language model.

[0065] In Natural Language Processing (NLP), embedding representations are used to transform large sparse vectors into a low-dimensional space that preserves semantic relations.

[0066] After performing syntactic analysis on the training text, a token sequence is generated. Based on the token sequence, an embedding representation of the text is generated, which is the token embedding. Specific implementation methods for generating the token embedding can be found in existing technologies, and this embodiment of the invention does not limit them.

[0067] S24. Train the pre-trained language model using the knowledge-enhanced text training data and the knowledge-enhanced positive sample data.

[0068] In this step S24, a preset pre-training task is performed using the knowledge-enhanced text training data and the knowledge-enhanced positive sample data. For example, mask token prediction can be performed using the knowledge-enhanced text training data, and contrastive learning can be performed using the positive sample data (e.g., embedded representations).

[0069] The training method for the knowledge-enhanced pre-trained language model provided in this embodiment of the invention can effectively learn the hierarchical structure information of the domain knowledge graph by performing hyperbolic space learning on the domain knowledge graph, which effectively supplements the semantic loss caused by the global sparsity of the domain knowledge graph. In addition, by using the hyperbolic entity embedding representation obtained by hyperbolic space learning, the learned hyperbolic entity information and the constructed positive sample information are fused to improve the semantic modeling ability of entities in the positive samples recalled by the training data. At the same time, knowledge enhancement is also performed on the original training data, further improving the representation ability of the training data and positive samples, and effectively improving the performance of the pre-trained language model in performing corresponding domain text tasks.

[0070] The following provides a further explanation of each step of the training method for the knowledge-enhanced pre-trained language model provided in the embodiments of the present invention.

[0071] In one embodiment, in step S21 above, the Poincaré sphere model can be used to learn the tree-like hierarchical structure of the domain knowledge graph to obtain the hyperbolic entity embedding representation of the domain knowledge graph.

[0072] Reference Figure 3 The tree-like hierarchical structure of the knowledge graph shown is from... Figure 3 As can be seen, the classes and entities in the knowledge graph have a tree-like hierarchical structure. For example, under Drug, it is further divided into Chinese medicine and Western medicine. Entities under the Chinese medicine category include Sanjin tablet and Jizhi syrup, while entities under the Western medicine category include erythromycin and penicillin, etc.

[0073] As mentioned earlier, the low entity coverage of domain KP leads to a lack of sufficient external knowledge injection into KEPLM. Furthermore, obscure domain terms are difficult to cover by the original vocabulary of open-domain PLM, thus impairing the semantic understanding capabilities of closed-domain KEPLM due to insufficient vocabulary. Hyperbolic entity embedding representations can provide more comprehensive semantic knowledge.

[0074] The Poincaré sphere model transforms the tree-like structure of key-point relationships (KPs) into a spherical structure. The Poincaré sphere model effectively captures the latent hierarchical structure of domain KPs.

[0075] Hyperbolic spaces possess stronger hierarchical representation capabilities. To compensate for the global semantic deficiencies of closed domains, a Poincaré sphere model is used to simultaneously learn the structural and semantic representations of the knowledge graph based on a hierarchical entity class structure. Two entities The distance between them is:

[0076]

[0077] Where H(.) denotes the learning representation space of the hyperbolic embedding. This refers to the arcosh function.

[0078] Will Defined as the set of observable hierarchical relationships between entities. Then, the distance between related objects is minimized to obtain hyperbolic embedding:

[0079]

[0080] Ф indicates , For the negative samples of ej. The entity type Embedding can be represented as: .

[0081] To address the semantic deficiencies caused by entity overlay, this invention injects the tree structure of the domain KG, rather than the entity embeddings, separately into the KEPLM to supplement the semantic information of the target entities identified from the pre-trained corpus. This not only captures richer semantic connections between triples but also captures the implicit graph structure information of the closed domain KG, thus mitigating the sparsity of global semantics.

[0082] In one embodiment, in step S22 above, in order to integrate the hyperbolic embedding into the context representation, for example, the embeddings of entity categories can be concatenated. Injected into entity representation middle:

[0083]

[0084] The activation function is 'h', '∥' represents feature fusion (concatenation), and hpj is an entity word. Positive sample embedding;

[0085] LN stands for LayerNorm Function. , , , .

[0086] In one embodiment, in step S23 above, the heterogeneous features of the fused entity embedding are... and text token Embedding To match relevant entities in the domain KG, such as entities with an overlapped word count greater than a threshold, and utilizing an M-layer aggregator as a knowledge injector, it can integrate learning fusion results from different levels. In each aggregator, two embeddings are fed into a multi-head self-attention layer denoted as Fm:

[0087]

[0088] Where v represents the v-th layer. Entities are embedded into a context-aware representation and retrieved again from the hybrid representation:

[0089]

[0090] These are parameters that need to be learned.

[0091] It is a hybrid fusion embedding. and These are the newly generated entity and text embeddings, namely the knowledge-enhanced positive sample data and the knowledge-enhanced text training data.

[0092] In order to train the pre-trained language model, negative sample data can also be pre-constructed in this embodiment of the invention;

[0093] Accordingly, in step S24 above, the pre-trained language model is trained in the following way: using the knowledge-enhanced text training data, the preset masked language model is trained to obtain the masked language model loss.

[0094] The knowledge-enhanced positive sample data is compared with the negative sample data to obtain the contrastive learning loss;

[0095] The total loss of the pre-trained language model is obtained based on the masked language model loss and the contrastive learning loss, and the pre-trained language model is trained based on the total loss.

[0096] Furthermore, the training objective mainly consists of two parts, including the masked language model loss LMLM and the contrastive learning loss LCL, as shown in the following formula:

[0097] λ1 and λ2 are hyperparameters.

[0098]

[0099] It is the temperature hyperparameter, and cos is the cosine similarity function.

[0100] In one embodiment, positive sample data can be pre-constructed, for example, in the following manner:

[0101] Random sampling is performed on entity pairs with first-order relations in the domain knowledge graph, where each entity pair includes a head entity and a tail entity.

[0102] Based on the sampling of the head entity, tail entity, and the relationship between the head and tail entities, a triplet text is constructed.

[0103] The embedded representations of triplet texts with different entity pairs are combined to generate positive sample data.

[0104] Specifically, the K nearest neighbor triplets of the target entity e0 are extracted as positive samples, as they are the closest to the target entity in the neighboring candidate subgraph structure. The semantic information contained in these triplets helps to enhance the contextual knowledge. To better aggregate the target entity and text token embedding, the K nearest neighbor triplets can be concatenated together to form a sentence.

[0105] The tree diagram of a knowledge graph displays the hierarchical structure of entities and entity categories, as shown in the reference diagram. Figure 4A As shown, by randomly sampling entity pairs with first-order relations, such as entity pair e0 and e1, or entity pair e0 and e5, etc.

[0106] The sampled entities e0 and e1 generate the triplet text: e0,r(e0,e1)e1; the sampled entities e0 and e5 generate the triplet text: e0,r(e0,e5)e5;

[0107] The embedded representations of triplet texts with different entity pairs are combined to generate positive sample data such as: [CLS]e0,r(e0,e1)e1[SEP]e0,r(e0,e5)e5……; where [CLS] is the text start identifier and [SEP] is the delimiter to distinguish different triplets.

[0108] Meanwhile, considering the encoder's sensitivity to location information, different location indices can be assigned to triples on different paths, but the tokens within each triple have the same location index, as shown in the diagram above.<e0,r(e0,e1),e1> They have the same location information.

[0109] In one embodiment, negative sample data is constructed, for example, in the following manner:

[0110] Use entity nodes in the domain knowledge graph as the center nodes of concentric circles;

[0111] Starting from the central node, and based on a preset relationship jump distance, randomly search outwards along different relationships to obtain the ending node; construct the text of different paths from the central node to the ending node;

[0112] Negative sample data is generated based on the embedding representation of text from different paths.

[0113] In this embodiment of the invention, negative samples are constructed based on point-biconnected components. In a knowledge graph of a closed domain, due to local density, nodes are closely connected to their neighbors, which is beneficial for graph search. Therefore, a large number of nodes far from the target entity are searched as negative samples. For example, starting with node e... start Using e0 as the center point, the search extends outwards along the relationship, with different preset jump distances. The end node can be obtained. , Indicates the jump distance. Representation diagram middle and The shortest path between them. For example, refer to... Figure 4A path 3 in the text, its For path 6, .

[0114] By combining the embedding data of negative samples with the same number of hops, we can obtain negative sample data.

[0115] When different jump distances are preset, the embedded representations of texts from different paths at the same jump distance can be combined to generate negative sample data at different levels (level equal to jump distance minus 1).

[0116] In other words, different jump distances can be used to construct negative samples with varying structural difficulty. For example, assuming a jump distance of 2, Figure 4A The paths 1, 2, and 3 shown are combined into a sentence, and the resulting negative samples are as follows:

[0117] [CLS] path1[SEP]path2 [SEP]path3……;

[0118] Among them, path1 is e0→e1→e2, path2 is e0→e5→e7, and path3 is e0→e9→e 10 ;

[0119] Furthermore, assuming the jump distance is 3, refer to Figure 4B As shown, path4, path5, and path6 are combined to form a sentence, and the resulting negative samples are as follows:

[0120] [CLS] path4[SEP]path5 [SEP]path6……;

[0121] Among them, path4 is e0→e1→e2→e3, path5 is e0→e5→e7→e8, and path6 is e0→e9→e 10 →e 11 .

[0122] The closer the jump distance, the more difficult it is to distinguish the semantic knowledge contained between the start node and the end node.

[0123] In one embodiment, during the process of generating negative samples, if there are at least two paths between the center node and the end node, the path with the shortest jump distance is selected from the at least two paths to generate negative sample data.

[0124] by Figure 4B Taking the example shown, there are multiple paths (i.e., doubly connected components) between the starting node e0 and the ending node e3. The shortest path will be selected for generating negative samples, for example, path 4 will be selected instead of path 7. Figure 4B (The path within the dashed box).

[0125] Negative sample embedding can be formulated as: , , Different layers representing negative samples (the number of layers equals the jump distance minus 1).

[0126] Taking entities in a medical knowledge graph as an example, let's assume e0 represents "COVID-19 infection", e1 represents "fever", and e2 represents "skin infection";

[0127] The sampling entities generate positive samples for e0-e1, that is, there is a very close relationship between the disease "COVID-19 infection" and the symptom "fever", and the distance between the two entities is relatively close.

[0128] The sampled entity nodes e0→e1→e2 are used as negative samples. In these negative samples, although there is a very close relationship between the disease "COVID-19 infection" and the symptom "fever", and "skin infection" may also cause the symptom "fever", the correlation between "COVID-19 infection" and "skin infection" is relatively weak, and the distance between these two entities is relatively far.

[0129] This invention constructs negative sample information of different difficulty levels and compares it with positive samples for learning. This enables the model to learn the more fine-grained semantic gaps of the injected knowledge triples, allowing the underlying KELPM to better distinguish the semantics of these adjacent entities. It also utilizes the local dense features of the domain knowledge graph to further solve the global sparsity problem.

[0130] To better illustrate the training method of the knowledge-enhanced pre-trained language model described above, we will use a knowledge-enhanced language representation learning framework applicable to various closed domains as an example to explain the method.

[0131] Reference Figure 5The framework shown includes a learning hyperbolic entity embedding module in the lower left corner, which uses a Poincaré sphere model to learn the tree-like hierarchical structure of the domain knowledge graph and obtain a hyperbolic entity embedding representation of the domain knowledge graph.

[0132] Domain knowledge graphs can generate positive and negative samples through sampling, and refer to... Figure 5 The bottom right module, Point-biconnected Component-Based Pos.&Neg.TriplesConstruction, is used to construct text containing positive, first-level, and second-level negative samples. This text is then input into... Figure 5 The upper-level Domain Knowledge Encoder performs vector encoding.

[0133] Figure 5 The upper part also includes a text encoder, which is responsible for encoding and outputting the text training data of the model. This text training data can be, for example, data in the form of token embeddings, i.e., outputting positive sample embeddings (Pos.Emb) and negative sample embeddings (Neg.Emb) to the domain knowledge encoder.

[0134] The domain knowledge encoder includes the Entity Space Infusion module and the Entity Knowledge Injector module.

[0135] The entity space fusion module is used to embed hyperbolic entities into representations and fuse them with positive sample data. By fusing the learned hyperbolic entity information with the positive sample data constructed using contrastive learning, the semantic modeling capability of entities in the positive samples recalled from the training data is improved.

[0136] The entity knowledge injection module is used to inject the fused positive sample data output by the entity space fusion module into the token embedding representation. It injects the knowledge-enhanced positive sample information into the original training data, and then obtains the enhanced training data representation and the positive sample representation respectively to learn the pre-training loss task.

[0137] The Pre-training Tasks module includes the Masked Token Prediction module and the Multi-level Knowledge-aware CL.

[0138] Among them, the multi-level knowledge perception enhancer uses the embedded label data of positive and negative samples to perform comparative learning in order to learn the finer-grained semantic gaps of triples in the injected knowledge graph. It constructs positive samples and negative samples of different difficulties through the local dense structural information of the knowledge graph to further make up for the lack of global semantics brought about by the aforementioned domain knowledge graph.

[0139] The aforementioned knowledge-enhanced language representation learning framework, applicable to various closed domains, learns knowledge-aware representations through implicit knowledge graph structures. It leverages the entity richness of hyperbolic embedding aggregators to supplement the semantic information of target entities and addresses semantic deficiencies caused by global sparsity. Furthermore, it constructs high-quality negative samples of knowledge triples through locally densely connected data augmentation to better capture subtle differences between similar triples.

[0140] Empirical studies on public benchmark datasets and industry scenario tests demonstrate that the aforementioned knowledge-enhanced language representation learning framework, applicable to various closed domains, can effectively improve the performance of domain-specific text tasks.

[0141] Based on the same inventive concept, this invention also provides an application method in the field of natural language, which uses a knowledge-enhanced pre-trained language model to perform tasks in the corresponding field.

[0142] The aforementioned knowledge-enhanced pre-trained language model was obtained through the training method described above.

[0143] The knowledge-enhanced pre-trained language model training method provided in this invention can be widely applied to various application scenarios in the natural language field (such as machine translation, intelligent customer service, information and public opinion analysis, sentiment analysis, document review and comparison, document structuring, etc.) to perform corresponding text tasks, such as named entity recognition, text classification, question answering (QA), question matching (QM), natural language reasoning, and negative entity recognition, etc. This invention does not limit the specific application scenarios and tasks.

[0144] Based on the same inventive concept, this embodiment of the invention also provides a training device for a knowledge-enhanced pre-trained language model. Since the principle of the problem solved by the training device for the knowledge-enhanced pre-trained language model is similar to that of the aforementioned training method for the knowledge-enhanced pre-trained language model, the implementation of this device can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0145] This invention provides a training apparatus for a knowledge-enhanced pre-trained language model, referring to... Figure 6 As shown, it includes:

[0146] Hyperbolic space learning module 61 is used to perform hyperbolic space learning on the domain knowledge graph to obtain hyperbolic entity embedding representations of the domain knowledge graph;

[0147] The entity space fusion module 62 is used to embed the hyperbolic entity into a representation and fuse it with pre-constructed positive sample data to obtain fused positive sample data.

[0148] The entity knowledge injection module 63 is used to inject the fused positive sample data into the existing text training data of the pre-trained language model to obtain knowledge-enhanced text training data and knowledge-enhanced positive sample data.

[0149] Training module 64 is used to train a pre-trained language model using knowledge-enhanced text training data and the knowledge-enhanced positive sample data.

[0150] This invention also provides a computing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned training method for a knowledge-enhanced pre-trained language model.

[0151] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned training method for a knowledge-enhanced pre-trained language model.

[0152] This invention also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it is used for training a knowledge-enhanced pre-trained language model as described above.

[0153] The training method, application method, and apparatus for knowledge-enhanced pre-trained language models provided in this invention address the global sparsity and local density characteristics of closed-domain knowledge graphs compared to open-domain data. By learning the structural hierarchy of the domain knowledge graph and fusing the structural learning results with positive samples from the pre-trained model, it effectively supplements the semantic gaps caused by the global sparsity of the domain knowledge graph, improving the semantic modeling ability of entities in positive samples. Simultaneously, it also enhances the original training data, further improving the representational capabilities of both the training data and positive samples, effectively improving the performance of the pre-trained language model in performing corresponding domain text tasks. Furthermore, it utilizes locally densely connected data augmentation to construct high-quality negative samples of knowledge triples. Comparative learning between negative and positive samples better captures subtle differences between similar triples, further compensating for the global semantic gaps caused by the aforementioned domain knowledge graph.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A training method for a knowledge-enhanced pre-trained language model, characterized in that, include: Hyperbolic space learning is performed on the domain knowledge graph to obtain the hyperbolic entity embedding representation of the domain knowledge graph; The hyperbolic entity embedding representation is fused with pre-constructed positive sample data to obtain fused positive sample data; The fused positive sample data is injected into the existing text training data of the pre-trained language model. The fused positive sample data and the text training data are interacted and fused through a knowledge injector to obtain knowledge-enhanced text training data and knowledge-enhanced positive sample data, respectively. The pre-trained language model is trained using knowledge-enhanced text training data and the knowledge-enhanced positive sample data.

2. The method as described in claim 1, characterized in that, Hyperbolic space learning is performed on the domain knowledge graph to obtain hyperbolic entity embedding representations, including: By using the Poincaré sphere model, the tree-like hierarchical structure of the domain knowledge graph is learned, and the hyperbolic entity embedding representation of the domain knowledge graph is obtained.

3. The method as described in claim 1, characterized in that, Also includes: Pre-construct negative sample data; Accordingly, training the pre-trained language model using the knowledge-enhanced text training data and the knowledge-enhanced positive sample data includes: Using knowledge-enhanced text training data, a pre-defined masked language model is trained to obtain the masked language model loss. The positive sample data enhanced by the knowledge are compared with the negative sample data to obtain the contrastive learning loss; The total loss of the pre-trained language model is obtained based on the masked language model loss and the contrastive learning loss, and the pre-trained language model is trained based on the total loss.

4. The method as described in claim 3, characterized in that, The total loss of the pre-trained language model is obtained based on the masked language model loss and the contrastive learning loss, including: The total loss of the pre-trained language model is obtained by calculating the weighted sum of the masked language model loss and the contrastive learning loss.

5. The method as described in claim 3, characterized in that, The positive sample data is pre-constructed in the following manner, including: Random sampling is performed on entity pairs with first-order relations in the domain knowledge graph, wherein the entity pairs include a head entity and a tail entity; Based on the sampled head entity, tail entity, and the relationship between the head and tail entities, a triplet text is constructed. The positive sample data is generated by combining the embedded representations of the triplet texts of different entity pairs.

6. The method as described in claim 3, characterized in that, The pre-constructed negative sample data includes: Use entity nodes in the domain knowledge graph as the center nodes of concentric circles; Starting from the central node, and according to a preset relationship jump distance, randomly search outwards along different relationships to obtain the ending node; construct text for different paths from the central node to the ending node; Negative sample data is generated based on the embedding representation of the text from the different paths.

7. The method as described in claim 6, characterized in that, The preset jump distance includes multiple different jump distances; Accordingly, negative sample data is generated based on the embedded representations of the text from the different paths, including: For different jump distances, the embedded representations of texts from different paths at the same jump distance are combined to generate negative sample data at different levels.

8. The method as described in claim 6, characterized in that, Also includes: If there are at least two paths between the center node and the end node, the path with the shortest jump distance is selected from the at least two paths to generate the negative sample data.

9. An application method in the field of natural language processing, characterized in that, The application method uses a knowledge-enhanced pre-trained language model to perform tasks in the corresponding domain. The knowledge-enhanced pre-trained language model is obtained through the training method of the knowledge-enhanced pre-trained language model as described in any one of claims 1-8.

10. A computing device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method for a knowledge-enhanced pre-trained language model as described in any one of claims 1-8 or the application method in the field of natural language as described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the training method for the knowledge-enhanced pre-trained language model as described in any one of claims 1-8 or the application method in the field of natural language as described in claim 9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is the training method for a knowledge-enhanced pre-trained language model as described in any one of claims 1-8 or the application method in the field of natural language as described in claim 9.

Images