A pre-training language model entity knowledge injection method, system and device
By calculating string similarity and using contrastive learning, the problem of insufficient entity semantic learning in the equipment field by pre-trained language models is solved. Through the proposed patented application, the application of pre-trained language models in the equipment field is realized, solving the technical problems existing in the prior art, realizing the patented application, and realizing the effective application of pre-trained language models in the equipment field, improving the efficiency of entity semantic learning and the robustness of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE QUARTERMASTER RES INST OF THE GENERAL LOGISTICS DEPT OF THE CPLA
- Filing Date
- 2022-09-06
- Publication Date
- 2026-06-09
Smart Images
Figure CN115423098B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and apparatus for injecting entity knowledge into a pre-trained language model. Background Technology
[0002] Currently, with the continuous application and promotion of artificial intelligence technology in various fields of society, pre-trained language models, as its cutting-edge technology, have become a mainstream research direction in industry and academia. Pre-trained language models learn the representation vectors of relevant words in context from large-scale text corpora in a self-supervised manner. These representation vectors contain general grammatical and semantic knowledge, and the fine-tuned models can be applied to downstream tasks such as semantic understanding and text generation.
[0003] While deep learning-based pre-trained language models possess powerful representational capabilities and can transfer grammatical and semantic knowledge, they rely heavily on large amounts of text training data. The long-tail distribution of this training data leads to sparse domain entities, making it difficult for pre-trained language models to fully learn the semantics of domain entities. Furthermore, in the equipment domain, terminology is often highly specialized, with multiple expressions often associated with a single domain term. Even the same entity may express completely different semantics in equipment-specific contexts compared to other general scenarios. These factors contribute to the current situation where pre-trained language models are still unable to be applied in the equipment domain.
[0004] Unlike deep learning-based pre-trained language models, symbolic knowledge possesses characteristics such as clear semantics, ease of organization, strong interpretability, intuitiveness, and ease of human understanding. Therefore, the introduction of symbolic knowledge can provide pre-trained language models with richer and more comprehensive information, overcoming the limitations of long-tailed training data in learning the semantics of domain entities and the resulting poor model robustness. However, symbolic knowledge and representation vectors have different spatial structures, making the introduction of symbolic knowledge into pre-trained language models a significant challenge. Summary of the Invention
[0005] The purpose of this invention is to provide a method, system, and apparatus for injecting entity knowledge into a pre-trained language model, so as to solve at least one of the above-mentioned technical problems existing in the prior art.
[0006] To address the aforementioned technical problems, the present invention provides a pre-trained language model entity knowledge injection method, comprising: obtaining entity names from text corpora by calculating string similarity; constructing an entity vector table by encoding entity semantics through a pre-trained language model; constructing training samples for entity injection; and injecting entity knowledge into the pre-trained language model through contrastive learning.
[0007] Furthermore, the entity name includes domain terms and their aliases, and consists of Chinese, English and / or punctuation marks.
[0008] Furthermore, the calculation of string similarity includes calculating the similarity between the domain term and the referent in the text corpus using a similarity algorithm. When the similarity is greater than a preset threshold, the referent is used as an alias of the current domain term.
[0009] Preferably, the similarity algorithm is the Levenstein algorithm, but other algorithms known in the art can also be used to achieve the same technical purpose.
[0010] Preferably, the preset threshold is 80%.
[0011] The algorithm described above facilitates the identification of aliases for existing entities, which helps to improve the problem of entity sparsity and enhances the learning efficiency of pre-trained language models for entity semantics.
[0012] Furthermore, the pre-trained language model is an autoregressive pre-trained language model.
[0013] Preferably, the pre-trained language model is a model known in the art, such as CPM, Pangu, or GPT.
[0014] The above model structure is used to encode entity semantics, realize symbolic knowledge vectorization, generate a representation vector for each entity, and thus construct an entity vector table, which facilitates the injection of symbolic knowledge into the pre-trained language model in subsequent operations.
[0015] Furthermore, the pre-trained language model uses an encoder as its model structure, which includes a self-attention layer and a fully connected layer. The self-attention layer achieves unidirectional attention to the text corpus through a self-attention mechanism.
[0016] Furthermore, the unidirectional attention refers to the fact that by calculating the similarity between the following text representation vector and the preceding text representation vector, the attention weight of the following text to the preceding text can be obtained, but the attention weight of the preceding text to the following text cannot be obtained.
[0017] By focusing on one direction as described above, the pre-trained language model is prevented from knowing the following text in advance, thus prompting the pre-trained language model to generate the most appropriate following text representation vector in complete accordance with the preceding text.
[0018] Furthermore, the pre-trained language model also includes a word segmenter for segmenting entity corpora and concatenating them into an input sequence.
[0019] Preferably, the length of the input sequence is 1024, but it can be set to other length values as needed.
[0020] Furthermore, when the length of the input sequence is insufficient, it is supplemented by padding values.
[0021] Preferably, the padding value is the English word "padding".
[0022] Furthermore, the pre-trained language model generates a representation vector for each entity based on the semantic encoding of the entity's descriptive text.
[0023] Furthermore, the training samples include triples, which include a text reference vector, a positive entity representation vector, and a negative entity representation vector.
[0024] Furthermore, the vector has a dimension of 2560, but it can be set to other dimension values as needed.
[0025] Furthermore, the steps for constructing training samples for entity injection include: performing entity alignment based on the text corpus to obtain the sub-word representation vectors corresponding to the text references that need to be injected with knowledge; performing weighted summation on several of the sub-word representation vectors to obtain the text reference representation vectors; selecting string-aligned entity representation vectors from the entity vector table as positive examples and entity representation vectors unrelated to the text references as negative examples to construct triplet training data.
[0026] Furthermore, the contrastive learning includes a loss function, which is used to narrow the distance between text references and entities expressing the same meaning in the vector space, and to widen the distance between text references and entities expressing different meanings in the vector space.
[0027] On the other hand, the present invention also provides a pre-trained language model entity knowledge injection device for the above-mentioned pre-trained language model entity knowledge injection method, characterized in that the device includes a processor, a memory, and a bus, wherein the memory stores instructions and data that can be read by the processor; the processor is used to call the instructions and data in the memory; and the bus connects the functional components to transmit information.
[0028] In another aspect, the present invention also provides a pre-trained language model entity knowledge injection system, characterized in that it includes a data receiving module, a data processing module, and a model output module:
[0029] The data receiving module is used to receive text corpus including entity semantics;
[0030] The data processing module includes: an entity name unit, a pre-trained language model unit, a training sample unit, and a contrastive learning unit.
[0031] The entity name unit obtains the entity name by calculating the similarity of strings in the text corpus;
[0032] The pre-trained language model unit is used to store the initial pre-trained language model, encode entity semantics based on entity description text, and generate an entity vector table.
[0033] The training sample unit is used to construct training samples for entity injection;
[0034] The contrastive learning unit injects entity knowledge into the pre-trained language model through contrastive learning.
[0035] The model output module is used to output a pre-trained language model after entity knowledge has been injected.
[0036] By adopting the above technical solution, the present invention has the following beneficial effects:
[0037] This invention provides a pre-trained language model entity knowledge injection method. Addressing the issue of entities in fields such as equipment having multiple aliases, this method identifies entity aliases using string similarity calculation. To address the problem of symbolic knowledge and model knowledge having different structures, this method encodes entity semantics for entities in fields such as equipment based on an autoregressive pre-trained language model. A unified encoder is used to encode the training text corpus and the context of entity semantics, achieving symbolic knowledge vectorization. Furthermore, to address the issue that text denotation vectors lack scene semantics in fields such as equipment, this method injects entity knowledge in fields such as equipment through contrastive learning, thereby promoting the application of pre-trained language models in scenarios related to equipment. Attached Figure Description
[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0039] Figure 1 A flowchart of the pre-trained language model entity knowledge injection method provided in this embodiment of the invention;
[0040] Figure 2 A schematic diagram illustrating the process of encoding equipment domain entity semantics using a pre-trained language model, as provided in an embodiment of the present invention.
[0041] Figure 3 A schematic diagram illustrating the process of injecting training samples with entity knowledge in the equipment field, as provided in this embodiment of the invention;
[0042] Figure 4A schematic diagram illustrating the principle of injecting entity knowledge in the equipment field based on contrastive learning technology, as provided in an embodiment of the present invention.
[0043] Figure 5 A diagram of the pre-trained language model entity knowledge injection system provided in the embodiments of the present invention. Detailed Implementation
[0044] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] The terms "first," "second," and "third," etc., in the specification, embodiments, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or modules. A method, system, product, or apparatus is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or apparatuses. "And / or" is used to indicate the selection of one or both of the two objects to which it is connected.
[0046] The present invention will be further explained below with reference to specific embodiments.
[0047] Before introducing the specific implementation methods, the inventive concept will be explained as follows to facilitate understanding of this embodiment:
[0048] Conventional pre-trained language models suffer from domain entity blind spots due to the long-tail distribution of text training data. Furthermore, the entity names in the equipment domain are domain-specific and have various variant aliases, exacerbating entity sparsity and hindering the effective learning of entity semantics by pre-trained language models. Moreover, the knowledge learned by pre-trained language models is in vector parameter form, while symbolic knowledge is in string form; these two have different structures and cannot be used uniformly. Therefore, this invention uses an autoregressive pre-trained language model to encode entity semantics in the domain, vectorizing symbolic knowledge. It then constructs an entity vector table for the equipment domain by calculating string similarity using the Levenstein algorithm, and subsequently injects the semantics of equipment entities into the pre-trained language model.
[0049] Because the understanding of entity semantics by a pre-trained language model depends on the context and frequency of the entity, and the same entity word often has different meanings in different scenarios, the lack of domain-specific context leads to incorrect understandings of entities in the equipment domain, resulting in poor model robustness. To address this, this invention constructs training data in the form of triplets (textual denotation vector, positive entity representation vector, and negative entity representation vector). Through contrastive learning, word vectors in the pre-trained language model's vocabulary are aligned with entity-aligned positive entity representation vectors in the vector space, and moved away from negative entity representation vectors expressing different meanings. This allows the pre-trained language model to maintain the original semantics of entities while adding entity semantics specific to the equipment domain.
[0050] like Figure 1 As shown in this embodiment, the pre-trained language model entity knowledge injection method includes the following steps:
[0051] Step 1: Calculate string similarity based on the Levenstein algorithm and filter out aliases of equipment domain entities from the text corpus;
[0052] Step 2: Encode entity semantics in the equipment domain based on an autoregressive pre-trained language model and construct an entity vector table;
[0053] Step 3: Construct training samples for injecting entities in the equipment domain;
[0054] Step 4: Inject equipment domain entity knowledge into the autoregressive language model through comparative learning.
[0055] The Levenstein algorithm for calculating string similarity includes: initializing matrix D; calculating the edit distance between domain term and text reference; and calculating the string similarity between domain term and text reference.
[0056] The above algorithm can be represented in pseudocode as follows:
[0057] input:termmention
[0058] 1) Initialize matrix D;
[0059] 2) Edit distance between computing domain terms and textual references;
[0060]
[0061] 3) String similarity between computational domain terms (term) and textual references (mention);
[0062] output:mention
[0063] Wherein, term represents a term in the equipment domain, and mention represents a reference item in the text corpus. When the string similarity between the reference item and the term is higher than 80%, the reference item is selected as an alias for the equipment domain entity.
[0064] In step 2 of the pre-trained language model entity knowledge injection method, the detailed process of the autoregressive pre-trained language model encoding entity semantics is as follows: Figure 2 As shown, it includes the following steps:
[0065] Step 21: Based on the entity text corpus, segment the entity text corpus using the word segmenter of the autoregressive pre-trained language model, and concatenate them into an input sequence of length 1024. When the length is less than 1024, use the English word "padding" to represent the padding value, such as... Figure 2 As shown in the lower left corner;
[0066] Step 22: Based on the input sequence, project it into a vector matrix of dimension 1024*2560 using an autoregressive pre-trained language model, and encode it using a multi-layer Transformer encoder, as follows. Figure 2 As shown on the left side of the middle;
[0067] Step 23: The Transformer encoder achieves unidirectional attention to the text through a self-attention mechanism with a mask. The processing is as follows: Figure 2 As shown on the right, the similarity between the following text and the preceding text representation vector can be calculated to obtain the attention weight for the preceding text, while the similarity between the preceding text and the following text representation vector cannot be calculated, so the attention weight for the following text cannot be obtained. This avoids the autoregressive pre-trained model knowing the following text in advance, thus prompting the autoregressive pre-trained language model to completely follow the preceding text to generate the most appropriate following text representation vector.
[0068] Step 24: The autoregressive pre-trained language model generates word representation vectors that correspond one-to-one with the words in the input sequence and have a dimension of 2560. Due to the self-attention mechanism of the Transformer encoder, the word representation vectors at the end of the sentence encode the semantics of the entire text. Therefore, the word representation vectors at the end of the sentence are used as the representation vectors of entities and aliases, thereby constructing an entity vector table.
[0069] Step 3 of the pre-trained language model entity knowledge injection method is as follows: Figure 3 As shown, the detailed process is as follows:
[0070] Step 31: Based on the text corpus, obtain the sub-word representation vectors corresponding to the text references that need to be injected with knowledge through entity alignment;
[0071] Step 32: Based on the aforementioned sub-word representation vectors, obtain the text reference item representation vector through weighted summation;
[0072] Step 33: Construct positive entity representation vectors (positive samples) and negative entity representation vectors (negative samples) based on text references, and construct triplet training data. The positive entity representation vector refers to the entity representation vector extracted from the entity vector table and aligned with the text references by strings; the negative entity representation vector refers to the entity representation vector extracted from the entity vector table and unrelated to the text references.
[0073] Step 4 of the pre-trained language model entity knowledge injection method includes the following specific processes: based on the language model function, adding triples to form a loss function, and achieving contrastive learning by minimizing the loss function. This is used to narrow the distance between text references and entities expressing the same meaning in the vector space, and to widen the distance between text references and entities expressing different meanings in the vector space. Figure 4 As shown; during training, positive example entity representation vectors (positive samples) and negative example entity representation vectors (negative samples) are used to provide a reference for the autoregressive pre-trained language model to learn the semantics of domain entities. The vector parameters of the positive example entity representation vectors and negative example entity representation vectors remain fixed. By fine-tuning the parameters of the text references, the goal is to inject equipment domain entity knowledge into the autoregressive pre-trained language model while maintaining the original semantics of the text references. The loss function is as follows:
[0074]
[0075] Among them, h mention Let e be a vector representing the text reference. pos Let e be the vector representing the positive instance entity. neg Let W be the vector representing the negative instance entities, and let W be the set of entities.
[0076] On the other hand, the present invention also provides a pre-trained language model entity knowledge injection device for the above-mentioned pre-trained language model entity knowledge injection method, characterized in that the device includes a processor, a memory, and a bus:
[0077] The memory stores instructions and data that can be read by the processor;
[0078] The processor is used to call instructions and data in the memory to execute the pre-trained language model entity knowledge injection method, the specific steps of which include:
[0079] Step 1: Calculate string similarity based on the Levenstein algorithm and filter out aliases of equipment domain entities from the text corpus;
[0080] Step 2: Encode entity semantics in the equipment domain based on an autoregressive pre-trained language model and construct an entity vector table;
[0081] Step 3: Construct training samples for injecting entities in the equipment domain;
[0082] Step 4: Inject equipment domain entity knowledge into the autoregressive language model through comparative learning;
[0083] The bus connects the various functional components to transmit information.
[0084] Furthermore, this invention also provides a pre-trained language model entity knowledge injection system, such as... Figure 5 As shown, it includes a data receiving module, a data processing module, and a model output module:
[0085] The data receiving module is used to receive text corpus including entity semantics;
[0086] The data processing module includes: an entity name unit, a pre-trained language model unit, a training sample unit, and a contrastive learning unit.
[0087] The entity name unit obtains the entity name by calculating the similarity of strings in the text corpus;
[0088] The pre-trained language model unit is used to store the initial pre-trained language model, encode entity semantics based on entity description text, and generate an entity vector table.
[0089] The training sample unit is used to construct training samples for entity injection;
[0090] The contrastive learning unit injects entity knowledge into the pre-trained language model through contrastive learning.
[0091] The model output module is used to output a pre-trained language model after entity knowledge has been injected.
[0092] In another implementation, this solution can be implemented using a device, which may include corresponding modules that perform one or more steps in the various embodiments described above. A module may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.
[0093] The processor executes the various methods and processes described above. For example, the method implementations in this scheme can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some implementations, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other implementations, the processor can be configured to execute one of the methods described above by any other suitable means (e.g., by means of firmware).
[0094] This device can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus connects various circuits, including one or more processors, memory, and / or hardware modules. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuitry, external antennas, etc.
[0095] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be divided into address buses, data buses, control buses, etc.
[0096] This invention addresses the issues of domain-specific entity names and multiple aliases in the equipment domain by using the Levenstein algorithm to calculate string similarity and filter entity aliases from text corpora. This constructs an equipment domain entity vector table, which is then used to inject the semantics of equipment entities into a pre-trained language model in subsequent processes. Furthermore, this invention addresses the problem of symbolic knowledge and model knowledge having different structures by using an autoregressive pre-trained language model to encode entity semantics for equipment domain entities. A unified encoder is used to encode the context of the training text corpus and entity semantics, achieving both symbolic knowledge vectorization and avoiding the generation of heterogeneous vectors. Finally, this invention addresses the issue of text denotation vectors lacking equipment domain scene semantics by using contrastive learning to inject equipment domain entity knowledge into the autoregressive pre-trained language model.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for injecting entity knowledge into a pre-trained language model, characterized in that, include: The entity name is obtained by calculating the similarity of the strings; Entity vector tables are constructed by encoding entity semantics through pre-trained language models; Construct training samples for entity injection; Entity knowledge is injected into a pre-trained language model through comparative learning. Specific methods for encoding entity semantics include: Step 21: Based on the entity text corpus, the entity text corpus is segmented according to the word segmenter of the autoregressive pre-trained language model, and concatenated into an input sequence with a length of 1024. When the length is less than 1024, the English word "padding" is used to supplement it. Step 22: Based on the input sequence, project it into a 1024-dimensional language model using an autoregressive pre-trained model. A vector matrix of 2560, encoded using a multi-layer Transformer encoder; Step 23: The Transformer encoder achieves one-way attention of text through a self-attention mechanism with a mask. That is, the text below can calculate the similarity with the vector of the text above to obtain the attention weight of the text above, while the text above cannot calculate the similarity with the vector of the text below, so it cannot obtain the attention weight of the text below. Step 24: The autoregressive pre-trained language model generates word representation vectors that correspond one-to-one with the words in the input sequence and have a dimension of 2560. Due to the self-attention mechanism of the Transformer encoder, the word representation vectors at the end of the sentence encode the semantics of the entire text. Therefore, the word representation vectors at the end of the sentence are used as the representation vectors of entities and aliases, thereby constructing an entity vector table.
2. The pre-trained language model entity knowledge injection method according to claim 1, characterized in that, The entity name includes domain terms and their aliases, and consists of Chinese, English and / or punctuation marks.
3. The pre-trained language model entity knowledge injection method according to claim 2, characterized in that, The calculation of string similarity includes calculating the similarity between the domain term and the reference item in the text corpus using a similarity algorithm. When the similarity is greater than a preset threshold, the reference item is used as an alias of the current domain term.
4. The pre-trained language model entity knowledge injection method according to claim 1, characterized in that, The pre-trained language model uses an encoder as its model structure. The encoder includes a self-attention layer and a fully connected layer. The self-attention layer achieves unidirectional attention to the text corpus through a self-attention mechanism.
5. The pre-trained language model entity knowledge injection method according to claim 1, characterized in that, The training samples include triples, which consist of a text reference vector, a positive entity representation vector, and a negative entity representation vector.
6. The pre-trained language model entity knowledge injection method according to claim 5, characterized in that, The text reference item representation vector is a representation vector obtained by weighted summation of several sub-word representation vectors corresponding to the text reference item; the positive example entity representation vector is an entity representation vector extracted from the entity vector table and aligned with the text reference item; the negative example entity representation vector is an entity representation vector extracted from the entity vector table that is unrelated to the text reference item.
7. A pre-trained language model entity knowledge injection device for the pre-trained language model entity knowledge injection method according to any one of claims 1 to 6, characterized in that, The device includes a processor, a memory, and a bus. The memory stores instructions and data that can be read by the processor; the processor is used to call the instructions and data in the memory; and the bus connects the various functional components to transmit information.
8. A pre-trained language model entity knowledge injection system, characterized in that, It includes a data receiving module, a data processing module, and a model output module: The data receiving module is used to receive text corpus including entity semantics; The data processing module includes: an entity name unit, a pre-trained language model unit, a training sample unit, and a contrastive learning unit. The entity name unit obtains the entity name by calculating the similarity of strings in the text corpus; The pre-trained language model unit is used to store the initial pre-trained language model, encode entity semantics based on entity description text, and generate an entity vector table; specifically, it includes: Step 21: Based on the entity text corpus, the entity text corpus is segmented according to the word segmenter of the autoregressive pre-trained language model, and concatenated into an input sequence with a length of 1024. When the length is less than 1024, the English word "padding" is used to supplement it. Step 22: Based on the input sequence, project it into a 1024-dimensional language model using an autoregressive pre-trained model. A vector matrix of 2560, encoded using a multi-layer Transformer encoder; Step 23: The Transformer encoder achieves one-way attention of text through a self-attention mechanism with a mask. That is, the text below can calculate the similarity with the vector of the text above to obtain the attention weight of the text above, while the text above cannot calculate the similarity with the vector of the text below, so it cannot obtain the attention weight of the text below. Step 24: The autoregressive pre-trained language model generates word representation vectors that correspond one-to-one with the words in the input sequence and have a dimension of 2560. Due to the self-attention mechanism of the Transformer encoder, the word representation vector at the end of the sentence encodes the semantics of the entire text. Therefore, the word representation vector at the end of the sentence is used as the representation vector of entities and aliases to construct an entity vector table. The training sample unit is used to construct training samples for entity injection; The contrastive learning unit injects entity knowledge into the pre-trained language model through contrastive learning. The model output module is used to output a pre-trained language model after entity knowledge has been injected.