Knowledge extraction method and system based on prompt learning

Abstract

The application provides a knowledge extraction method and system based on prompt learning, embedding obtained unstructured text data to obtain a cache vector value; embedding a homogeneous prompt string to obtain a homogeneous prompt symbol vector value, and embedding a heterogeneous prompt character string to obtain a heterogeneous prompt character vector value; splicing the cache vector value, the homogeneous prompt symbol vector value and the heterogeneous prompt character vector value to obtain a spliced vector, taking the spliced vector as a cache vector value of a pre-training language model; adopting a regular matching method to obtain structured data from text data generated by the pre-training language model; the application uses automatically encoded prompt characters, automatically learns a potential semantic representation of a label, and solves a knowledge extraction problem in a general way through a generative large-scale pre-training language model, thereby improving the precision and efficiency of knowledge extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a knowledge extraction method and system based on prompt learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Structured data refers to data in a standardized format with a well-defined structure. In contrast, unstructured data refers to data with irregular structure. In the field of natural language processing, unstructured data generally refers to text information described in natural language, which is easier to understand but difficult to retrieve; structured data is the extraction of entities and their attributes from text and representation with specific data structures. Knowledge extraction converts unstructured text into structured data according to certain rules, and can establish relationships between structured data according to text information.

[0004] The named entity recognition task solves the problem of identifying entities (objective existence of objects or unified name of concepts) and attribute fields in text, the relationship classification task solves the problem of judging the relationship between two entities, and the relationship extraction task identifies entities in text and judges the relationship type between entities. The above tasks have one characteristic, that is, according to a specific class label, a specific text segment is selected from unstructured text, and finally a structured knowledge representation is generated. The above tasks are collectively referred to as knowledge extraction tasks.

[0005] The generative large-scale pre-training language model is a neural network model based on the Transformer architecture, where generative refers to a left-to-right text generation method similar to writing text; large-scale pre-training refers to a model training method that uses a large amount of text for unsupervised learning; language model is a conditional probability model based on neural network, which can calculate the probability of the following text according to the preceding text and take the text with the maximum probability.

[0006] At present, the related method for knowledge extraction task still stays in conditional random field, BERT and other sequence labeling methods. These methods have two major problems:

[0007] (1) Lack of sufficient prior knowledge, although there are methods to apply pre-training language models to knowledge extraction tasks, but due to the insufficient size of the model, the learning cost of the model rises and the effect shrinks;

[0008] (2) Lack of a unified solution to the knowledge extraction task. Traditional methods often only target a specific task and do not design a general model to solve the knowledge extraction task. SUMMARY

[0009] In order to solve the problems of the prior art, the present application provides a knowledge extraction method and system based on prompt learning, which uses automatically encoded prompt characters to automatically learn the latent semantic representation of labels, and solves the knowledge extraction problem in a general way through a generative large-scale pre-training language model, thereby improving the accuracy and efficiency of knowledge extraction.

[0010] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0011] In a first aspect, the present application provides a knowledge extraction method based on prompt learning.

[0012] A knowledge extraction method based on prompt learning includes the following processes:

[0013] Embedding the obtained unstructured text data to obtain a cache vector value;

[0014] Embedding the homogeneous prompt string to obtain a homogeneous prompt symbol vector value, and embedding the heterogeneous prompt character string to obtain a heterogeneous prompt character vector value;

[0015] Concatenating the cache vector value, the homogeneous prompt symbol vector value and the heterogeneous prompt character vector value to obtain a concatenated vector, and using the concatenated vector as the cache vector value of the large-scale pre-training language model;

[0016] Using a regular matching method to obtain structured data from the text data generated by the large-scale pre-training language model.

[0017] As a further limitation of the first aspect of the present application, the obtained unstructured text data is embedded to obtain a cache vector value, including:

[0018] Using a large-scale generative pre-training language model to embed the original input text to obtain a cache vector value.

[0019] As a further limitation of the first aspect of the present application, the homogeneous prompt string is embedded to obtain a homogeneous prompt symbol vector value, and the heterogeneous prompt character is embedded to obtain a heterogeneous prompt character vector value, and the embedding includes:

[0020] Direct vector embedding, using Embedding operation to map characters to a learnable vector; or using LSTM or RNN for embedding; or using Transformer architecture for embedding.

[0021] As a further limitation of the first aspect of the present application, the training of the large-scale pre-training language model includes:

[0022] The supervised data set is divided according to the relationship category or the annotation type, and a divided supervised data set is obtained;

[0023] According to the divided supervised data set and the prompt template, a generated supervised data set with the prompt template is obtained, and according to the generated supervised data set with the prompt template, a spliced vector is used as a cache vector value of the large-scale pre-training language model to train the large-scale pre-training language model.

[0024] As a further limitation of the first aspect of the application, in the training of the large-scale pre-training language model, the parameters of the entire model are updated; or the parameters of the embedding process of the homogeneous prompt string and the heterogeneous prompt string are updated.

[0025] As a further limitation of the first aspect of the application, the cache vector value, the homogeneous prompt symbol vector value and the heterogeneous prompt character vector value are spliced to obtain a spliced vector, comprising:

[0026] The order of the spliced vector is the homogeneous prompt symbol vector value, the cache vector value and the heterogeneous prompt character vector value; or the order of the spliced vector is the heterogeneous prompt character vector value, the homogeneous prompt symbol vector value and the cache vector value; or the order of the spliced vector is the homogeneous prompt symbol vector value, the heterogeneous prompt character vector value and the cache vector value.

[0027] Secondly, the application provides a knowledge extraction system based on prompt learning.

[0028] A knowledge extraction system based on prompt learning, comprising:

[0029] An unstructured text data embedding module configured to embed the obtained unstructured text data to obtain a cache vector value;

[0030] A prompt symbol embedding module configured to embed the homogeneous prompt string to obtain a homogeneous prompt symbol vector value, and to embed the heterogeneous prompt string to obtain a heterogeneous prompt character vector value;

[0031] A vector splicing module configured to splice the cache vector value, the homogeneous prompt symbol vector value and the heterogeneous prompt character vector value to obtain a spliced vector, and to use the spliced vector as a cache vector value of a large-scale pre-training language model;

[0032] A structured data generation module configured to use a regular matching method to obtain structured data from text data generated by the large-scale pre-training language model.

[0033] As a further limitation of the second aspect of the application, the training of the large-scale pre-training language model comprises:

[0034] The supervised data set is divided according to the relationship category or the annotation type, and a divided supervised data set is obtained;

[0035] According to the divided supervised data set and the prompt template, a generative supervised data set with the prompt template is obtained, and according to the generative supervised data set with the prompt template, the training of the large-scale pre-training language model is performed by taking the splicing vector as a cache vector value of the large-scale pre-training language model.

[0036] In a third aspect, a computer readable storage medium is provided, which stores a program, and the program is executed by a processor to implement the steps of the knowledge extraction method based on prompt learning according to the first aspect of the present application.

[0037] In a fourth aspect, an electronic device is provided, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements the steps of the knowledge extraction method based on prompt learning according to the first aspect of the present application when executing the program.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] 1. The present application innovatively proposes a knowledge extraction method and system based on prompt learning, which uses automatically encoded prompt characters to automatically learn the latent semantic representation of labels, and solves the knowledge extraction problem in a general way through a generative large-scale pre-training language model, thereby improving the accuracy and efficiency of knowledge extraction.

[0040] 2. The present application innovatively proposes a knowledge extraction method and system based on prompt learning, which can simultaneously solve the relationship extraction and named entity recognition tasks, and the proposed prompt symbol with label attributes can learn better label representation during the training process, thereby improving the accuracy of knowledge extraction.

[0041] The advantages of the additional aspects of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0043] Figure 1 The training phase flowchart of the large-scale pre-training language model provided for the first embodiment of the present application;

[0044] Figure 2 The inference phase flowchart provided for the first embodiment of the present application;

[0045] Figure 3 This is a schematic diagram of the model architecture provided in Embodiment 1 of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0049] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0050] Example 1:

[0051] Embodiment 1 of this invention provides a knowledge extraction method based on cue learning. Cue learning leverages the capabilities of large-scale pre-trained language models to design text that assists these models in generating text that meets specific requirements. The method includes the following steps:

[0052] Step 1: Design the knowledge architecture related to knowledge extraction, that is, design entity types, attribute types and relationship types.

[0053] Step 2: Construct a supervised dataset based on the knowledge architecture and text data, and divide the supervised dataset into training set, validation set and test set according to the ratio of 80%:10%:10%.

[0054] Step 3: Data classification and segmentation.

[0055] For the relation extraction task, all data is split according to different relation labels, and all entity pairs under each data point belong to the same relation category.

[0056] For sequence labeling tasks, the data is split according to different labeling categories. Specifically, let the dataset D = {(x1,y1),(x2,y2),...,(x...} n ,y n )}, where the relation category or label type R={r1,r2,...,rm}, then D will be divided into Where r i ∈R.

[0057] Step 4: Prompt Engineering. Design prompt templates for knowledge extraction tasks:

[0058] x n Entity Pairs:

[0059] Pair1:(Subject:____,Object1:____,Object2:____,…,ObjectN:____)

[0060] Pair2:(Subject:____,Object1:____,Object2:____,…,ObjectN:____)

[0061] …

[0062] PairN:(Subject:____,Object1:____,Object2:____,…,ObjectN:____)

[0063] The prompts are as follows: "Entity Pairs:" indicates the start of the relationship extraction task; "Pair1" to "PairN" indicate the extraction of the head entity and all tail entities pointed to by the head entity; "Subject" indicates the extraction of the head entity; "Object1" to "ObjectN" indicate the extraction of all tail entities pointed to by the head entity, with no prompt if there are no tail entities; the "PairN" prompt can be expanded based on the number of entity pairs, and "ObjectN" can be expanded based on the number of tail entities; x n Entity Pairs: "for x" n Data format after adding the prompt template x n ';Will Fill in the blanks corresponding to the prompt model to form the training text with the prompt template.

[0064] Finally, construct according to the prompt template. Generative supervised datasets Used for training generative knowledge extraction models.

[0065] Step 5: Set up homogeneous and heterogeneous prompts. This embodiment provides two prompting methods: homogeneous prompts, which means setting the same prompt characters for each category label; and heterogeneous prompts, which means setting different prompt characters for each category label.

[0066] First, set the homogeneous prompt string Where l s is the length of the homogeneous prompt string; second, set the heterogeneous prompt string for each label category Where, l e is the length of the heterogeneous prompt string,

[0067] Step 6: Use the neural network embedding method to embed the T and P sets obtained in step 5 to obtain the vector representation of the character sets T and P And The size of the embedding should conform to the size specified by the cache vector parameter (past_key_values) of the Transformer model.

[0068] The embedding methods that can be selected in this step are: (1) direct vector embedding, using the Embedding operation to map characters to a learnable vector; (2) using LSTM or RNN for embedding; (3) using the Transformer architecture for embedding.

[0069] Step 7: Use a large-scale generative pre-training language model (the overall architecture is shown in Figure 3 ) to embed the original input text x n to obtain the cache vector value x n '(The cache vector is an intermediate variable of the Transformer model, representing the key and value of the word vector in each layer of self-attention mechanism, used to calculate the attention weight value of the self-attention mechanism).

[0070] Step 8: For the cache vector values obtained through step 6 and step 7, that is, Through different combination methods as the cache vector value of the large-scale pre-training language model, finally guide the generated result. Possible combination methods are: (1) concatenation according to the order of ; (2) concatenation according to the order of ; (3) concatenation according to the order of .

[0071] Step 9: Use the D' dataset (i.e. the generative supervised dataset), and combine x n ' according to the method of step 8, and then train the model, as shown in Figure 1 , there are two training methods, one is full parameter training, that is, updating the parameters of the entire model; the other is half parameter training, only updating the parameters of the embedding method in step 6.

[0072] Step 10: For the inference or relationship generation stage, as Figure 2As shown, repeat steps 5-8, taking the unstructured text data to be processed as input, and using regular expression matching to extract structured data from the generated text.

[0073] Example 2:

[0074] Embodiment 2 of the present invention provides a knowledge extraction system based on prompting learning, comprising:

[0075] The unstructured text data embedding module is configured to embed the acquired unstructured text data to obtain cached vector values;

[0076] The prompt embedding module is configured to: embed homogeneous prompt strings to obtain homogeneous prompt vector values, and embed heterogeneous prompt strings to obtain heterogeneous prompt character vector values;

[0077] The vector concatenation module is configured to concatenate cached vector values, homogeneous prompt vector values, and heterogeneous prompt character vector values ​​to obtain a concatenated vector, which is then used as the cached vector value for a large-scale pre-trained language model.

[0078] The structured data generation module is configured to extract structured data from text data generated by large-scale pre-trained language models using regular expression matching.

[0079] The working methods of each module of the system are the same as those of the knowledge extraction method based on prompting learning provided in Embodiment 1, and will not be repeated here.

[0080] Example 3:

[0081] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the knowledge extraction method based on prompting learning as described in Embodiment 1 of the present invention.

[0082] Example 4:

[0083] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the knowledge extraction method based on prompting learning as described in Embodiment 1 of the present invention.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for knowledge extraction based on prompt learning, characterized in that, The method comprises the following steps: embedding the obtained unstructured text data to obtain a cache vector value; constructing a supervised data set according to the knowledge architecture and the text data, dividing the supervised data set according to the relationship category or the annotation type to obtain a divided supervised data set, setting a homogeneous prompt string and a heterogeneous prompt string, the homogeneous prompt string refers to setting the same prompt string for each category label, the heterogeneous prompt string refers to setting different prompt strings for each category label, embedding the homogeneous prompt string to obtain a homogeneous prompt vector value, and embedding the heterogeneous prompt string to obtain a heterogeneous prompt character vector value; splicing the cache vector value, the homogeneous prompt vector value and the heterogeneous prompt character vector value to obtain a spliced vector, taking the spliced vector as a cache vector value of a pre-training language model, including that the order of the spliced vector is the homogeneous prompt vector value, the cache vector value and the heterogeneous prompt character vector value in sequence; or the order of the spliced vector is the heterogeneous prompt character vector value, the homogeneous prompt vector value and the cache vector value in sequence; or the order of the spliced vector is the homogeneous prompt vector value, the heterogeneous prompt character vector value and the cache vector value in sequence; obtaining structured data from text data generated by the pre-training language model by using a regular matching method.

2. The knowledge extraction method based on prompt learning according to claim 1, wherein embedding the obtained unstructured text data to obtain a cache vector value comprises: using a generative pre-training language model to embed the original input text to obtain the cache vector value.

3. The knowledge extraction method based on prompt learning according to claim 1, wherein embedding the homogeneous prompt string to obtain the homogeneous prompt vector value and embedding the heterogeneous prompt character to obtain the heterogeneous prompt character vector value comprises: direct vector embedding, using an Embedding operation to map the character into a learnable vector; or using LSTM or RNN for embedding; or using a Transformer architecture for embedding.

4. The knowledge extraction method based on prompt learning according to claim 1, wherein training the pre-training language model comprises: obtaining a generative supervised data set with a prompt template according to the divided supervised data set and the prompt template, and training the pre-training language model by taking the spliced vector as the cache vector value of the pre-training language model according to the generative supervised data set with the prompt template.

5. The knowledge extraction method based on prompt learning according to claim 4, wherein in the training of the pre-training language model, the parameters of the entire model are updated; or the parameters of the embedding process of the homogeneous prompt string and the heterogeneous prompt string are updated.

6. A prompt-based learning knowledge extraction system, characterized in that, comprises: an unstructured text data embedding module configured to embed the obtained unstructured text data to obtain a cache vector value; The prompt embedding module is configured to: construct a supervised data set according to the knowledge architecture and the text data, divide the supervised data set according to a relationship category or a labeling type to obtain a divided supervised data set, set a homogeneous prompt string and a heterogeneous prompt string, the homogeneous prompt string refers to setting the same prompt string for each category label, the heterogeneous prompt string refers to setting different prompt strings for each category label, embed the homogeneous prompt string to obtain a homogeneous prompt vector value, and embed the heterogeneous prompt string to obtain a heterogeneous prompt character vector value; The vector splicing module is configured to: splice the cache vector value, the homogeneous prompt vector value and the heterogeneous prompt character vector value to obtain a spliced vector, and use the spliced vector as the cache vector value of the pre-training language model, including: the order of the spliced vector is the homogeneous prompt vector value, the cache vector value and the heterogeneous prompt character vector value in sequence; or, the order of the spliced vector is the heterogeneous prompt character vector value, the homogeneous prompt vector value and the cache vector value in sequence; or, the order of the spliced vector is the homogeneous prompt vector value, the heterogeneous prompt character vector value and the cache vector value in sequence; The structured data generation module is configured to: use a regular matching method to obtain structured data from text data generated by the pre-training language model.

7. The prompt learning-based knowledge extraction system of claim 6, wherein The training of the pre-training language model includes: obtaining a generative supervised data set with a prompt template according to the divided supervised data set and the prompt template, and training the pre-training language model by using the spliced vector as the cache vector value of the pre-training language model according to the generative supervised data set with the prompt template.

8. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the prompt learning-based knowledge extraction method of any one of claims 1-5.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized by The processor executes the program to implement the steps in the prompt learning-based knowledge extraction method of any one of claims 1-5.

Images