A concept extraction system based on knowledge hint
By using topic knowledge as language prompts in the knowledge graph, combined with the concept extraction system of BERT and pointer network, the problems of extraction deviation and low accuracy in the existing technology are solved, and high-quality multi-grained concept extraction is achieved, which significantly improves the knowledge graph Completion and extension performance.
Patent Information
- Application Number
- CN202210419758.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-20
AI Technical Summary
There is a decimation bias in the concept extraction method in the existing knowledge graph, resulting in low decimation accuracy, and large-scale pre-trained language models rely on co-occurrence associations rather than semantic correlations, resulting in poor generalization ability.
Using a concept extraction system based on knowledge prompts, by obtaining topic knowledge from the existing knowledge graph as language prompts, combining the pre-trained language model BERT and pointer network with bidirectional coding representation, the prompt builder and concept extractor are built to reduce the extraction bias and Improve the quality of concept extraction.
It significantly improves the accuracy of concept extraction and multi-grained extraction capabilities, reduces extraction bias, enhances the performance of downstream tasks, and performs excellently in knowledge graph completion and ontology expansion.
Smart Images

Figure CN114911947B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and knowledge engineering technology, and specifically relates to a concept extraction system based on knowledge prompts, referred to as KPCE. Background Art
[0002] Cognitive intelligence refers to enabling computers to have the ability to understand the world similar to humans. One of the main ways for humans to understand the world is through language, so natural language processing is an important part of cognitive intelligence. Reflecting on the process of human understanding of natural language, it is not difficult to find that understanding natural language can be regarded as a mapping process from text to knowledge. With the help of existing background knowledge, humans can understand language more easily. Due to the diversity and complexity of the semantics of natural language sentences, computers also need background knowledge to understand natural language. However, for computers, understanding different entities and concepts requires a large enough data set, and understanding the relationship between entities and concepts requires rich semantic relationships. Therefore, high-quality structured knowledge is the key to helping computers understand natural language. In recent years, knowledge graphs (KGs) have attracted widespread attention as a structured form of knowledge. Knowledge graphs are composed of entities, concepts, and relationships. Entities are things in the real world, also known as objects or instances. Concepts are hypernyms of entities and can be understood as categories of entities. For example, when we mention "scientists", we generally refer to a class of people engaged in scientific research, rather than a specific scientist. Attributing an entity to one or several specific concepts can be regarded as a conceptualization process.
[0003] The concepts in the knowledge graph enable machines to better understand natural language, thus playing an important role in many knowledge-intensive fields, such as personalized recommendations, question-answering systems, and common sense reasoning. Although many works have made great efforts in building knowledge graphs in recent years, the concepts in existing knowledge graphs are still far from perfect. For example, in the widely used Chinese knowledge graph CN-DBpedia, there are nearly 17 million entities, but there are only 270,000 concepts in total, and more than 20% of the entities do not even have concepts (hypernyms). Although Probase is a large English knowledge graph, only 30% of the fine-grained concepts have two or more modifiers. The present invention focuses on extracting multi-granular concepts from the descriptive text of entities to complete the existing knowledge graph.
[0004] Most of the existing text-based concept acquisition methods adopt the scheme of directly extracting concepts from text, which can be divided into pattern matching methods and learning-based methods. Pattern matching methods can obtain high-quality concepts, but because such methods rely heavily on templates, they have poor generalization ability and low recall rate. Learning-based methods use large-scale pre-trained language models fine-tuned with labeled data to extract concepts. However, a disadvantage of these methods based on large-scale pre-trained language models that cannot be ignored is the problem of "extraction bias".
[0005] Extraction bias refers to the situation where the extracted text fragments (candidate concepts) are predicted based on their contextual (co-occurrence) associations rather than their true semantic relevance, thus resulting in low extraction accuracy. The root cause of the extraction bias lies in the pre-training process of large-scale pre-trained language models. During pre-training, large-scale pre-trained language models mine statistical associations from a large corpus and tend to make predictions based on co-occurrence correlations rather than true semantic relationships between text strings. For example, for Figure 1 For the entity Louisa May Alcott in the text, these concept extraction models based on large-scale pre-trained language models tend to extract "novel" and "writer" as their concepts together, because these two concepts often appear together in many texts. Obviously, "novel" is a biased and wrong concept.
[0006] The challenge of extraction bias prompted the present invention to adopt the idea of language prompts. Through proper design, language prompts can guide large-scale pre-trained language models to better utilize the knowledge obtained from pre-training, thereby greatly improving the performance of downstream tasks. Therefore, the present invention constructs language prompts by acquiring the subject knowledge in the knowledge graph. The subject prompt helps to discover the knowledge-based association between the target entity and the candidate extraction fragment (candidate concept), and thus can guide the large-scale pre-trained language model to distinguish biased concepts and effectively reduce the extraction bias. Summary of the invention
[0007] The purpose of the present invention is to provide a concept extraction system based on knowledge hints to achieve high-quality multi-granularity concept extraction.
[0008] The concept extraction task proposed by the present invention can be expressed as follows: given an entity e and its related descriptive text x, a set of concepts of e should be extracted from x. Based on this model, the concept extraction model combined with knowledge hints proposed by the present invention can extract massive concepts to improve the performance of downstream tasks, such as knowledge graph completion, ontology expansion, etc. Please note that a necessary condition for realizing concept extraction is to ensure that the given text contains rich concepts. The present invention uses the entity summary text obtained from an online encyclopedia or knowledge base as input, because the summary generally clearly gives the concept of the entity. In order to construct the Chinese training set, the present invention extracted entities and their concepts and summary texts from CN-DBpedia. In order to construct the English training set, the present invention obtained entities and their concepts from Probase, and obtained the summary text of the entity from Wikipedia.
[0009] The concept extraction system combined with knowledge prompts provided by the present invention is referred to as KPCE, and its architecture diagram is as follows: Figure 1 As shown. The extraction bias of large-scale pre-trained language models is significantly alleviated under the guidance of language prompts based on topic knowledge. Specifically, the topic of the target entity is used as an explicit discrete language prompt, so there is no need for complex template design. The topic is obtained from the knowledge of the existing knowledge graph (including IsA relations and entity abstract text), which helps the model discover knowledge-based associations between the target entity and the candidate extraction fragments (candidate concepts). Therefore, the topic prompt can guide the large-scale pre-trained language model to distinguish biased concepts and eliminate extraction bias more effectively. In addition, because neural networks are continuous in nature, discrete language prompts may not be optimal. Therefore, the present invention also automatically searches for prompts in the continuous space to bridge the gap between the pre-training stage of large-scale pre-trained language models and downstream natural language processing tasks. The present invention uses a pre-trained language model represented by bidirectional encoding, referred to as BERT, and random tags as trainable continuous prompts to make the model converge faster. Although the way to fuse these two prompts into a large-scale pre-trained language model is simple, it has been proven to be effective and avoids the impact of complex language prompt templates on downstream task performance. In addition, the present invention adds a pointer network to the language-cue-based BERT, which can repeatedly extract the same segment in the text with an appropriate selection threshold, thereby successfully extracting high-quality concepts of multiple granularities.
[0010] Specifically, the concept extraction system based on knowledge prompts provided by the present invention includes two modules: a prompt builder and a concept extractor; wherein:
[0011] (1) Prompt Builder: In this key component of KPCE, the topic of a given entity is used as a knowledge-guided prompt for a large-scale pre-trained language model BERT. In addition, a trainable continuous prompt is added to enhance concept extraction performance;
[0012] (2) Concept Extractor: Based on the constructed prompts, the concept extractor uses prompts to guide BERT and a pointer network to extract multi-granular, high-quality concepts from the input text.
[0013] The specific details of each part of the model are elaborated below, where bold lowercase letters represent vectors and bold uppercase letters represent matrices.
[0014] (1) Prompt Builder
[0015] In the framework of the present invention, the KPCE model uses BERT as a large-scale pre-trained language model and integrates external knowledge through a language prompt-based paradigm to enhance the concept extraction effect of BERT. Specifically, KPCE uses the topic of a given entity as a knowledge-guided prompt, which is identified based on external knowledge from an existing knowledge graph. During the pre-training process, large-scale pre-trained language models mine statistical associations from a large corpus and tend to make predictions based on co-occurrence correlations rather than true semantic relationships between text strings. Therefore, concept extraction models based on large-scale pre-trained language models tend to extract all concepts related to the entity at the same time. However, related concepts do not necessarily belong to the superordinate concepts of the entity, resulting in extraction bias and low-quality extraction results. However, if the model can be prompted to input the topic of the entity. In this case, the concept extraction model based on a large-scale pre-trained language model reduces the possibility of extracting biased concepts, thereby effectively alleviating the problem of extraction bias. For example, with Figure 1 For the entity Louisa May Alcott in the text, the concept extraction model based on the large-scale pre-trained language model tends to extract both "novel" and "writer" as concepts of Louisa May Alcott. However, if the model can be prompted that the subject of Louisa May Alcott is a person rather than a book, in this case, the concept extraction model based on the large-scale pre-trained language model will reduce the possibility of extracting "novel" or increase the possibility of extracting "writer".
[0016] (1.1) Knowledge-guided language prompt construction. Take CN-DBpedia as an example. The present invention first randomly extracts 1 million entities from the knowledge graph and obtains their existing concepts (hypernyms). Then, KPCE selects the top 100 concepts with the most entities (instances) to form a "typical concept" set, which can cover more than 99.8% of the entities in the knowledge graph. Next, KPCE uses spectral clustering and adaptive K-means algorithms to cluster these typical concepts into several groups, each corresponding to a topic. In order to achieve spectral clustering, KPCE first uses the overlap coefficient to measure the similarity between two concepts:
[0017]
[0018] Among them, ent(c 1 ) and ent(c 2 ) are concepts c 1 and concept c 2 Entity set. δ is a parameter used to avoid the situation where the denominator is zero when some concepts have no entities in the knowledge graph. Based on the above similarity, a similarity matrix of typical concepts can be constructed. In order to determine the optimal number of concept clusters, the present invention calculated the silhouette coefficient (SC) and CalinskiHarabaz index (CHI) of 2 to 30 clusters, from which it was found that the best number of clusters is 17. Therefore, the typical concepts are clustered into 17 groups, and a topic name is defined for each group. In order to identify the topic of a given entity, the present invention predicts the topic of the entity summary as one of the above 17 typical topics through a BERT-based classifier. In order to train a BERT-based topic classifier, the present invention randomly extracts 40,000 entities and their existing concepts in the knowledge graph. Based on the concept clustering results, the topic of each entity can be determined. Specifically, the following tags are used as input to the classifier:
[0019] {[CLS]E[SEP]X[SEP]}(1)
[0020] Where [CLS] and [SEP] are special marks (tokens). 1 , e 2 , ..., e q} is the tag sequence of a given entity e, X = {x 1 , x 2 , ..., x n} is the token sequence of the input text x. Through the multi-head self-attention operation on the input token sequence, the classifier obtains the final hidden state (vector) of the token [CLS], that is, Compute the topic probability distribution:
[0021] P(topic)∈R 17 ,
[0022] Among them, N 1 is the total number of layers, d 1 is the vector dimension. Then, the topic text The topic with the highest probability is identified as the topic of x, which is calculated as follows:
[0023] H0=EW1+B1, (2)
[0024] H l = transformer_encoder(H l-1 ), 1≤l≤N 1 , (3)
[0025]
[0026] topic text =argmax(P(topic i )), 1≤i≤17. (5)
[0027] Where E∈R (q+n+3)×d , is the initial embedding word vector matrix of all input tokens, and d is the embedding size. is the hidden matrix of layer l, It is the Nth 1 The hidden matrix of the layer; is from obtained; in addition, are all trainable parameters; q is the dimension of the vector E, and n is the dimension of the vector X.
[0028] (1.2) Trainable continuous prompt construction. Neural networks are inherently continuous, so the above-mentioned knowledge graph-based topics may not be optimal. Therefore, the present invention further combines trainable continuous prompts to enhance the prompt effect on large-scale pre-trained language models. In order to construct continuous prompts, KPCE uses random tags from BERT. Specifically, for a given entity e, m tags are first randomly selected from the BERT vocabulary, which constitute a random tag set, denoted as C = {c 1 , c 2 , ..., c m Assume that the subject tag sequence of e is denoted by T = {t 1 , t 2 , ..., t k}, then the connection of C and T serves as a comprehensive prompt for e. Next, KPCE connects the token sequence of the language prompt with E and X to form the complete input token sequence of the extraction model:
[0029] {[CLS]CT[SEP]E[SEP]X[SEP]}. (6)
[0030] To encode the characters in T, E, and X, KPCE uses BERT’s word embedding. KPCE uses a bidirectional long short-term memory network (LSTM) and a two-layer perceptron (MLP) activated by ReLU to encode continuous cues.
[0031] O C =BiLSTM(E C ), (7)
[0032] H C =MLP(O C ). (8)
[0033] Among them, E C ∈R m×d is the randomly initialized embedding matrix of C, O C and H C is an m×d matrix. Then, input the result into formula (6), and we get:
[0034] {emb([CLS])H C emb(T[SEP]E[SEP]X[SEP])}, (9)
[0035] Among them, H C ={h 1 ,h 2 , ..., h m} is a trainable embedding group, and emb(·) represents the operation of obtaining BERT word embeddings. After training, it can find the optimal continuous prompt beyond the BERT vocabulary.
[0036] (2) Concept Extractor
[0037] Formula 9 is input into the prompt-based BERT and pointer network to obtain candidate fragments (candidate concepts) and their corresponding concept confidence scores. The pointer network is specifically used to extract multi-granular concepts. Through the multi-head self-attention operation on the input embedding, BERT outputs the final hidden state (matrix), that is, where d 2 is the vector dimension, N 2 is the total number of layers. Use A pointer network is constructed to predict the probability of each character being the start and end position of the extracted concept. start , p end ∈R (k+m+q+n+4) to represent the probabilities of all characters being the start and end positions respectively. They are calculated as:
[0038]
[0039] in, B∈R (k+m+q+n+4)×2 are all trainable parameters. To generate the extraction result, only the probability of each character in the entity summary text is considered. Given an extraction fragment, i and x j As the characters of its starting position and ending position respectively, the confidence score cs of the extracted fragment ij ∈R, can be calculated as:
[0040]
[0041] Accordingly, the pointer network model generates a ranked list of candidate concepts and their confidence scores, and outputs concepts with confidence scores greater than a selection threshold. Note that a segment can be repeatedly output as the same subsequence of multiple extracted concepts by appropriate selection thresholds. For example, Figure 2 As shown in Figure 2, when the confidence score threshold is set to 0.30, “writer” is extracted multiple times as a subsequence of three concepts of different granularity. Therefore, this strategy enables KPCE to extract multi-granularity concepts.
[0042] In the training process, KPCE uses the cross entropy function CrossEntropy(·) as the loss function of the KPCE model. Specifically, assuming that the set y start ∈N k+m+q+n+4 (y end ∈N k+m+q+n+4 ) contains the true label (0 or 1) for each input character that is the start (or end) position of the concept. Then, for the prediction of these two cases, there are the following two training losses:
[0043] L start =CrossEntropy(p start ,y start ), (12)
[0044] L end =CrossEntropy(p end ,y end ), (13)
[0045] Then, the overall training loss is
[0046] L=αL start +(1-α)L end (14)
[0047] Where α∈(0,1) is a control parameter. The present invention uses Adam to optimize the loss function. Assume Θ Cis the set of parameters for the trainable continuous cue, Θ B is the remaining parameter set of the model. KPCE optimizes Θ by the following objective C (or Θ B ),
[0048]
[0049] Through this training method, KPCE can obtain the optimal continuous prompts, thereby guiding BERT to achieve high-quality multi-granularity concept extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a framework diagram of the concept extraction model KPCE of the present invention.
[0051] Figure 2 It is a diagram of the pointer network of the present invention. DETAILED DESCRIPTION
[0052] The advantages of the present invention are further described below through specific implementation examples and comparative experiments.
[0053] 1. Evaluation setting. Because some extracted concepts do not exist in the knowledge graph, they cannot be automatically evaluated. In order to evaluate the extraction performance of all models, the experiment invited some volunteers to evaluate whether the extracted concepts are correct. The experiment represents the concepts that are already hypernyms of a given entity and exist in the knowledge graph as EC (existing concepts); the correct concepts that are hypernyms of a given entity and do not exist in the knowledge graph are represented as NC (new concepts). Since the extracted concepts of a given entity may already exist in the knowledge graph as hypernyms of other entities, the NC in the evaluation statistics corresponds to a new IsA relationship for the given entity, rather than a unique new concept. A total of four annotators were employed to ensure the quality of the evaluation. Each concept is marked as 0, 1, or 2 by three annotators, where 0 represents an incorrectly predicted concept for a given entity, and 1 and 2 represent EC and NC, respectively. If the results of the three annotators are different, a fourth annotator will be hired for a final check.
[0054] 2. Hyperparameter settings. The experiments were conducted on a dual GeForce GTX 1080Ti workstation with 32G memory and Torch1.7.1 environment. BERT-base with 12 layers and 12 self-attention heads was used as the topic classifier and concept extractor in KPCE. The training settings for the topic classifier are: d = 768, batch size = 16, learning rate = 3e-5, dropout rate = 0.1, and training epoch = 2. The training settings for the concept extractor are: d = 768, m = 30, batchsize = 4, learning rate = 3e-5, dropout rate = 0.1, and training epoch = 2. In addition, α in Formula 14 is set to 0.3, and the selection threshold of candidate fragments in the concept extractor is set to 0.12 according to parameter adjustment.
[0055] 3. Comparative experimental results and analysis. The evaluation results comparing KPCE with all baseline models are listed in Table 1, where EC# and NC# are the number of ECs and NCs extracted from 500 test instances, respectively. From the table, it can be found that KPCE can extract more new concepts for a given entity, and even obtain concepts that are not in the knowledge graph. Therefore, KPCE can extract rich concepts and its performance is better than the baseline model. The KPCE model is further compared with the following ablation variants to verify the effectiveness of the two prompts. KPCE-T is a variant of the concept extractor without knowledge-guided prompts (T); KPCE-C is a variant of the concept extractor without continuous prompts (C); KPCE-T&C is a variant without two prompts. The present invention investigates their performance on the Chinese test set and the English test set, which are shown in Table 2. According to the results, KPCE-C and KPCE-T outperform KPCE-T&C, verifying that adding knowledge-guided prompts and continuous prompts both show the greatest performance improvement. The superiority of KPCE-T over KPCE-C verifies that topic prompts are more effective than continuous prompts in guiding BERT to achieve accurate concept extraction.
[0056] Table 1
[0057]
[0058] Table 2
[0059]
[0060] In addition, Table 3 shows an entity Korean alphabet, and its first 5 extracted segments and the corresponding confidence scores of the segments, whose topic is identified as technology in Probase. From the extraction results of this case, it can be found that two hints (T&C) can guide the BERT-based extractor to reduce the confidence scores of incorrectly extracted segments (the 1st and 5th spans in KPCE-T&C) and increase the confidence scores of extracted segments (such as "system", "writing system"), thereby obtaining more accurate extraction results. Next, the experiment compares KPCE with KPCE LDA A comparison was made, where topics are keywords obtained by using the LDA algorithm on the summaries of all entities. The optimal number of LDA topic classes was also determined to be 17. For a given entity, its topic is identified as the keyword with the highest probability in its topic class. In addition, the experiment also compared KPCE with ERNIE, which can implicitly learn prior knowledge of phrases and entities. The comparison results are listed in Table 4, which shows that the knowledge-guided prompts designed by KPCE make more full use of the value of external knowledge than the other two schemes.
[0061] Table 3
[0062]
[0063] Table 4
[0064]
[0065] The experiment further ran KPCE for all entities present in CN-DBpedia to supplement the knowledge graph with a large number of concepts. The relevant statistics of the extracted concepts are listed in Table 5, where UC\# is the number of unique concepts. It shows that KPCE can extract new concepts for each entity from its descriptive text and supplement a large number of unique concepts that do not exist in the knowledge graph. Therefore, the KPCE model can achieve large-scale concept completion of existing knowledge graphs. In addition, the present invention applied KPCE on Meituan, a well-known e-commerce platform in China. After running KPCE, it was found that KPCE extracted 81,800 new concepts (new IsA relations) from the text with an accuracy of 82.66%. It can be considered that KPCE can be applied to different fields to achieve cross-domain concept acquisition.
[0066] Table 5
[0067] Data Types EC# UC# NC# UNC# Raw data 223,434,251 270,025 - - Extracting data 8,483,718 2,043,771 7,623,111 1,945,510 .
[0068] References:
[0069] [1]Bo X,Yong X,Liang J,et al.CN-DBpedia:A Never-Ending ChineseKnowledge Extraction System[C] / / International Conference on Industrial,Engineering and Other Applications of Applied Intelligent Systems.Springer,Cham,2017.
[0070] [2]Wu W,Li H,Wang H,et al.Probase:A probabilistic taxonomy for textunderstanding[C] / / Proceedings of the 2012ACM SIGMOD International Conferenceon Management of Data.2012:481-492.
[0071] [3]Liu X,Zheng Y,Du Z,et al.GPT understands,too[J].arXiv preprintarXiv:2103.10385,2021.
[0072] [4]Vinyals O,Fortunato M,Jaitly N.Pointer networks[J].Advances inneural information processing systems,2015,28.
[0073] [5]Devlin J,Chang M W,Lee K,et al.BERT:Pre-training of deepbidirectional transformers for language understanding[J].arXiv preprintarXiv:1810.04805,2018。
Claims
1. A concept extraction system based on knowledge prompts, in which: The concept extraction task is described as follows: given an entity e and its related descriptive text x, a set of concepts of e should be extracted from x. Based on this model, a concept extraction model based on knowledge prompts is proposed to extract massive concepts to improve the performance of downstream tasks. The model is characterized by guiding large-scale pre-trained language models to better utilize the knowledge obtained in pre-training by designing language prompts, thereby improving the performance of concept extraction. The model consists of two modules: a prompt builder and a concept extractor. Among them: (1) Prompt Builder: The topic of a given entity is used as a pre-trained language model for bidirectional encoding representation, referred to as BERT’s knowledge-guided prompt. In addition, a trainable continuous prompt is added to enhance concept extraction performance. (2) Concept Extractor: Based on the prompts from the builder, the concept extractor uses the prompts to guide BERT and a pointer network to extract multi-granular, high-quality concepts from the input text; The prompt builder uses BERT as a large-scale pre-trained language model and integrates external knowledge through a language prompt-based paradigm to enhance the concept extraction effect of BERT; In the following, bold lowercase letters represent vectors, and bold uppercase letters represent matrices; Specifically, the topic of a given entity is used as a knowledge-guided hint, which is identified based on external knowledge from an existing knowledge graph; during the pre-training process, a large-scale pre-trained language model mines statistical correlations from a large corpus and makes predictions based on co-occurrence correlations rather than true semantic relationships between text strings, so that the concept extraction model based on the large-scale pre-trained language model extracts all concepts related to the entity at the same time; (1.1) Knowledge-guided language prompt construction; First, 1 million entities are randomly extracted from the knowledge graph and their existing concepts, i.e., hypernyms, are obtained. Then, the top 100 concepts with the most entities are selected to form a typical concept set, which covers more than 99.8% of the entities in the knowledge graph. Next, these typical concepts are clustered into several groups using spectral clustering and adaptive K-means algorithms, each corresponding to a topic. In order to implement spectral clustering, the overlap coefficient is first used to measure the similarity between two concepts: Among them, ent(c1) and ent(c2) are the entity sets of concepts c1 and c2 respectively, and δ is a parameter used to avoid the situation where the denominator is zero when some concepts have no entities in the knowledge graph; based on the above similarities, a similarity matrix of typical concepts is constructed; in order to determine the optimal number of concept clusters, the silhouette coefficient (SC) and CalinskiHarabaz index (CHI) of 2 to 30 clusters are calculated, and the best number of clusters is 17; therefore, the typical concepts are clustered into 17 groups, and a topic name is defined for each group; in order to identify the topic of a given entity, the topic of the entity summary is predicted as one of the above 17 typical topics by a BERT-based classifier; in order to train a BERT-based topic classifier, 40,000 entities and their existing concepts in the knowledge graph are randomly extracted, and the topic of each entity is determined based on the concept clustering results; specifically, the following tags are used as input to the classifier: {[CLS]E[SEP]X[SEP]} (1) Among them, [CLS] and [SEP] are special tags; E = {e1, e2, ..., e q } is the tag sequence of a given entity e, X = {x1, x2, ..., x n } is the token sequence of the input text x; through the multi-head self-attention operation on the input token sequence, the classifier obtains the final hidden state of the token [CLS], that is, Compute the topic probability distribution: P(topic)∈R 17 , Among them, N1 is the total number of layers, d1 is the vector dimension; then, the topic text The topic with the highest probability is identified as the topic of x, which is calculated as follows: H0=EW1+B1, (2) H l =transformor_encoder(H l-1 ),1≤l≤N1, (3) topic text =argmax(P(topic i )),1≤i≤17; (5) Where E∈R (q+n+3)×d , is the initial embedding word vector matrix of all input tokens, and d is the embedding size; is the hidden matrix of layer l, is the hidden matrix of the N1th layer; is from obtained; in addition, are all trainable parameters; q is the dimension of vector E, n is the dimension of vector X; (1.2) Trainable continuous prompt construction; To construct continuous hints, random tokens from BERT are used; specifically, for a given entity e, m tokens are first randomly selected from BERT’s vocabulary to form a random token set, denoted as C = {c1, c2, ..., c m }; Assume that the subject tag sequence of e is recorded as T = {t1, t2, ..., t k }, then the connection of C and T serves as a comprehensive prompt for e; next, the token sequence of the language prompt is connected with E and X to form the complete input token sequence of the extraction model: {[CLS]CT[SEP]E[SEP]X[SEP]}; (6) To encode the characters in T, E, and X, BERT’s word embedding is used; specifically, a bidirectional long short-term memory network (LSTM) and a two-layer perceptron (MLP) activated by ReLU are used to encode continuous prompts: O C =BiLSTM(E C ), (7) H C =MLP(O C ). (8) Among them, E C ∈R m×d is the randomly initialized embedding matrix of C, O C and H C is an m×d matrix; then, input the result into formula (6) to obtain: {emb([CLS])H C emb(T[SEP]E[SEP]X[SEP])}, (9) Among them, H C ={h1, h2, ..., h m } is a trainable embedding group, and emb(·) represents the operation of obtaining BERT word embeddings; after training, it can find the optimal continuous prompt beyond the BERT vocabulary; In the concept extractor, Formula 9 is input into the prompt-based BERT and pointer network to obtain candidate fragments, i.e., candidate concepts, and their corresponding concept confidence scores; wherein the pointer network is used to extract multi-granularity concepts; through the multi-head self-attention operation on the input embedding, BERT outputs the final hidden state, i.e. Where d2 is the vector dimension and N2 is the total number of layers; use Construct a pointer network to predict the probability of each character as the start and end position of the extracted concept; use p start , p end ∈R (k+m+q+n+4) To represent the probabilities of all characters being the start and end positions, they are calculated as: in, B∈R (k+m+q+n+4)×2 , are all trainable parameters; To generate the extraction result, we only consider the probability of each character in the entity summary text. Given an extraction fragment, we can use x and x j As the characters of its starting position and ending position respectively, the confidence score cs of the extracted fragment ij ∈R, the calculation formula is: Accordingly, the pointer network model generates a ranked list of candidate concepts and their confidence scores, and outputs concepts with confidence scores greater than a selection threshold; In the training process, the cross entropy function CrossEntropy(·) is used as the loss function of the KPCE model; specifically, assuming that the set y start ∈N k+m+q+n+4 or end ∈N k+m+q+n+4 Contains the true label for each input character that is the start or end position of the concept, then there are two training losses for the prediction of these two cases: L start =CrossEntropy(p start ,y start ), (12) L end =CrossEntropy(p end ,y end ), (13) Then, the overall training loss is L=αL start +(1-α)L end (14) Among them, α∈(0,1) is the control parameter; Use Adam to optimize the loss function; assuming Θ C is the set of parameters for the trainable continuous cue, Θ B is the remaining set of parameters of the model; Θ is optimized by the following objective C or Θ B , Through training, the optimal continuous prompts are obtained to guide BERT to achieve high-quality and multi-granular concept extraction.
2. The concept extraction system based on knowledge prompt according to claim 1 is characterized in that: Entities, concepts, and summary texts are extracted from CN-DBpedia to construct a Chinese training set. Entities and concepts are obtained from Probase, and summary texts of entities are obtained from Wikipedia to construct an English training set.
