A relation class inference system and method based on prompt learning and context perception

Abstract

This invention discloses a relation category inference system and method based on cue learning and context awareness. First, text preprocessing is performed to obtain standardized text content. A cue learning template is designed based on the connection between the text and relation words. The text is substituted into the cue learning template, a label word mask is introduced, and cue phrases are concatenated. Then, a knowledge-enhanced word mapper is used to obtain the label words at the mask positions for supervised learning. When labeling relation word triples, the label words are restored to relation words and entity words through the word mapper. Next, the text is divided into preceding and following text according to the position of the relation words. Then, unsupervised clustering is performed on the context separately, inferring that different relation words in the same category belong to the same category. This overcomes the limitation that relation categories rely solely on the relation words themselves without considering contextual information, and solves the problem of relation type labeling for unlabeled text in incremental relation extraction scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and particularly relates to computer information extraction technology. It mainly relates to a relationship category inference system and method based on cue learning and context awareness. Background Technology

[0002] With the development of information technology, the amount of text information on the Internet is constantly increasing. Plain text information is difficult for computers to utilize efficiently, while structured data display and storage methods such as knowledge graphs have greater advantages in use and can be better processed by computers. Therefore, structured data has higher utilization value. Extracting high-quality, structured, and valuable information from complex, multi-source, and heterogeneous data is an important topic in current academia. Extracting structured information from unstructured text and storing it in a structured database or knowledge graph is called information extraction. Information extraction is divided into three main directions: entity extraction, relation extraction, and event extraction. Among them, relation extraction technology, as a key link in information extraction technology, can identify or classify the relationship type between given entities in text. For example, from the text "Steve Jobs founded Apple," the relationship type between "Steve Jobs" and "Apple" can be determined as either "founder" or "company manager."

[0003] However, traditional relation extraction focuses only on the learning performance of models on static datasets with predefined relation types. Real-world situations are far more complex, making it difficult to predefine relation types in unlabeled text data within the target domain using expert systems. Furthermore, unlabeled data increases over time, and the types of relations contained within it also expand. How models learn from dynamically increasing data becomes a new challenge in relation extraction, leading to research on incremental relation extraction tasks. That is, the set of relations that the model needs to extract expands with data accumulation, and identifying new relation types from the newly added text data becomes the challenge of incremental relation extraction.

[0004] In relation category inference, the main focus is on inferring the relation category information contained in the given unlabeled text. For example, from the text "Steve Jobs founded Apple," humans can easily identify the relation type as "founded" or "established," while manually labeling relations is costly. Rule-based relation identification methods identify verbs in the sentence as candidate relation words, filter relation words containing dependency relations nsubj and dobj as meaningful relation words, and then rank the relation word frequencies to infer the relation category information of the unlabeled text. These methods are limited by rule design and struggle to adapt to the ever-increasing amount of unlabeled text in incremental relation extraction scenarios. Directly inferring relation categories from unlabeled text is also susceptible to noise in the text, so it's necessary to first extract necessary information from the text, such as entity relation triples, and then use unsupervised clustering and other methods to mine and extract the relation types expressed in that information.

[0005] With the rapid development of deep learning, pre-trained models utilize massive amounts of data during the pre-training stage, allowing them to fully engage with various expressions and semantic information in real-world text. Transferring the knowledge learned during pre-training during fine-tuning for downstream tasks effectively improves performance. Hint learning eliminates the need for adding any network structure after the pre-trained model; instead, it concatenates hints after the test data, fine-tuning the hints to obtain predictions, thus improving model efficiency and reducing storage space. Hint learning has achieved good results in zero-shot and few-shot scenarios in text reasoning, question answering, and relation classification. However, research in information extraction is relatively slow. This inspired this patent to use hint learning to first extract entity relation triples as feature information from unlabeled text, and then use unsupervised clustering to infer relation type information in unlabeled text, thus solving the problem of relation type labeling in unlabeled text during incremental relation extraction. Summary of the Invention

[0006] This invention addresses the problems in existing technologies by providing a relationship category inference system and method based on cue learning and context awareness. First, text preprocessing is performed to obtain standardized text content. Then, a cue learning template is designed based on the connection between the text and relation words. This template consists of text, cue text, and a mask. The cue text comprises cue words and three masks, with the masks corresponding to entity relation triples in the text. The text is substituted into the cue learning template, introducing the label word mask and concatenating the cue phrase. Next, a knowledge-enhanced word mapper is used to obtain the label words at the mask positions for supervised learning. When labeling relation word triples, the label words are restored to relation words and entity words through the word mapper. Then, the text is divided into preceding and following text based on the relation word positions. Unsupervised clustering is then performed on the context, inferring that different relation words in the same category belong to the same category. This overcomes the limitation that relation categories rely solely on the relation words themselves without considering contextual information, solving the problem of relation type labeling for unlabeled text in incremental relation extraction scenarios.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a relationship category inference method based on cue learning and context awareness, specifically including the following steps:

[0008] S1, Data Initialization: Collect entity relation triples and text containing triples to form a labeled dataset. At the same time, collect unlabeled text from the target domain to construct an unlabeled sample set. Preprocess the collected text data, standardize the length of entities and relations, delete cross-sentence triples, and remove special symbols from sentences.

[0009] S2, Constructing a Hint Learning Template: Design a hint learning template based on the relationship between text and relational terms. The template consists of text, hint text, and masks. The hint text consists of hint words and three masks, where the masks correspond to entity relation triples in the text. The template's English form is:

[0010] Inthissentence,therelationbetweenentity[MASK]1and

[0011] entity[MASK]2is[MASK]3

[0012] Where [MASK]1 and [MASK]2 represent entities to be predicted, each with a length of one word, and [MASK]3 represents a relation phrase to be predicted, with a length of three consecutive words. That is, the relation between entities [MASK]1 and [MASK]2 is [MASK]3.

[0013] S3, Knowledge-enhanced word mapper: After passing through the knowledge-enhanced word mapper, the label words at the mask positions are obtained for supervised learning, and the label words are restored into relation words and entity words through the word mapper;

[0014] S4, Construct a relation triplet annotation model: Calculate the Euclidean distance score between the feature vector of the hidden layer at the mask location and the feature vector of the candidate label. The candidate label with the highest score is the model's prediction of the mask location. The objective function is:

[0015] p(y|d)=p([MASK]=V enhanced (y)|T(d))

[0016] Among them, V enhanced (y) represents the tag word obtained by mapping the category word y to the knowledge-enhanced word mapper, where y is the category word, and T(d) represents the text d substituted into the template T;

[0017] S5, Context-Aware Unsupervised Clustering: The text is divided into preceding and following texts according to the position of relation words, and word segmentation is performed to obtain feature vectors for the preceding and following texts respectively. Unsupervised clustering is performed on the feature vectors of the preceding and following texts respectively. The optimal number of cluster centers is obtained by using the inflection point method. Relation words that satisfy the condition that the preceding text is in the same cluster category and the following text is in the same cluster category are classified into the same relation type. Then, the word with the highest frequency in the relation word set is selected as the relation category word.

[0018] To achieve the above objectives, the present invention also adopts the following technical solution: a relation category inference system based on cue learning and context awareness, comprising a relation word triple annotation module and a relation clustering module.

[0019] In the relation word triple annotation module, text preprocessing is performed to obtain standardized text content. The text is then substituted into the prompt learning template, a label word mask is introduced, and a prompt phrase is concatenated. After passing through the knowledge-enhanced word mapper, the label words at the mask positions are obtained for supervised learning. When annotating relation word triples, the label words are restored to relation words and entity words through the word mapper.

[0020] In the relation clustering module, the text is divided into preceding and following text according to the position of relation words. Then, unsupervised clustering is performed on the context separately to infer that different relation words in the context belong to the same category.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) This patent designs and proposes a cueing template for text features required for relation category inference, which can extract entity relation triple information from unlabeled text.

[0023] (2) This patent uses sub-word segmentation and a thesaurus of synonyms to augment the word mapper in the prompt learning process, thereby expanding the coverage of the tagged words in the word mapper.

[0024] (3) This patent proposes to use a context-aware method to cluster the context of the labeled relation words, taking into account the influence of the context information of the relation words on the relation category, breaking through the limitation that the relation category depends only on the relation words themselves without considering context information. Attached Figure Description

[0025] Figure 1 This is a flowchart of the relationship category inference system based on prompting learning and context awareness of the present invention;

[0026] Figure 2 This is a flowchart of step S5 in Embodiment 2 of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0028] Example 1

[0029] A relation category inference system based on cue learning and context awareness includes a relation word triple annotation module and a relation clustering module, and its workflow is as follows: Figure 1 As shown, in the relation word triplet annotation module, text preprocessing is performed to obtain standardized text content, the text is substituted into the prompt learning template, a tag word mask is introduced and a prompt phrase is concatenated, and then the knowledge-enhanced word mapper is used to obtain the tag words at the mask position for supervised learning. When annotating relation word triplets, the tag words are restored to relation words and entity words through the word mapper.

[0030] During the text preprocessing process, the initialization steps of the relation category inference task are as follows: collect entity relation triples and texts containing triples to form a labeled dataset L, then collect unlabeled texts from the target domain to construct an unlabeled sample set U; subsequently, perform text preprocessing to standardize the length of entities and relations, delete cross-sentence triples, and remove special symbols from sentences.

[0031] In the process of constructing the prompt learning template, the prompt template provides hints for the annotation information needed to obtain the text from the pre-trained model. Specifically, the prompt template is manually designed based on the relationship between the text and relation words. The template consists of text, prompt text, and masks. The prompt text consists of prompt words and three masks, with the masks corresponding to entity relation triples in the text. The task of marking entity relation triples in the text is then constructed into a cloze test task, and the relation word triple annotation model is used to predict the words at the mask positions in the text substituted with the prompt template.

[0032] The knowledge-enhanced word mapper specifically includes: segmenting entity words and relation words, using the generated sub-words as label words, predicting mask positions using the predicted label words, and then restoring them to entity words and relation words through the word mapper. It also utilizes a thesaurus to search for words with similar meanings to expand the label words for entity words and relation words, and automatically assigns conflicting label words to entity words or relation words with fewer label words.

[0033] In the relation clustering module, the text is divided into preceding and following text according to the position of relation words. Then, unsupervised clustering is performed on the context and the optimal number of cluster centers is obtained by using the inflection point method. Relation words that satisfy the condition that the preceding text is in the same cluster category and the following text is in the same cluster category are classified into the same relation type. Then, the word with the highest frequency in the relation word set is selected as the relation category word.

[0034] This system breaks through the limitation that relation categories rely solely on the relation terms themselves without considering contextual information, and solves the problem of relation type labeling in unlabeled text in incremental relation extraction scenarios. It can extract entity relation triple information from unlabeled text and is a relation category inference system with low manual cost.

[0035] Example 2

[0036] A relation category inference method based on cue learning and context awareness includes the following steps:

[0037] Step S1, initialization of the relation category inference task, is as follows:

[0038] A labeled dataset L is formed by collecting entity-relation triples and text containing triples. Then, an unlabeled sample set U is constructed by collecting unlabeled text from the target domain. The lengths of entities and relations are standardized: entities are standardized to one word, and relations to three words. Empty entities and empty relations are removed from the dataset. Text containing triples that are not in the same sentence is deleted. Special characters in sentences are removed.

[0039] In the process of standardizing relations, if the relational phrase is longer than three words, the verb part is retained using part-of-speech tagging tools, and the rest is truncated; if the relational phrase is shorter than three words, it is expanded by adding prepositions or auxiliary verbs.

[0040] Step S2: Construct a prompt learning template. The prompt template provides hints on the annotation information needed to obtain text from the pre-trained model. Specifically:

[0041] Based on the relationship between text and relational terms, a cue template is manually designed. The template consists of text, cue text, and masks. The cue text comprises cue words and three masks. To guide the BERT pre-trained model in outputting information relevant to the relation triplet annotation task, contextual cue words, explicit relational cue words, and explicit entity cue words are added. The masks correspond to entity relation triplet information in the text. The English forms of the designed cue template T and s are as follows:

[0042] Inthissentence,therelationbetweenentity[MASK]1and

[0043] entity[MASK]2is[MASK]3

[0044] In this sentence, the relationship between entity [MASK]1 and entity [MASK]2 is [MASK]3.

[0045] In this context, [MASK]1 and [MASK]2 represent entities to be predicted, each with a length of one word, and [MASK]3 represents relational phrases to be predicted, each with a length of three consecutive words.

[0046] For each input text d∈L, input the text into the template T(d):

[0047] dInthissentence,therelationbetweenentity[MASK]1and

[0048] entity[MASK]2is[MASK]3

[0049] The task of tagging entity relation triples in text is transformed into a cloze test task. Then, the relation word triple labeling model is used to predict the words at the mask positions in the text with the prompt template.

[0050] Step S3, a knowledge-enhanced word mapper;

[0051] The model segments entity words and relation words, using the generated sub-words as label words. It predicts the mask positions, including the predicted label words, and then restores them to entity words and relation words via a word mapper. A thesaurus is used to search for words with similar meanings to expand the label words for entity words and relation words. Conflicting label words are automatically assigned to entity words or relation words with fewer label words. In step S2, the task of annotating relation triples in the text is transformed into a cloze test task, i.e., predicting entity and relation words at mask positions. If only entity and relation words within the target range are predicted, their distribution is relatively concentrated; therefore, this patent uses sub-word and thesaurus methods to expand them. During the training phase, the predicted entity and relation words are converted into predicted label words through a word mapper. During the testing phase, the entity and relation prediction results of the relation triple annotation model are restored to label words.

[0052] Step S4: Construct a relation triplet annotation model;

[0053] The text substituted into the prompt template is parsed into a token sequence using the BERT token parser. The pre-trained BERT language model is then used to encode the token sequence, converting words into IDs from the vocab vocabulary, and adding sentence category symbols and sentence segmentation symbols. Word vectors for the tokens are obtained by loading pre-trained parameters, resulting in a word vector matrix for the token sequence. These vectors are then used for feature encoding by the BERT model to obtain the latent feature vectors for each word. The candidate tag words for relation triples are also processed in the same way to obtain their latent feature vectors from the BERT model. Finally, the Euclidean distance score is calculated between the feature vectors of the hidden layer at the mask position and the feature vectors of the candidate tag words. The candidate tag word with the highest score is the model's prediction of the mask position. The objective function is:

[0054] p(y|d)=p([MASK]=V enhanced (y)|T(d))

[0055] Where V enhanced (y) represents the word mapper after knowledge enhancement in step S3, which maps the original entity relation words to label words. y is the entity relation word, i.e., the category word. T(d) represents the text d substituted into the template T. The cross-entropy loss function is used. The scores calculated for the label words in the predicted word text at the mask position are processed by the Softmax function, substituted into the loss function to calculate the loss, and then the parameters of the relation triplet labeling model are updated through backpropagation. In the prediction stage, the word mapper is used to restore the labels to obtain entity words and relation words, and the relation triplet labeling results of the text are obtained.

[0056] Step S5, context-aware unsupervised clustering, specifically as follows: Figure 2 As shown:

[0057] Initialize the Word2Vec encoder. Divide the text into preceding and following paragraphs based on the position of relational words, using d... a and d b This means that word segmentation tools are used to segment the preceding and following text. The preceding text is converted into tokens using the Word2Vec encoder's vocab vocabulary, and the corresponding Word2Vec word vectors are obtained from the tokens. Then, the sentence's feature information is obtained by averaging the feature information of the words.

[0058]

[0059] Among them, w i For text d a The i-th word in For the word w i The corresponding word vector, |d a | represents the number of words contained in the preceding text. Then, the feature vector h of the following text is obtained using the same method. b .

[0060] Then, unsupervised clustering methods were used to cluster the preceding and following texts to mine the similarity between the texts. Euclidean distance was used as the metric for the similarity between the preceding and following texts of the two texts.

[0061]

[0062] in and Let each represent the preceding text of two different texts. Initialize K cluster centers. Calculate the distance from the latent vector of each preceding text to a cluster center, and assign the preceding text to the cluster with the closest latent vector. After partitioning all preceding texts, calculate the sum of the distances from each preceding text to its cluster center. Then, recalculate the cluster center vector for each cluster, calculate the sum of the distances, and calculate the difference between this sum and the previous distance. If the difference is less than a threshold, terminate the clustering and return the partitioned data; otherwise, continue iteratively partitioning the samples. Use the inflection point method to search for the optimal number of cluster centers to obtain the final preceding text clustering result C. a The i-th cluster set is used This indicates that the text consists of the preceding text. The same method is used to obtain the clustering results for the following text, using... Let j represent the j-th cluster set in the following text.

[0063] For text d 1 and d 2 Contexts are divided based on relational words. Relational words that appear in the same cluster in the preceding text and in the same cluster in the following text are inferred to be of the same relation type.

[0064]

[0065] Where r 1 and r 2 Representing text d respectively 1 and d 2 Based on the relational terms marked in step S4, C k This represents the k-th cluster set generated by the context-aware relation clustering module. Finally, the word with the highest frequency in each relation word set is selected as the relation category word to complete the relation category inference.

[0066] Test case

[0067] This test case uses the following metrics for evaluation: On the relation category inference dataset, precision, recall, and F1 are used to evaluate the prediction performance of the relation category inference results. Precision is the number of correctly predicted samples out of those predicted as positive; recall is the number of correctly predicted samples out of those that are actually positive; F1 is the harmonic mean of precision and recall, used to balance precision and recall. Calculation. In practical use, the balance between precision and recall reflects the model's accurate and effective recognition rate of the data. The closer all three values ​​are to 1, the better the model's performance. Perform relation category inference on the test set, using the three metrics for validation, or use the model to infer relation categories in unlabeled text in real-world scenarios.

[0068] The information extraction module based on cue learning in this method was specifically tested on the public datasets OIE2016, WEB, and NYT, and the results are shown in the table below:

[0069]

[0070] As can be seen from the table above, the method of the present invention is superior to the comparative methods. This method can extract entity relation triple information from unlabeled text, expand the coverage of labeled words in the word mapper, and solve the problem of relation type labeling of unlabeled text in incremental relation extraction scenarios.

[0071] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A relation category inference method based on cue learning and context awareness, characterized in that, The method includes the following steps: S1, Data Initialization: Collect entity relation triples and text containing triples to form a labeled dataset. At the same time, collect unlabeled text from the target domain to construct an unlabeled sample set. Preprocess the collected text data, standardize the length of entities and relations, delete cross-sentence triples, and remove special symbols from sentences. S2, Constructing a Hint Learning Template: Design a hint learning template based on the relationship between text and relational terms. The template consists of text, hint text, and masks. The hint text consists of hint words and three masks, where the masks correspond to entity relation triples in the text. The template's English form is: In this sentence, the relation between entity[MASK]1and entity[MASK]2is[MASK]3 Where [MASK]1 and [MASK]2 represent entities to be predicted, each with a length of one word, and [MASK]3 represents a relation phrase to be predicted, with a length of three consecutive words. That is, the relation between entities [MASK]1 and [MASK]2 is [MASK]3. S3, Knowledge-enhanced word mapper: After passing through the knowledge-enhanced word mapper, the label words at the mask positions are obtained for supervised learning, and the label words are restored into relation words and entity words through the word mapper; S4, Construct a relation triplet annotation model: Calculate the Euclidean distance score between the feature vector of the hidden layer at the mask location and the feature vector of the candidate label. The candidate label with the highest score is the model's prediction of the mask location. The objective function is: p(y|d)=p([MASK]=V enhanced (y)|T(d)) Among them, V enhanced (y) represents the tag word obtained by mapping the category word y to the knowledge-enhanced word mapper, where y is the category word, and T(d) represents the text d substituted into the template T; S5, Context-Aware Unsupervised Clustering: The text is divided into preceding and following texts according to the position of relation words, and word segmentation is performed to obtain feature vectors for the preceding and following texts respectively. Unsupervised clustering is performed on the feature vectors of the preceding and following texts respectively. The optimal number of cluster centers is obtained by using the inflection point method. Relation words that satisfy the condition that the preceding text is in the same cluster category and the following text is in the same cluster category are classified into the same relation type. Then, the word with the highest frequency in the relation word set is selected as the relation category word.

2. The relationship category inference method based on cue learning and context awareness as described in claim 1, characterized in that, In step S1, the entity length is standardized to one word, the relation length is standardized to three words, and empty entities and empty relations in the collected text dataset are deleted.

3. The relationship category inference method based on cue learning and context awareness as described in claim 2, characterized in that: In step S1, if the length of the relational phrase is more than three words, the verb part is retained using a part-of-speech tagging tool, and the rest is truncated; if the length of the relation is less than three words, the relation is expanded by adding prepositions or auxiliary verbs.

4. A relation category inference method based on cue learning and context awareness as described in claim 2 or 3, characterized in that: In step S2, the task of marking entity relation triples in the text is constructed into a cloze test task, and the relation word triple labeling model is used to predict the words at the mask positions in the text with the prompt learning template.

5. The relationship category inference method based on cue learning and context awareness as described in claim 1, characterized in that, Step S3 further includes: S31: Segment entity words and relation words, use the generated sub-words as label words, predict the mask position of the model, predict the label words during prediction, and then restore them to entity words and relation words through the word mapper; S32: Expand tag words by searching for words with similar meanings in a thesaurus of synonyms, and automatically assign conflicting tag words to entity words or relation words with fewer tag words.

6. The relationship category inference method based on cue learning and context awareness as described in claim 1, characterized in that, Step S4 further includes: S41: The text substituted into the prompt template is parsed into a token sequence by the BERT token parser, and the pre-trained language model BERT is used to encode the token sequence; S42: Predict the mask in the token sequence, calculate the similarity score between the feature vector of the hidden layer at the mask position and the feature vector of the candidate tag word, and predict the original tag word at the mask position. S43: In the prediction stage, the word mapper is used to restore the labels to obtain entity words or relation words, and the triplet annotation results of the text are obtained.

7. A relation category inference method based on cue learning and context awareness as described in claim 5 or 6, characterized in that, In step S5, during the unsupervised clustering of the feature vectors of the preceding and following texts, an unsupervised clustering method is used to mine the similarity between texts. The clustering uses Euclidean distance as the measure of the similarity between the preceding and following texts of two texts. K cluster centers are initialized, the distance from the latent vector of each text's preceding text to the cluster center is calculated, and the preceding text is assigned to the cluster with the closest distance. After all the preceding texts are divided, the sum of the distances from each text's preceding text to the cluster center is calculated, and then the cluster center vector of each cluster is recalculated. Then the distance sum is calculated and the difference is calculated with the previous distance. If the difference is less than a threshold, the clustering ends and the data result of the division is returned; otherwise, the sample division continues iteratively.

8. A relation category inference system based on cue learning and context-aware learning using the method as described in claim 1, characterized in that, This includes a relation term triple annotation module and a relation clustering module. In the relation word triple annotation module, text preprocessing is performed to obtain standardized text content. The text is then substituted into the prompt learning template, a label word mask is introduced, and a prompt phrase is concatenated. After passing through the knowledge-enhanced word mapper, the label words at the mask positions are obtained for supervised learning. When annotating relation word triples, the label words are restored to relation words and entity words through the word mapper. In the relation clustering module, the text is divided into preceding and following text according to the position of relation words. Then, unsupervised clustering is performed on the context separately to infer that different relation words in the context belong to the same category.

Images