Unsupervised open relation extraction text processing method based on interaction of large and small models

Abstract

An unsupervised open relation extraction text processing method based on interaction of large and small models comprises the steps that firstly, a large language model generates a preliminary pseudo relation label from an input statement, and important clues are provided for subsequent feature relation clustering; secondly, the pseudo labels are used for supervising and fine tuning of a small language model, probability distribution of each category is generated through feature learning of relation instances and a relation guidance feedback algorithm, and the pseudo labels are optimized; and finally, selecting a sample with a high confidence probability as a demonstration sample, and providing positive feedback for the large language model so as to reprocess instances with uncertain relationships to improve relationship extraction. The performance of the InstructORE model is superior to that of an existing non-supervision baseline model, and the InstructORE model has relation extraction performance equivalent to that of a weak supervision method in most cases. The method provides powerful support for construction of knowledge maps, knowledge questions and answers and other applications, and has wide application prospects and popularization value.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and natural language processing, and in particular to an unsupervised open relation extraction text processing method based on large and small model interaction. Background Art

[0002] Relation extraction aims to identify semantic relationships between entities from massive unstructured text or multimodal data, thereby providing knowledge triples for knowledge graph construction. Traditional relationship extraction is mainly limited to closed, specific domains. Although it has achieved excellent performance on fixed corpora, the variability of relationships and uncertainty in the field make it difficult for existing relationship extraction methods to cope with real-world application scenarios. In addition, in the face of the emergence of large-scale text data, it is necessary to automatically extract unknown relationships from massive texts. Therefore, research on relationship extraction technology for open domains is particularly important.

[0003] Relation extraction for open domains can realize the extraction of relations between entities or knowledge triples without predefined relation sets, which is also a common relation extraction scenario in reality. At present, there are two main types of relation extraction methods for open domains: one is to mine new relation triples <entity 1, relation, entity 2> through text and entities, that is, open relation extraction; the other is open triple extraction. This invention will study open relation extraction.

[0004] Early open relation extraction methods were mainly based on pattern recognition and grammatical analysis. These methods rely on predefined patterns or rules to identify relation triplets. With the development of machine learning technology, researchers began to explore open relation extraction methods based on machine learning. These methods use labeled training data for supervised learning to learn how to identify and extract relations from text. Recently, the application of transfer learning and pre-trained models in open relation extraction has also increased. These methods use large-scale text data for training, and then fine-tune or transfer learning on specific tasks to improve the performance of relation extraction. As a key link in information extraction, open relation extraction not only has important research value, but also has broad application prospects. It has applications in the construction of knowledge graphs, information retrieval, question-answering systems, natural language understanding, and other aspects.

[0005] Although research on open relation extraction has made some progress at home and abroad, the existing open relation extraction methods are mainly based on unsupervised self-regulation methods of models or weakly supervised learning that relies on additional training data, so they still face many challenges. The main problems faced by open relation extraction are data sparsity and diversity. In the text data that needs to be processed by open relation extraction, the frequency of occurrence of specific relations may be very low, making the model training data sparse and difficult to capture enough features to accurately identify relations, resulting in low relation extraction accuracy. In practical applications, text data often contains a lot of noise and uncertainty, which will reduce the accuracy of the model. At the same time, the relationship expression forms are also diverse, making it more difficult to extract new relations. Secondly, due to the lack of unified standards and specifications, there may be differences in the expression of relations between different data sources, which increases the difficulty of extraction. In addition, the relationship representation also needs to take into account the complex interactions and dependencies between entities, which makes the relationship representation more complicated. Some previous studies have used transfer learning methods to achieve good results in new relationship discovery, but a large number of training data sets are often scarce and costly in actual real-world scenarios, and high-quality training data requires reliable knowledge bases or manual annotations.

[0006] Compared with the prior art, the differences are as follows:

[0007] Technical comparison with patent CN113011189A "Open entity relationship extraction method, device, equipment and storage medium"

[0008] The structure of patent CN113011189A is mainly based on a single pre-trained generative model, such as UniLM or GPT; the core innovation of this patent is the introduction of a collaborative mechanism between large language models (such as GPT-4, etc.) and small language models (such as BERT, etc.), and the unsupervised deep optimization is achieved through preliminary label generation, adaptive clustering, and guided feedback phased optimization. The two are essentially different in overall model structure design.

[0009] The unsupervised generation model of patent CN113011189A does not have subsequent feedback optimization after completing the initial relationship extraction; the innovation of this patent is to introduce an entropy-driven feedback mechanism to screen labels, add high-confidence samples to the "demonstration sample pool", and use these high-confidence samples to provide positive feedback to the large language model to correct the model's generation errors of low-confidence labels. There is an essential difference between the two in the model feedback mechanism.

[0010] Patent CN113011189A can cope with the needs of extracting open relations within a certain range, but because the model does not have multi-layer optimization capabilities, its scalability and stability may be limited in large-scale data scenarios; this patent provides multi-stage optimization and adaptive feedback mechanisms to enable it to cope with large-scale and diversified data, and is suitable for building complex knowledge graphs, question-answering systems, natural language understanding and other open domain scenarios. The two are essentially different in applicable scenarios. Summary of the invention

[0011] To solve the above technical problems, the present invention proposes an unsupervised open relation extraction text processing method based on large and small model interaction, which aims to accurately identify the semantic associations between entities from large-scale unstructured text data to support applications such as knowledge graph construction and knowledge question answering. First, in the preliminary relationship generation stage, a large language model (such as GPT-4, etc.) is used to receive input sentences and entity pairs, form instructions through the designed prompt template, and then use these instructions to generate preliminary pseudo-relationship labels and entity pair labels. Next, in the relationship recognition optimization stage, the entity types and relationship labels generated in the first stage are clustered and refined, and a small language model (such as BERT, etc.) is trained in a self-supervised manner to obtain the fine-grained probability distribution of the output label of each input sentence on all current clustering categories. Finally, in the instructive relationship demonstration feedback stage, the present invention introduces a relatively complex instructive demonstration selection mechanism, which explicitly places samples with clearer probabilities (i.e., low entropy) in the sample demonstration pool, which provides positive feedback to the large language model, forcing the large language model to reprocess those unclear samples to provide more accurate pseudo-relationship labels. This paper cleverly combines the advantages of large pre-trained language models (such as GPT-4, etc.) and small language models (such as BERT, etc.). Through three-stage iterative optimization, it realizes the relationship extraction task in unsupervised scenarios and improves the accuracy of identifying entity relationships in text data.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] The unsupervised open relation extraction text processing method based on large and small model interaction is characterized by comprising the following steps:

[0014] 1) Preliminary relation generation: Input the sentences and corresponding entities into the large language model to generate preliminary pseudo-relation labels;

[0015] Using a pre-trained large language model to receive input sentences and entity pairs (h i ,t i ), form instructions c through the designed prompt template i ; These instructions are then used to prompt a large language model to generate preliminary pseudo relation labels and entity pair labels

[0016] 2) Relationship recognition optimization: Cluster and refine the entity types and relationship labels generated in the first stage, and train a small language model in a self-supervised manner to obtain a fine-grained probability distribution of the output label of each input sentence over all current clustering categories;

[0017] Use a pre-trained small language model to encode the text description of the entity and obtain a preliminary pseudo label In the original sentence i The context feature f in i ; Cluster the obtained preliminary feature set through an adaptive clustering strategy, and use the clustering category as a training set to fine-tune the small language model to obtain the refined pseudo-label l i ; Use these optimized pseudo-labels as self-supervision signals to further fine-tune the small language model;

[0018] 3) Guiding relationship demonstration feedback: Select samples with high confidence probability as demonstration examples to provide positive feedback to the large language model for reprocessing instances with uncertain relationships to improve the accuracy and quality of labels;

[0019] Use the fine-tuned small language model in each sample (s i ,h i ,t i ) i The probability distribution p of the upper generation i Then, based on the entropy-driven demonstration sample selection mechanism, the entropy is used to measure the uncertainty level of each instance and select samples with high confidence probability distribution, that is, the entropy value H(p i ) Samples below the threshold are used as demonstration samples. These samples are placed in the sample demonstration pool to guide the large language model to reprocess instances with uncertain relationships to provide more accurate pseudo-relationship labels.

[0020] As a further improvement of the present invention, large language models including GPT-4, PaLM 2, and LLaMA 2 are used, and small language models including BERT, MiniLM, and ALBERT are used for interaction.

[0021] As a further improvement of the present invention, step 1) initial relationship generation is specifically as follows:

[0022] Definition 1 Closed relation extraction: For a given set of sentences Predefined relationship set Extract each sentence s i Medium Entity i and tail entity t i A specific relation r in the relation set Rk , where N s Represents the number of sentences in the text collection, N r Represents the number of relations in the relation set, 1≤k≤N r ;

[0023] Definition 2 Open relation extraction: Open relation extraction is to extract a set of relations from a given text corpus without giving a predefined set of relations R. Head entity h i and tail entity t i The task is formalized as a clustering task, which clusters all the relationship sets corresponding to the current sentence set S. grouped into specific categories. The output of open relation extraction is defined as

[0024] Where K is the number of categories of defined relationship clusters, and the relationship of each text sentence instance will be placed in one of the cluster categories. k∈[1,K];

[0025] For an input sentence s i and the corresponding entity pair h in the sentence i ,t i , design the prompt template, and use s i 、h i and t i Fill the position slots of the template to form instruction c i , and then use these instructions to generate preliminary pseudo relation labels in a large language model and entity pair labels As shown in formula (1):

[0026]

[0027] and generate and Entity tag of .

[0028] As a further improvement of the present invention, step 2) relationship identification optimization is specifically as follows:

[0029] In the prompt command text c i Input a large language model and obtain preliminary generation results Finally, a small language model is used to obtain the contextual features of the preliminary pseudo-labels in the original sentence. Subsequently, the refined soft pseudo-labels are identified through a specific clustering method and further used to fine-tune the small language model to generate the probability of each label.

[0030] By incorporating special markers [E start ] and [Eend ] to update the original input sentence s i , indicating the corresponding head entity and the tail entity The boundary position, in addition, [R start ] and [R end ] is included in the relationship tag The purpose of introducing these special tokens is to obtain richer features of all available information such as text and labels in the small language model. The original sentence s i Combined with the generated relationship labels After the update, the sentence is optimized in form. It is expressed as shown in formula (2):

[0031]

[0032] For the optimized The original input sentence is parsed for syntactic dependency to extract the dependency path between the head entity and the tail entity. The average pooling algorithm is then used for all the words on the dependency path to generate the dependency feature f dep ∈R d , as shown in formula (3):

[0033]

[0034] in Indicates about special word [E start ]、[E end ]、[R start ]、[R end ], in addition, MP(.) and AP(.) represent the maximum pooling function and the average pooling function, represents a dependency path of length d, in (s i ,h i ,t i ) in which the clustering feature f i ∈R d is formed by connecting the features of entities, relations, and dependencies, which makes f i Both semantic and syntactic information are considered, as shown in formula (4):

[0035]

[0036] in Indicates the connection symbols between various hidden features;

[0037] Use a small language model to obtain a preliminary feature set Finally, the purpose of pseudo-relationship label clustering is to group F into K cluster categories, where K is a fixed value or manually defined, and an adaptive clustering strategy is adopted.

[0038] As a further improvement of the present invention, the adaptive clustering strategy includes two steps:

[0039] (1) Using high-dimensional features to represent f i ∈R d The feature conversion method to convert to a low-dimensional feature representation uses a nonlinear mapping network g φ ,in is the transformed dimension;

[0040] (2) Learn a soft assignment to assign all n instances to K centroids, thus achieving better clustering results. This strategy promotes high-confidence assignment and is insensitive to the predefined number of clusters;

[0041] First, apply k-means to obtain K initial centroids Then, using the t-student distribution, each instance feature is calculated here and centroid The feature similarity of each sample s is then calculated i The soft assigned label q ik , as shown in formula (5):

[0042]

[0043] Among them, α is a hyperparameter used to adjust the clustering algorithm, using the frequency statistics of the entire feature set to be able to distribute the sample p ik An alternative estimate is made, as shown in formula (6):

[0044]

[0045] in is a normalization factor. In order to adaptively improve the spatial proximity of related entity pairs, soft allocation is used and auxiliary distribution KL divergence between to optimize the nonlinear network g φ , for each sample s i Generate finely optimized pseudo-label l i , formally, the loss of soft clustering is and pseudo-label l i As shown in formula (7):

[0046]

[0047] Using these optimized pseudo labels As a self-supervisory signal, further tune the small language model to obtain a better SLM θ , with relation classification loss As shown in formula (8):

[0048]

[0049] in is the cross entropy loss, onehot(.) returns a one-hot vector for the pseudo-label, and the sentence, head entity, and tail entity are concatenated as input x i , to optimize SLM θ .

[0050] As a further improvement of the present invention, step 3) the guiding relationship demonstration feedback is as follows:

[0051] At this stage, the adjusted SLM is used θ , in each sample (s i ,h i ,t i ) generates a probability distribution p over all pseudo labels i , where the higher the probability value in a certain relationship category, the clearer the small model's choice of relationship for the instance is. Then use the entropy H(p i ) is used to measure the uncertainty level p of each instance i , as shown in formula (9):

[0052]

[0053] If the entropy is small, it means high certainty, and if the entropy is large, it means high ambiguity.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention proposes an unsupervised open relation extraction text processing method based on large and small model interaction, aiming to accurately identify the semantic association between entities from large-scale unstructured text data. The present invention believes that in open relation extraction, some human annotation and selection operations can be replaced by large language models. The rich prior knowledge of large language models (such as GPT-4, etc.) helps to generate robust and reliable pseudo labels, which can be used to fine-tune small language models. In turn, small language models (such as BERT, etc.) can enhance the capabilities of large language models by selecting more instructive demonstration examples. This continuous interaction between large and small language models realizes the extraction of effective data while enhancing their respective capabilities. Secondly, the model gradually improves the accuracy and reliability of relation extraction through three-stage iterative optimization. This iterative mechanism enables the model to continuously learn and adapt to new data and improve extraction performance. Finally, the present invention introduces a complex instructive demonstration selection mechanism, which provides positive feedback to the large language model by selecting samples with high confidence probability as demonstration examples, which is used to reprocess instances with uncertain relations and improve the accuracy and quality of labels. The InstructORE model outperforms the existing unsupervised baseline model and has comparable relation extraction performance to weakly supervised methods in most cases. This invention provides strong support for building knowledge graphs and knowledge question-answering applications, and has broad application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a logic flow chart of the present invention;

[0057] Figure 2 The model process of the present invention is Figure 1 ;

[0058] Figure 3 The model process of the present invention is Figure 2 ;

[0059] Figure 4 The model process of the present invention is Figure 3 . DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0061] The present invention proposes an open relation extraction model InstructORE based on large and small model interaction, which aims to accurately identify the semantic association between entities from large-scale unstructured text data to support applications such as knowledge graph construction and knowledge question answering. First, in the preliminary relation generation stage, a large language model (such as GPT-4, etc.) is used to receive input sentences and entity pairs, form instructions through the designed prompt template, and then use these instructions to generate preliminary pseudo-relation labels and entity pair labels. Next, in the relation identification optimization stage, the entity types and relation labels generated in the first stage are clustered and refined, and a small language model (such as BERT, etc.) is trained in a self-supervised manner to obtain the fine-grained probability distribution of the output label of each input sentence on all current clustering categories. Finally, in the instructive relation demonstration feedback stage, the present invention introduces a relatively complex instructive demonstration selection mechanism, which explicitly places samples with clearer probabilities (i.e., low entropy) in the sample demonstration pool, which provides positive feedback to the large language model, forcing the large language model to reprocess those unclear samples to provide more accurate pseudo-relation labels. This paper cleverly combines the advantages of large pre-trained language models (such as GPT-4, etc.) and small language models (such as BERT, etc.). Through three-stage iterative optimization, it realizes the relationship extraction task in unsupervised scenarios and improves the accuracy of identifying entity relationships in text data.

[0062] As a specific embodiment of the present invention, the present invention provides a logic flow chart as follows Figure 1 The model flow chart is shown in Figure 2 Figure 3 Figure 4 An open relation extraction model InstructORE based on the interaction of large and small models is shown, including the following steps:

[0063] 1) Initial relationship generation:

[0064] Definition 1 (Closed Relation Extraction): For a given set of sentences Predefined relationship set Extract each sentence s i Medium Entity i and tail entity t i A specific relation r in the relation set R k , where N s Represents the number of sentences in the text collection, N r Represents the number of relations in the relation set, 1≤k≤N r .

[0065] Definition 2 (Open Relation Extraction): Open relation extraction is to extract relations from a given text corpus without giving a predefined relation set R. Head entity h i and tail entity ti The task is formalized as a clustering task, which clusters all the relationship sets corresponding to the current sentence set S. grouped into specific categories. The output of open relation extraction is defined as Where K is the number of categories of the defined relation clusters. Each relation of a text sentence instance will be placed in one of the cluster categories, for example k∈[1,K].

[0066] Previous work on weakly supervised methods combined pseudo-label pre-training and manual intervention to achieve the best new relation extraction results for weakly supervised open relation extraction models. This paper assumes that the rich real-world knowledge embedded in large language models can also help generate preliminary pseudo-labels. Figure 2 , this embodiment uses GPT-4 as the large language model in the first stage. GPT-4 (Generative Pre-trained Transformer 4) is the latest GPT series model released by OpenAI and is currently the most powerful text generation model. It is a large-scale multimodal model that uses a wide range of real corpora and complex semantic tasks on the Internet in the pre-training stage. It can accept image and text inputs and generate text outputs. The text output task is still an autoregressive word prediction task. In the first stage, the present invention uses the powerful generation ability of GPT-4 to provide high-value pseudo-labels for open relationship extraction tasks.

[0067] For an input sentence s i and the corresponding entity pair h in the sentence i ,t i The present invention designs a prompt template and uses s i 、h i and t i Fill the position slots of the template to form instruction c i These instructions are then used to prompt GPT-4 to generate preliminary pseudo-relation labels and entity pair labels As shown in formula (1):

[0068]

[0069] For example, in the text "Two years ago, Stephen Stone organized a grand wedding for his beloved Linda Skyler in the romantic city of Paris to express his deep love and commitment", Stephen Stone and Linda Skyler are the head entity and tail entity of the sentence. The prompt texts constructed based on the text and entities all adopt a continuous writing format, for example: "There is a sentence 'Two years ago... in Paris', and the entity types of Stephen Stone and Linda Skyler are XXX and XXX. The relationship between Stephen Stone and Linda Skyler is XXX". In order to ensure that the output of the large language model is more focused on a certain relationship semantics, the present invention also limits the word count, adding the prompt "Please answer no more than three words" in the entire prompt text string and then inputting it into the large language model GPT-4. In a similar situation, GPT-4 can also generate and Entity labels for . In the above example, the initial relations generated from the GPT-4 output are is "spouse", entity tag and They are all “characters”.

[0070] In addition, during the first iteration between the large language model and the small language model, the present invention uses a zero-shot method to input the prompt set into GPT-4 to directly generate preliminary pseudo-labels. As the subsequent iteration process unfolds, the third stage of guided relationship demonstration feedback can help the overall model generate more robust pseudo-labels. Then, the prompt text string is constructed by adding a small number of demonstration samples.

[0071] 2) Relationship identification optimization

[0072] This embodiment uses BERT as a small language model in the second stage. This stage uses the Encoder part of Transformer, which is a self-encoding language model. Usually, the pre-trained language model is pre-trained from a large amount of unlabeled data, mainly to obtain a more general semantic representation, thereby improving the effect of downstream tasks; secondly, it can provide better initialization parameters for the model, so as to have better generalization performance and accelerate convergence. At present, BERT is mainly applied to downstream tasks in two ways, one is the feature extraction method, and the other is the fine-tuning method. The present invention adopts a fine-tuning method, which can use the BERT model to initialize the weights of the network, continuously adjust all the weight parameters of the model during the training process, and finally train a model that can be better applied to specific tasks. The present invention uses BERT to encode the text description of the entity, and performs semantic fusion by combining the contextual encoding representation of different Transformer blocks. The entity description encoder used in the present invention mainly includes three parts: an input layer, an encoding layer, and a mapping layer. BERT has a much larger number of parameters than GPT-4, and can achieve excellent selection performance in a certain task through fine-tuning. Therefore, it is selected as the small language model for optimizing relations in the open relation extraction model InstructORE proposed in the present invention.

[0073] like Figure 3 , and then change the prompt command text c i Input GPT-4 and obtain preliminary generation results Finally, the BERT model is used to obtain the contextual features of the preliminary pseudo-labels in the original sentence. Subsequently, the refined soft pseudo-labels are identified through a specific clustering method and further used to fine-tune BERT to generate the probability of each label. The process and algorithm details of this stage will be detailed below.

[0074] By incorporating special markers [E start ] and [E end ] to update the original input sentence s i , which represent the corresponding head entities and the tail entity In addition, [R start ] and [R end ] is included in the relationship tag The purpose of introducing these special tokens is to obtain richer features of all available information such as text and labels in BERT. For example, Figure 2 In the original sentence s i Combined with the generated relationship labels After being updated to "[CLS]...[Person]Steven Stone[\Person]...[Person]Linda Schuyler[\Person]...[SEP]" the relationship between them is "[Relation]Spouse[\Relation]". In terms of form, the sentence has been optimized. It is expressed as shown in formula (2):

[0075]

[0076] At the same time, in order to analyze the optimized The present invention performs syntactic dependency parsing on the original input sentence to extract the dependency path between the head entity and the tail entity. Figure 2 In the example, the dependency path is The present invention further uses an average pooling algorithm for all word units on the acquired dependency path to generate a dependency characteristic f dep ∈R d , as shown in formula (3):

[0077]

[0078] in Indicates about special word [E start ]、[E end ]、[R start ]、[R end ] are the hidden features between them. In addition, MP(.) and AP(.) represent the maximum pooling function and the average pooling function, represents a dependency path of length d. For example, in (s i ,h i ,t i ) in which the clustering feature f i ∈R d is formed by connecting the features of entities, relationships, and dependencies. This makes f i Both semantic and syntactic information are considered, as shown in formula (4):

[0079]

[0080] in Indicates the connection symbols between hidden features.

[0081] Use the BERT language model to obtain a preliminary feature set Finally, the purpose of pseudo-relation label clustering in the present invention is to group F into K cluster categories (where K can be manually defined). These cluster categories can be used as annotations in the training data set of supervised BERT. The training set is used to fine-tune BERT and then obtain refined labels. Based on the experience of some previous clustering work, the present invention adopts an adaptive clustering strategy, which includes two steps: (1) using a feature conversion method from a high-dimensional feature representation fi∈Rd to a low-dimensional feature representation, where a nonlinear mapping network g is used φ ,in is the transformed dimension; (2) learn a soft assignment to assign all n instances to K centroids, thereby achieving better clustering results. This strategy promotes high-confidence assignment and is insensitive to the predefined number of clusters.

[0082] Here we first apply k-means to obtain K initial centroids Then, using the t-student distribution, each instance feature is calculated here and centroid The feature similarity of each sample s is then calculated i The soft assigned label q ik , as shown in formula (5):

[0083]

[0084] Where α is a hyperparameter used to adjust the clustering algorithm. From another perspective, the present invention also uses the frequency statistics of the entire feature set to be able to distribute the sample p ik An alternative estimate is made, as shown in formula (6):

[0085]

[0086] in is the normalization factor. In order to adaptively improve the spatial proximity of related entity pairs, soft allocation is used and auxiliary distribution KL divergence between to optimize the nonlinear network g φ , for each sample s i Generate finely optimized pseudo-label l i Formally, the loss of soft clustering is and pseudo-label l i As shown in formula (7):

[0087]

[0088] Using these optimized pseudo labels As a self-supervisory signal, further fine-tune the small model BERT to obtain a better BERTθ , with relation classification loss As shown in formula (8):

[0089]

[0090] in is the cross entropy loss, and onehot(.) returns a one-hot vector for the pseudo-label. The sentence, head entity, and tail entity are concatenated as input x i , to optimize BERT θ .

[0091] 3) Mentoring Relationship Demonstration Feedback

[0092] like Figure 4 At this stage, we use the fine-tuned BERT θ , in each sample (s i ,h i ,t i ) generates a probability distribution p over all pseudo labels i , where the higher the probability value in a certain relationship category, the clearer BERT’s choice of relationship for this instance is. Then use the entropy H(p i ) is used to measure the uncertainty level p of each instance i , as shown in formula (9):

[0093]

[0094] If the entropy is small, it means high certainty, and if the entropy is large, it means high ambiguity.

[0095] Based on the above considerations, the present invention designs a guiding relationship feedback mechanism in Algorithm 1, which is mainly based on entropy-driven demonstration sample selection (Entropy-Driven Demonstration Selection). Samples with high confidence assignments, whose entropy values ​​are lower than the threshold u, will be clearly placed in the sample demonstration pool A. On the contrary, samples with vague and uncertain relationships need further evaluation. For example, if sample s i Exceeds the entropy threshold u and exhibits a vague relationship r j , then select D relations from pool A corresponding to r j The high probability demonstration of s serves as feedback to the model's first-stage preliminary relationship generation. This process expects the large language model to better adjust s in context learning. i If the number of iterations exceeds the manually set hyperparameter I max , then use formula (7) i To organize the results to produce the final output

[0096] The following table will introduce Algorithm 1 in detail. The algorithm includes input part, sample set S, relationship set L, probability distribution set P, number of clusters K, demonstration sample set D (lines 1-6); the first step is to set a set A to represent the total set of relationship guidance samples of all relationship categories (line 7); the second step is to traverse the K relationship categories and create a set of relationship guidance sample sets for each relationship category (lines 7-10); the third step is to double traverse all samples and relationship categories, calculate the entropy of the relationship probability of each sample by formula (9), and compare the entropy calculated by the sample with the preset threshold. If the relationship is less than , the sample and the optimized relationship label are spliced, and then the spliced ​​sample is added to the corresponding relationship subset (lines 11-17); the fourth step is to double traverse the number of relationship categories K and the subsets in set A, sort the set of each relationship guidance sample set according to the size of the entropy, and finally obtain the sorted demonstration sample set of all relationship categories (lines 18-20).

[0097]

[0098]

[0099] Example 1

[0100] In the embodiment, the unsupervised open relation extraction text processing method InstructORE proposed in the present invention based on the interaction of large and small models is carried out on two representative datasets Fewrel and Tacred. The existing unsupervised methods (VAE, Etype+, SelfORE) and weakly supervised methods (RSN, RoCORE, MatchPrompt, ASCORE) are selected as baseline models for comparison, and their performance in B 3 , V-measure and ARI. 3 The index is used to evaluate the accuracy of clustering; V-measure is used to measure the balance and completeness of clustering; ARI is used to measure the consistency of clustering.

[0101] The specific implementation is as follows: First, for the experimental set: (1) Fewrel, it includes 80 relationship types and a total of 56,000 instances (2) Tacred, it includes 41 relationship types and 21,773 instances. The experiment divides the Fewrel dataset into 64 predefined relationships and 16 new relationships, and randomly selects 1,600 instances from the new relationships as the test set. For Tacred, the instances marked as no_relation are removed by default, and 30 predefined relationships and 10 new relationships are selected respectively, and 15% of the instances are randomly selected from the new relationships as the test set. However, it is not necessary to use predefined datasets in the experiment, but all experiments are conducted in an unsupervised manner. The large language model GPT-4 contains rich real-world content and is currently the most outstanding generative language model, so the experiment uses GPT-4 as the instruction-based large language model. I max The maximum number of iterations is set to 8. On the Fewrel and Tacred datasets, the cluster number K is set to 16 and 10, respectively. In equation (4), the value of α is set to 1. The number of demonstrations D for feedback is set to 6. The maximum entropy u on the relationship is set to 2.5. The experiment uses the Stanford text parser to generate the dependency tree, and uses the pre-trained bert-base-uncased 12-layer transformer as the sentence encoder model. To prevent overfitting, all BERT parameters are frozen in the experiment, and only the 8th layer is fine-tuned. AdamW is used, and the training batch size and learning rate are 128 and 1e-4, respectively.

[0102] The InstructORE model and the baseline model were tested on the experimental set, and the performance of the models on three types of indicators was compared. It was found that despite not using a training data set or manual annotation, InstructORE outperformed the unsupervised model on all indicators, and even surpassed some excellent weakly supervised models such as MatchPromp and ASCORE on most indicators, and achieved very competitive results on some indicators. 3 and V-measure, significantly outperforming ASCORE by 1.3% and 1.8% respectively. 3InstructORE outperforms all other top models, surpassing MatchPrompt by 1.9%, and shows competitive results in terms of V-measure. Then for the ARI metric on the Fewrel and Tacred datasets, although ASCORE and MatchPrompt achieved the highest scores, InstructORE still provides very close results, only 0.6% lower than ASCORE on Fewrel and 1.3% lower than MatchPrompt on Tacred. In addition, in terms of average score (Avg.), InstructORE shows a good advantage on both datasets, 84.7% and 87.1% on Fewrel and Tacred, respectively, both surpassing the highest average scores of current advanced baseline models.

[0103] Obviously, the InstructORE model of the present invention surpasses some of the current state-of-the-art baseline models without additional data or manual annotation, and demonstrates strong extraction performance in different indicators and datasets, proving the effectiveness of the present invention.

[0104] Example 2

[0105] In the continuous interaction between the large language model and the small model, this embodiment evaluates the performance trend of InstructORE. For the instructive relationship demonstration feedback stage, 2, 4, 6, and 8 demonstration examples are provided to the model in turn. The observed results are B 3 The F1 scores of and V-measure continue to improve with the number of iterations, especially in the first few rounds (for example, the increase from the first round to the second round is 5%-8%, and the increase from the second round to the third round is 3%-4%). ​​In addition, the appropriate number of demonstration examples will also significantly affect the contextual learning ability of the large language model. Compared with InstructORE experimented with 2 or 4 demonstration examples, InstructORE experimented with 6 or 8 demonstration examples achieved 6%-9% higher F1 performance. In addition, as the number of iterations increases, the model extraction performance using 6 and 8 demonstration examples tends to converge, which shows the effectiveness of large language models in low-resource settings.

[0106] Example 3

[0107] In order to evaluate the generalization ability of InstructORE on different data scales, this embodiment randomly extracts 80%, 60%, 40% and 20% of the data from Fewrel and Tacred respectively. Since the two baseline models RoCORE and SelfORE provide the latest available source code, InstructORE is compared and evaluated with them here. It is observed that, on the one hand, InstructORE shows strong performance in low-resource settings, and stably obtains excellent F1 scores of 76% and 82% with only 20% of the samples. On the other hand, as the sample size decreases, the performance of the three methods will decrease. However, compared with RoCORE and SelfORE, InstructORE shows a more gentle decay characteristic. The performance gap between InstructORE and the two baselines widens as the sample size decreases, which also reflects the robustness of the present invention in scenarios with different data volumes.

[0108] Example 4

[0109] In order to study the ability to discriminate fuzzy relations of different difficulty, this embodiment manually selects samples of two different relations, and evaluates the effects of the three indicators of InstructORE under the four samples of different difficulty. It is observed that both the first group (part vs. subject) and the third group (siblings vs. spouse) show low ambiguity, and InstructORE and some of its variant models have achieved satisfactory experimental results. In contrast, the second group (mother vs. child) and the fourth group (headquarters location vs. headquarters city) involve semantically similar relations. InstructORE produced relatively outstanding results overall, but when the instructive relationship demonstration feedback was removed, the F1 value dropped by 26.1% and 17.6% to the greatest extent, which also shows the importance of the interaction between the large language model and the small language model.

[0110] Example 5

[0111] In order to explore the contribution of each semantic feature in the clustering module, this example performs an ablation exploration experiment on the semantic features of different clusters on the Fewrel and Tacred datasets, where Rel represents the relationship type feature, Ent represents the entity and entity type feature, Dep represents the dependency path feature between entity pairs, and All represents all features. The order in which the performance decreases is observed to be as follows:

[0112] InstructORE-All>InstructORE-Rel>InstructORE-Ent>InstructORE-Dep.

[0113] Relative to the InstructORE variant that removes a single other semantic feature, the -Rel variant shows the worst performance, producing an average decrease of 8.6% and 6.8% in the three indicators on the Fewrel and Tacred datasets, respectively. It can be seen that the relation types generated by the large language model play a dominant role.

[0114] From the average values ​​of the indicators after removing the entity and entity type features, we can see that the performance of InstructORE also dropped significantly by 3.3% and 1.7% without Ent. This fully demonstrates that entity features play a complementary role in clustering features and that entity features effectively supplement overall features.

[0115] The performance of InstructORE's relation identification optimized clustering module is slightly reduced when the dependency path feature is removed, with the average score decreasing by 1.3% and 0.8%, respectively. This indicates that the dependency path feature between entity pairs helps to solve the problem of missing information about relation types and entity types to a certain extent.

[0116] Example 6

[0117] This example selects SelfORE and RoCORE as the baselines, and selects three difficult examples to compare with InstructORE, each of which consists of two sentences with fuzzy relations. The following specifically analyzes the relationship recognition of these three models on these three difficult examples.

[0118] In Example 1 (across vs. adjacent), both sentences describe the relationship between a river and a place, with subtle differences in semantics. While all models correctly classified the "across" relation in the first sentence, SelfORE and RoCORE misunderstood the second sentence. Neither unsupervised self-iteration nor the addition of additional transfer learning training data could correct the second sentence from a "across" relation to an "adjacent" relation. In Example 2 (part vs. member), SelfORE completely misclassified the "part" relation in the first sentence as a "member" relation, and misidentified the "member" in the second sentence as a "follows" relation; RoCORE made an incorrect prediction in the second sentence, misidentifying "member" as a "part" relation. InstructORE accurately distinguished the difference between the two instances by iterating the pseudo-relation type labels, accurately distinguishing the two ambiguous sentences and dividing them into the correct relationship set. In Example 3 (military rank vs. voice type), although the two relationships are not very ambiguous, the unsupervised method SelfORE makes incorrect judgments in both cases, identifying both relationships as "subject" relationships, while the weakly supervised RoCORE corrects these problems. This also reflects that the self-supervised iteration of the small language model will also make unpredictable errors in some simple instance scenarios, and external supervision is quite available to solve this problem.

[0119] In summary, the InstructORE model shows higher accuracy, ability to correct misclassification, iterative optimization capability, and robustness when processing fuzzy relation instances. These advantages enable InstructORE to have better performance in the open relation extraction task.

[0120] Example 7

[0121] This embodiment illustrates the pseudo-relationship optimization process between the entities “D-Day” and “Operation Overlord” in the iteration of the large and small models in the sentence “After training in the United States, he served in the European Theater of Operations, receiving the Distinguished Unit Citation for his Operation Overlord on D-Day”, where feedback sample demonstrations and probabilities for the “part” and “member” categories are given.

[0122] When using the large language model GPT-4 to generate pseudo-labels for relations and entities, no relation-guiding demonstration examples were added. Instead, pseudo-relation and entity labels were generated only by design instructions and the capabilities of the large language model itself, and then feedback demonstration example guidance was formally entered. In the first round, the probability between the relations "part" and "member" (53% and 37% probability) was not very clear, and the predicted probability of the two was only 16% different. The relation label of the given demonstration example was "close", which was not very close to the accurate label "part" in semantic similarity and space, but to a certain extent, it could provide optimization ideas for the representation between the entities of "D-Day" and "Operation Overlord" in the future. In the second round, the probability between the relations "part" and "member" (71% and 24% probability) expanded to 47%, among which a demonstration example relation label selected by the small language model BERT was "adjacent", which gradually approached the label "part". In the third round, the probability between the relations "part" and "member" (82% and 16% probability) further expanded to 67%. By the fourth round, the probability between the relations "part" and "member" (82% and 16% probability) expanded to a very significant 83%, and the sample selection of the small language model gave accurate "part" relationship guidance feedback samples after a series of relationship recognition optimizations. Finally, after 5 iterations, the probability distribution became very certain that the probabilities of "part" and "member" were 93% and 2% respectively, and the correct result was obtained.

[0123] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. An unsupervised open relation extraction text processing method based on large and small model interaction, characterized by: The steps include: 1) Preliminary relation generation: Input the sentences and corresponding entities into the large language model to generate preliminary pseudo-relation labels; Using a pre-trained large language model to receive input sentences and entity pairs (h i ,t i ), form instructions c through the designed prompt template i ; These instructions are then used to prompt a large language model to generate preliminary pseudo relation labels and entity pair labels 2) Relationship recognition optimization: Cluster and refine the entity types and relationship labels generated in the first stage, and train a small language model in a self-supervised manner to obtain a fine-grained probability distribution of the output label of each input sentence over all current clustering categories; Use a pre-trained small language model to encode the text description of the entity and obtain a preliminary pseudo label In the original sentence i The context feature f in i ; Cluster the obtained preliminary feature set through an adaptive clustering strategy, and use the clustering category as a training set to fine-tune the small language model to obtain the refined pseudo-label l i ; Use these optimized pseudo-labels as self-supervision signals to further fine-tune the small language model; 3) Guiding relationship demonstration feedback: Select samples with high confidence probability as demonstration examples to provide positive feedback to the large language model for reprocessing instances with uncertain relationships to improve the accuracy and quality of labels; Use the fine-tuned small language model in each sample (s i ,h i ,t i ) i The probability distribution p of the upper generation i Then, based on the entropy-driven demonstration sample selection mechanism, the entropy is used to measure the uncertainty level of each instance and select samples with high confidence probability distribution, that is, the entropy value H(p i ) Samples below the threshold are used as demonstration samples. These samples are placed in the sample demonstration pool to guide the large language model to reprocess instances with uncertain relationships to provide more accurate pseudo-relationship labels.

2. The unsupervised open relation extraction text processing method based on large and small model interaction according to claim 1 is characterized by: Large language models including GPT-4, PaLM 2, and LLaMA 2 are used, and small language models including BERT, MiniLM, and ALBERT are used for interaction.

3. The unsupervised open relation extraction text processing method based on large and small model interaction according to claim 1 or 2, characterized in that: Step 1) Initial relationship generation, as follows: Definition 1 Closed relation extraction: For a given set of sentences Predefined relationship set Extract each sentence s i Medium Entity i and tail entity t i A specific relation r in the relation set R k , where N s Represents the number of sentences in the text collection, N r Represents the number of relations in the relation set, 1≤k≤N r ; Definition 2 Open relation extraction: Open relation extraction is to extract a set of relations from a given text corpus without giving a predefined set of relations R. Head entity h i and tail entity t i The task is formalized as a clustering task, which clusters all the relationship sets corresponding to the current sentence set S. grouped into specific categories. The output of open relation extraction is defined as Where K is the number of categories of defined relationship clusters, and the relationship of each text sentence instance will be placed in one of the cluster categories. For an input sentence s i and the corresponding entity pair h in the sentence i ,t i , design the prompt template, and use s i 、h i and t i Fill the position slots of the template to form instruction c i , and then use these instructions to generate preliminary pseudo relation labels in a large language model and entity pair labels As shown in formula (1): and generate and Entity tag of .

4. The unsupervised open relation extraction text processing method based on large and small model interaction according to claim 1 or 2, characterized in that: Step 2) Relationship identification optimization, as follows: In the prompt command text c i Input the large language model and obtain the preliminary generation result L= Finally, a small language model is used to obtain the contextual features of the preliminary pseudo-labels in the original sentence. Subsequently, the refined soft pseudo-labels are identified through a specific clustering method and further used to fine-tune the small language model to generate the probability of each label. By incorporating special markers [E start ] and [E end ] to update the original input sentence s i , indicating the corresponding head entity and the tail entity The boundary position, in addition, [R start ] and [R end ] is included in the relationship tag The purpose of introducing these special tokens is to obtain richer features of all available information such as text and labels in the small language model. The original sentence s i Combined with the generated relationship labels After the update, the sentence is optimized in form. It is expressed as shown in formula (2): For the optimized The original input sentence is parsed for syntactic dependency to extract the dependency path between the head entity and the tail entity. The average pooling algorithm is then used for all the words on the dependency path to generate the dependency feature f dep ∈R d , as shown in formula (3): in Indicates about special word [E start ]、[E end ]、[R start ]、[R end ], in addition, MP(.) and AP(.) represent the maximum pooling function and the average pooling function, represents a dependency path of length d, in (s i ,h i ,t i ) in which the clustering feature f i ∈R d is formed by connecting the features of entities, relations, and dependencies, which makes f i Both semantic and syntactic information are considered, as shown in formula (4): in Indicates the connection symbols between various hidden features; Use a small language model to obtain a preliminary feature set Finally, the purpose of pseudo-relationship label clustering is to group F into K cluster categories, where K is a fixed value or manually defined, and an adaptive clustering strategy is adopted.

5. The unsupervised open relation extraction text processing method based on large and small model interaction according to claim 4 is characterized by: The adaptive clustering strategy includes two steps: (1) Using high-dimensional features to represent f i ∈R d The feature conversion method to convert to a low-dimensional feature representation uses a nonlinear mapping network g φ ,in is the transformed dimension; (2) Learn a soft assignment to assign all n instances to K centroids, thus achieving better clustering results. This strategy promotes high-confidence assignment and is insensitive to the predefined number of clusters; First, apply k-means to obtain K initial centroids Then, using the t-student distribution, each instance feature is calculated here and centroid The feature similarity of each sample s is then calculated i The soft assigned label q ik , as shown in formula (5): Among them, α is a hyperparameter used to adjust the clustering algorithm, using the frequency statistics of the entire feature set to be able to distribute the sample p ik An alternative estimate is made, as shown in formula (6): in is a normalization factor. In order to adaptively improve the spatial proximity of related entity pairs, soft allocation is used and auxiliary distribution KL divergence between to optimize the nonlinear network g φ , for each sample s i Generate finely optimized pseudo-label l i , formally, the loss of soft clustering is and pseudo-label l i As shown in formula (7): Using these optimized pseudo labels As a self-supervisory signal, further tune the small language model to obtain a better SLM θ , with relation classification loss As shown in formula (8): in is the cross entropy loss, onehot(.) returns a one-hot vector for the pseudo-label, and the sentence, head entity, and tail entity are concatenated as input x i , to optimize SLM θ .

6. The unsupervised open relation extraction text processing method based on large and small model interaction according to claim 5 is characterized by: Step 3) Guidance relationship demonstration feedback, as follows: At this stage, the adjusted SLM is used θ , in each sample (s i ,h i ,t i ) generates a probability distribution p over all pseudo labels i , where the higher the probability value in a certain relationship category, the clearer the small language model is in selecting the relationship for this instance. Then, the entropy H(p i ) is used to measure the uncertainty level p of each instance i , as shown in formula (9): If the entropy is small, it means high certainty, and if the entropy is large, it means high ambiguity.

Images