Zero sample entity relationship extraction method and system based on large language model information enhancement and inverse translation learning
By performing information enhancement and anti-translation learning on large language models, the problem of insufficient relationship extraction performance and interpretability in the existing technology in a small sample scenario is solved, and higher accuracy and interpretability are achieved.
Patent Information
- Application Number
- CN202411792002.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art is difficult to maintain high relationship extraction performance and interpretability in a small sample scenario.
The information enhancement and anti-translation learning method based on large language models is adopted, and the logical framework of candidate relationships is constructed by introducing additional information and language common sense, and the output of the pre-trained model is iteratively improved through anti-translation learning.
It improves the performance and interpretability of the model in a zero-sample environment, improves the accuracy of relationship extraction, and reaches the current most advanced technical level.
Smart Images
Figure CN119721038A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and in particular, relates to a zero-sample entity relationship extraction method and system based on large language model information enhancement and back-translation learning. Background Art
[0002] Relation extraction is a core task in natural language processing, which aims to identify and extract semantic relationships between entities from unstructured text data. It is usually an important part of information extraction and is widely used in knowledge graph construction, question-answering systems, information retrieval and other fields.
[0003] In recent years, with the development of large language models and computing power, many methods based on large language models have been applied to relation extraction tasks, and relation extraction technology has also made rapid progress. For given text information, the relationship categories between entities in a specific relation set are identified. However, existing methods often have difficulty maintaining high performance and interpretability in few-sample scenarios. Summary of the invention
[0004] In order to address the shortcomings of the prior art, the present invention provides a zero-sample entity relationship extraction method and system based on large language model information enhancement and back-translation learning. Flexible and practical additional information is introduced through a large language model. At the same time, language common sense is introduced to construct a logical framework between candidate relationships, and back-translation learning is performed on the pre-trained model, thereby improving the performance and interpretability of the model.
[0005] To this end, the present invention adopts the following technical solutions:
[0006] On the one hand, the present invention provides a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning, comprising:
[0007] Get the original text and a set of relations that the text is concerned with, and convert the relation words into text features with explanatory information;
[0008] Use the pre-trained language model to encode the obtained text features and obtain text attention features;
[0009] According to the text attention feature, cross attention is used to calculate the attention scores between entities and entities, and between entities and relational words in the text, and candidate bigrams are screened out according to the scores, and the candidate bigrams are concatenated to obtain initial triplets;
[0010] Modify the initial triples according to the rules in the knowledge base of the back-translation learning framework to obtain pseudo labels; use the pseudo labels to iterate the back-translation learning of the results of the pre-trained language model until the output of the pre-trained language model no longer conflicts with the knowledge base;
[0011] Generate relation triplets using the pre-trained language model after back-translation learning iterations.
[0012] Furthermore, the pre-trained language model is an ALBERT model.
[0013] Furthermore, the knowledge base is constructed based on the language common sense of the large language model and the knowledge common sense provided by Wikipedia.
[0014] Furthermore, the dimension of the relational features is between 128 and 512.
[0015] Furthermore, the training loss function is a focal loss function.
[0016] Furthermore, the relational words are converted into text features with explanatory information, including: using llama-3 to convert the relational words into text features with explanatory information.
[0017] Furthermore, the llama-3 model is used to transform the relational words into text features with explanatory information, including:
[0018] The text representation of each sentence is E = {E [CLS] , E1, E2, ..., E L , E [SEP]}, each relational descriptive statement is represented by R = {R [CLS] , R1, R2, ..., R M , R [SEP]}, [CLS] and [SEP] are the head and tail tags of the sentence, used to mark the starting and ending positions;
[0019] The tokenizer converts text input into numeric input T = {T [CLS] , T1, T2, ..., T L , T [SEP]}, and transform the relational descriptive statement into Tr={Tr [CLS] , Tr1, Tr2, ..., Tr M ,Tr [SEP]},in L is the length of the original text, and M is the length of the descriptive sentence.
[0020] In another aspect, the present invention further discloses a zero-sample entity relationship extraction system based on large language model information enhancement and back-translation learning, comprising:
[0021] The text feature acquisition module acquires the original text and a set of relations that the text is concerned with, and converts the relation words into text features with explanatory information;
[0022] The attention feature acquisition module uses the pre-trained language model to encode the acquired text features and obtain text attention features;
[0023] An initial triplet acquisition module, for the text attention feature, uses cross attention to calculate the attention scores between entities and entities, and between entities and relational words in the text, and selects candidate bigrams according to the scores, and concatenates the candidate bigrams to obtain initial triplet;
[0024] The back-translation learning iteration module modifies the initial triples according to the rules in the knowledge base to obtain pseudo labels; the pseudo labels are used to iterate the back-translation learning of the results of the pre-trained language model until the output of the pre-trained language model no longer conflicts with the knowledge base;
[0025] The relation prediction module generates relation triplets using the pre-trained language model after back-translation learning iterations.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1) Generate descriptive information of relationship categories through large language models, enhance the quality of input information, and improve the final prediction results.
[0028] 2) In a zero-shot environment, the knowledge base of the back-translation learning framework is used to correct the interpretable preliminary predictions generated by the initial pre-trained model into relatively credible and interpretable pseudo-labels, and then invested in the training and iteration of the initial model, so that the final model and its predictions have higher accuracy and interpretability.
[0029] 3) The present invention can effectively improve the accuracy of relation extraction, and its performance on multiple data sets reaches the results of SOTA. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0031] Figure 1 A flowchart of a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning in an embodiment of the present invention;
[0032] Figure 2 This is a schematic diagram of reverse translation learning in an embodiment of the present invention;
[0033] Figure 3 It is a schematic diagram of the model in the embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0036] like Figure 1-3 As shown, an embodiment of the present invention provides a zero-sample entity relationship extraction method based on large language model information enhancement and reverse translation learning. First, the candidate relationship is converted into a set of descriptive texts by llama3 (llama3 is the third generation of the large language model "Large Language Model Meta AI" or "Large Language Model Meta AI" based on transformer, which has stronger performance under the same level of parameters). After merging with the input text, it is converted into a primary feature acceptable to the model by Tokenizer (marker generator). Then the primary feature is input into the initial pre-trained language model to obtain the attention representation of the text. Then self-attention operation and threshold screening are performed to obtain entity-entity and entity-relationship tuples, and candidate entity-relationship-entity triples are obtained after splicing. The candidate triples are input into the reverse translation learning module, and are corrected according to the rules in its preset knowledge base to obtain pseudo labels. After that, the pre-trained language model is trained and iterated using pseudo labels until the reverse translation learning module no longer returns the correction to the output of the pre-trained language model, that is, the prediction given by the pre-trained language model no longer conflicts with the knowledge base preset by the reverse deduction learning module, thereby outputting the final prediction.
[0037] The specific steps are as follows:
[0038] S1. Obtaining primary features: The input includes the original text and a set of relations that the text is concerned with. A large language model (llama3) is used to generate descriptive sentences for the relations, which are then merged with the original text and converted by the Tokenizer into input that the model can accept.
[0039] The text representation of each sentence is E = {E [CLS] , E1, E2, ..., E L , E [SEP]}, each relational descriptive statement is represented by R = {R [CLS] , R1, R2, ..., R M , R [SEP]}, [CLS] and [SEP] are the head and tail tags of the sentence, which are used to mark the start and end positions. Tokenizer converts text input into numerical input T = {T [CLS] , T1, T2, ..., T L , T [SEP]}, and transform the relational descriptive statement into Tr={Tr [CLS] , Tr1, Tr2, ..., Tr M ,Tr [SEP]},in L is the length of the original text, and M is the length of the descriptive sentence.
[0040] S2. Obtain attention features: Use the pre-trained model ALBERT to encode the text. ALBERT (an enhanced version of the Bert (Bidirectional Encoder Representations from Transformers) model) can encode the contextual information of words and sentences in the text, including the words of the original text and descriptive sentences about the relationship, which helps the model understand the complex context in the text.
[0041] In the embodiment of the present invention, the ALBERT model is used for text encoding. 1 ,Tr 2 ...} Generate a contextual word embedding sequence through ALBERT The initial symbol encoding of the relation description statement is retained as the encoding of the entire relation. Lr is the total number of relationships involved.
[0042] S3, obtain entity-relationship-entity triples: Take the encoded original text and the sentence start tokens of each relationship to perform cross-attention calculations to obtain the relevance. Concatenate the highly relevance "entity-entity", "entity-relationship", and "relationship-entity" tuples to obtain the "entity-relationship-entity" relationship triples output in this round.
[0043] Specifically, the cross attention mechanism is used to t Calculate the correlation of the sequence.
[0044] For any input:
[0045]
[0046] Where s is the total length of the input, s = L + Lr. Each row in the matrix represents a vector representation of a character or a relationship.
[0047] Let: K = U t ×W K (2)
[0048] Q=U t ×W Q (3)
[0049] V=U t ×W V (4)
[0050]
[0051] Where W K , W Q , W V ∈R d×s .
[0052] Get the matrix Attention(K, Q, V):
[0053]
[0054] If there is a ij >θ, then it is considered that the ith character is connected to the th character.
[0055] The connection pairs are restored to character pairs, that is, "entity-entity", "entity-relationship" and "relationship-entity" tuples. Then, the "entity-relationship" and "relationship-entity" tuples with common relationships are first concatenated into "entity-relationship-entity" relationship triplets, and then the above triplets are modified according to the "entity-entity" tuple list to obtain the relationship triplets output in this round.
[0056] S4, reverse translation learning iteration: The Knowledge Base of the reverse translation learning framework used in the present invention includes two parts: the rich language common sense rules contained in the llama3 model, and the external knowledge provided by Wikipedia. According to the rules provided by the Knowledge Base, the triples output in this round are corrected, and after eliminating internal contradictions, they are used as pseudo labels for model training and iteration. Until the output of the model can no longer be found to be internally contradictory by the Knowledge Base.
[0057] Specifically, Figure 2 As shown, the triples obtained in S3 are restored to simple sentences, that is, "entity e1" and "entity e2" have "relationship r". The above sentences are transferred to the llama3 model to query its internal logical contradictions. Then query the Wikipedia knowledge base through the wikipedia-api interface to query its common sense errors. Find the minimum change that makes the triple not contradictory to the knowledge base within the scope allowed by the original sentence, and update the triple. According to the splicing rules in S3, a correct triple must correspond to three correct bigrams, so bigrams can be restored from the triples as pseudo labels for training.
[0058] Use pseudo labels to train and iterate ALBERT, and repeat S2 to S4 until no modifications are returned in S4.
[0059] The loss function is defined as:
[0060]
[0061] Where L uses focal loss:
[0062] L=-α t (1-f t ) γ log(f t )(8)
[0063]
[0064] Among them, α is a balancing factor used to balance the ratio of positive and negative samples. γ is an adjustment factor used to adjust the model's attention to rare samples. f is the model's predicted probability value for a certain set of relationships.
[0065] S5, obtain the final relationship prediction: After iterations of back-translation learning, the ALBERT model that is consistent with the Knowledge Base is obtained, and then the final relationship prediction is obtained through S1, S2, and S3.
[0066] After the final model is obtained through iteration, the prediction of each relationship is output by the model through S1~S3. Figure 2 shown.
[0067] The performance of our model is demonstrated by the values of Matthews correlation coefficient (MCC), precision, recall, accuracy, and Macro-F1. The formula for calculating the Matthews correlation coefficient is as follows:
[0068]
[0069] In the above formula, x i and i Represent the predicted value and label value respectively. When calculating accuracy, precision, recall and macro F1 value, the predicted value x i becomes 0 or 1. Taking 0.5 as the dividing line, the predicted value greater than 0.5 becomes 1, and the predicted value less than 0.5 becomes 0. The calculation formulas for accuracy, precision, recall and macro F1 are as follows:
[0070]
[0071] In the above equation, the definitions of TP, FP, TN, and FN are as follows:
[0072]
[0073] Corresponding to the entity relationship extraction method in the above embodiment, the present invention further discloses a zero-sample entity relationship extraction system based on large language model information enhancement and back-translation learning, comprising:
[0074] The text feature acquisition module obtains the original text and a set of relations that the text is concerned with, and uses a large language model to convert the relational words into text features with explanatory information;
[0075] The attention feature acquisition module uses the pre-trained language model to encode the acquired text features and obtain text attention features;
[0076] An initial triplet acquisition module, for the text attention feature, uses cross attention to calculate the attention scores between entities and entities, and between entities and relational words in the text, and selects candidate bigrams according to the scores, and concatenates the candidate bigrams to obtain initial triplet;
[0077] The back-translation learning iteration module modifies the initial triples according to the rules in the knowledge base to obtain pseudo labels; the pseudo labels are used to iterate the back-translation learning of the results of the pre-trained language model until the output of the pre-trained language model no longer conflicts with the knowledge base;
[0078] The relation prediction module generates relation triplets using the pre-trained language model after back-translation learning iterations.
[0079] As for the entity relationship extraction system of the embodiment of the present invention, since it corresponds to the entity relationship extraction method in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the entity relationship extraction method in the above embodiment, which will not be described in detail here.
[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A zero-shot entity relationship extraction method based on large language model information enhancement and back-translation learning, characterized in that: include: Get the original text and a set of relations that the text is concerned with, and use a large language model to convert the relation words into text features with explanatory information; Use the pre-trained language model to encode the obtained text features and obtain text attention features; According to the text attention feature, cross attention is used to calculate the attention scores between entities and entities, and between entities and relational words in the text, and candidate bigrams are screened out according to the scores, and the candidate bigrams are concatenated to obtain initial triplets; Modify the initial triples according to the rules in the knowledge base of the back-translation learning model to obtain pseudo labels; Use pseudo labels to iterate the back-translation learning of the results of the pre-trained language model until the output of the pre-trained language model no longer conflicts with the knowledge base; Generate relation triplets using the pre-trained language model after back-translation learning iterations.
2. According to claim 1, a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning is characterized in that: The pre-trained language model is an ALBERT model.
3. According to claim 1, a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning is characterized in that: The knowledge base is constructed based on the language common sense of the large language model and the knowledge common sense provided by Wikipedia.
4. According to claim 1, a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning is characterized in that: The dimension of relation features ranges from 128 to 512.
5. According to claim 1, a zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning is characterized in that: The training loss function is the focal loss function.
6. The zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning according to claim 1 is characterized in that: The large language model is a llama-3 model.
7. The zero-sample entity relationship extraction method based on large language model information enhancement and back-translation learning according to claim 6 is characterized in that: The llama-3 model is used to transform relational words into text features with explanatory information, including: The text representation of each sentence is E = {E [CLS] ,E1,E2,...,E L ,E [SEP] }, each relational descriptive statement is represented by R = {R [CLS] ,R1,R2,...,R M ,R [SEP] }, [CLS] and [SEP] are the head and tail tags of the sentence, used to mark the starting and ending positions; The tokenizer converts text input into numeric input T = {T [CLS] ,T1,T2,...,T L ,T [SEP] }, and transform the relational descriptive statement into Tr={Tr [CLS] ,Tr1,Tr2,...,Tr M ,Tr [SEP] },in L is the length of the original text, and M is the length of the descriptive sentence.
8. A zero-shot entity relationship extraction system based on large language model information enhancement and back-translation learning, characterized in that: include: The text feature acquisition module obtains the original text and a set of relations that the text is concerned with, and uses a large language model to convert the relational words into text features with explanatory information; The attention feature acquisition module uses the pre-trained language model to encode the acquired text features and obtain text attention features; An initial triplet acquisition module, for the text attention feature, uses cross attention to calculate the attention scores between entities and entities, and between entities and relational words in the text, and selects candidate bigrams according to the scores, and concatenates the candidate bigrams to obtain initial triplet; The back-translation learning iterative module modifies the initial triples according to the rules in the knowledge base to obtain pseudo labels; Use pseudo labels to iterate the back-translation learning of the results of the pre-trained language model until the output of the pre-trained language model no longer conflicts with the knowledge base; The relation prediction module generates relation triplets using the pre-trained language model after back-translation learning iterations.