Information extraction algorithm based on large model extension label enhancement

By using an information extraction algorithm enhanced by expanding labels based on a large model, the problem of spurious associations in information extraction tasks of large language models is solved. By expanding and verifying relation types, the performance of the model in relation extraction, named entity recognition and event detection tasks is enhanced.

CN119005322AActive Publication Date: 2024-11-22EAST CHINA UNIV OF SCI & TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411017638.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

Existing large-scale language models exhibit spurious associations in information extraction tasks, particularly in zero-sample or few-sample scenarios where they struggle to effectively perform relation extraction, named entity recognition, and event detection tasks.

Method used

An information extraction algorithm based on large model expanded labels is adopted. It goes through three stages: forward label expansion, backward label verification, and expanded label application. It expands predefined relations using the training dataset, verifies the consistency of entity pairs using the verification dataset, and integrates the retained expanded relations into the test set to enhance model performance.

Benefits of technology

It improves the performance of large models in information extraction tasks, reduces spurious associations, and enhances the effectiveness of models with limited data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005322A_ABST
    Figure CN119005322A_ABST
Patent Text Reader

Abstract

The invention discloses an information extraction algorithm based on large model extension label enhancement. The method comprises the following steps of: by taking relationship extraction as an example, firstly, expanding a predefined relationship type by using a text in a training data set and a corresponding head-tail entity pair; secondly, extracting a head-tail entity pair from a text in the verification data set by using an extended relationship, and if the extracted head-tail entity pair is consistent with a real entity pair, reserving a corresponding extended relationship; and finally, integrating the reserved extension relationship into a predefined relationship set, and executing a relationship extraction task on a test set by utilizing the large model, so as to enhance the performance of the model on the relationship extraction task. The method is suitable for carrying out an information extraction task in the field of natural language processing by utilizing the large model, a false association phenomenon existing when the large model executes the information extraction task is proved by designing an information extraction algorithm based on large model extension label enhancement, and the performance of the large model is further improved by utilizing an extension type label.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, specifically to the technical field of text information extraction, and more specifically to providing an information extraction algorithm based on large model extended label enhancement. Background Art

[0002] Information extraction is an important natural language processing task that aims to extract predefined types of information from unstructured text sources. Information extraction tasks usually include relation extraction, named entity recognition, and event detection. Information extraction often faces challenges in scenarios with limited data, such as zero-sample or few-sample settings, where traditional models may have difficulty performing the task effectively.

[0003] Recently, large language models such as ChatGPT have become a fundamental pillar in the field of natural language processing, which can achieve high performance with only a few example instructions without tuning a large number of parameters. Currently, the methods of applying large models to information extraction tasks can be roughly divided into four categories: (1) directly using large models for reasoning; (2) merging large language models with small language models; (3) using small language models to perform knowledge distillation from large models; and (4) fine-tuning large models for instructions.

[0004] However, when large models such as ChatGPT are applied to information extraction tasks, there is a phenomenon of "false association". For example, in relation extraction tasks, these models can usually correctly identify entity pairs even if the presented relation types are semantically unrelated to the predefined types. This phenomenon exists on both Chinese and English datasets. Similarly, this "false association" phenomenon also exists on Chinese and English datasets for named entity recognition and event detection tasks. Although the current trend is to use large models to perform information extraction tasks, there is still room for improvement in their performance.

[0005] In summary, this interdisciplinary field urgently needs to design a new information extraction algorithm to solve the above problems. Summary of the invention

[0006] In view of this, the present invention proposes an information extraction algorithm based on large model extended label enhancement. The algorithm includes three basic stages: forward label expansion, backward label verification and application of extended labels. In order to provide a more comprehensive explanation, the relationship extraction task is taken as an example: in stage one, the predefined relationship types are expanded using the text and the corresponding head and tail entity pairs in the training data set; in stage two, the head and tail entity pairs are extracted from the text in the verification data set using the extended relations in stage one, and if the extracted head and tail entity pairs are consistent with the real entity pairs, the corresponding extended relations are retained; in stage three, the retained extended relations are integrated into the predefined relationship set, and the relationship extraction task is performed on the test set using the large model, so as to enhance the performance of the model in the relationship extraction task.

[0007] The present invention takes relation extraction as an example, and the specific steps are:

[0008] S1. Use a large model to expand predefined relations from the text and corresponding head-tail entity pairs in a given training set;

[0009] S2, use the extended relationship in S1 to extract the head and tail entity pairs from the text in the verification data set. If the extracted entity pairs are consistent with the standard entity pairs, the corresponding extended relationship is retained;

[0010] S3. Integrate the retained extended relations into the predefined relation set and apply it on the test set to enhance the performance of the large model on the relation extraction task.

[0011] Further, step S1 includes:

[0012] S11. Select all samples with relation r from the training set. For each sample, connect the sentence with the corresponding head and tail entity pairs. Then input these combinations into the large model to prompt the large model to generate new relations, so as to obtain the extended set R′ of each relation r. r ;

[0013] S12. Considering that the relations extended by different r∈R may be the same, we eliminate the extended sets R′ of each r∈R. r The repeated relationships in the collection ensure the uniqueness between the collections.

[0014] Further, step S2 includes:

[0015] S21. Select all samples with relation r from the validation set. For each r′, connect the sentence in each selected sample with r′ and input it into the large model, prompting the large model to generate head-tail entity pairs.

[0016] S22: The ratio of the overlap counts of the entity pairs obtained in S21 and the standard entity pairs to the standard entity pair counts is used as the accuracy rate, and based on this, the F1 scores of the extended relations r′ of all samples related to r are calculated. If the F1 score is zero, then the extended relations r′ is calculated from R′. r Delete the extended relation r′.

[0017] Further, step S3 includes:

[0018] S31, from R′ r Select the top k valid extended relations with the highest F1 score for each r∈R, and merge R with the selected extended relations among all relations in R;

[0019] S32. Input each sentence in the test set and the merged relation set into the big model, and design a thinking chain to guide the big model to generate triples through the following steps: first, identify a set of head entity pairs and tail entity pairs from the sentence; second, select a relation from the provided relation set for each pair to form a triple; finally, evaluate the rationality of each triple, and only retain triples that are judged to be reasonable.

[0020] After adopting the above strategy, the positive effects of the present invention are:

[0021] In the field of information extraction, the present invention aims to solve the problem that traditional models are difficult to effectively perform information extraction tasks under limited data scenarios. This paper proposes a novel information extraction algorithm based on large model extended label enhancement, which effectively proves the "false correlation" phenomenon that exists when large models perform information extraction tasks, and further uses extended type labels to improve the performance of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the extended tag construction method and application provided by the present invention;

[0023] Figure 2 This is a flow chart of the information extraction algorithm based on large model extended label enhancement of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more obvious and understandable, the present invention is further described in detail below in conjunction with embodiments and accompanying drawings of the embodiments.

[0025] See also Figure 2 , Figure 2It is a schematic diagram of the principle of the information extraction algorithm based on the large model extended label enhancement of the present invention. The method includes (taking relation extraction as an example): first, using the text and corresponding head and tail entity pairs in the training data set to expand the predefined relation type; second, using the extended relationship to extract the head and tail entity pairs from the text in the verification data set, if the extracted head and tail entity pairs are consistent with the real entity pairs, the corresponding extended relationship is retained; finally, the retained extended relationship is integrated into the predefined relationship set, and the large model is used to perform the relation extraction task on the test set, so as to enhance the performance of the model in the relation extraction task.

[0026] The specific steps include:

[0027] S1. Use the large model to extend the existing relation r from the text and entity pairs (h, t) in the given training set to create an extended relation r′. After performing this extension on all samples related to relation r, the extended relation set R′ is obtained. r ;

[0028] S11. Select all samples with relation r from the training set. For each sample, connect the sentence with the corresponding head and tail entity pairs. Then input these combinations into the large model to prompt the large model to generate new relations, so as to obtain the extended set R′ of each relation r. r In order to improve the generation effect of the large model in this process, the role definition, instructions and examples are designed as prompts in the large model input. The specific contents are as follows:

[0029] 1. Prompt on Chinese dataset:

[0030] (1) Role definition: “You are currently a senior relationship generation expert.”

[0031] (2) Instructions: “Your task is to generate relationship types for the head and tail entity pairs in the given text, and follow the following rules when generating:

[0032] 1) Using head-tail entity pairs, such as (ischemic stroke, MRI);

[0033] 2) Combine the context of a given text;

[0034] 3) Generate possible relationship types. ”

[0035] (3) Example: “Here is an example of an instruction to help you understand the task:

[0036] Input: 'Encephalitis@ should be treated with plasma exchange in consultation with a hematologist. Encephalitis@ is usually treated with 4-5 plasma exchanges every other day. 'Head-tail entity pair: [(encephalitis, plasma exchange)]

[0037] Output: [Assistive Treatment]"

[0038] 2. Prompts on the English dataset:

[0039] (1) Role definition: "You are currently a senior relation generation expert."

[0040] (2) Instruction: "Your objective is to generate relation types for head-tail entity pairs within a given text, adhering to the following rules:

[0041] 1) Utilize head-tail entity pairs, e.g., (noisy-channel architecture, transformation model).

[0042] 2) Incorporate the context of the given text.

[0043] 3) Generate relation types."

[0044] (3) Example: "The following is an example of instruction to help you understand the task:

[0045] Input: ‘We present a text mining method for finding synonymous expressions based on the distributional hypothesis in a set of coherent corpora.’ Head-tail entity pair: (text mining method, synonymous expressions)

[0046] Output: [USED-FOR]"

[0047] S12. The extended set R′ of the specific relation r obtained according to S11 r, considering that the relations extended by different r∈R may be the same, the repeated relations appearing in each extended set of each r∈R are eliminated to ensure the uniqueness between the sets.

[0048] S2. Select samples with relation r from the validation set. For each selected sample and each r′∈R′ r , input the sample together with r′ into the large model and prompt it to generate head-tail entity pairs from the sample. If the generated entity pairs cannot be aligned with the real entity pairs, r Delete r′ from the r If r remains non-empty and the relations in the set are completely unrelated to r in semantics, the phenomenon of “false association” holds;

[0049] S21. Select all samples with relationship r from the validation set. For each r′, connect the sentences in each selected sample with r′ and input them into the large model, prompting the large model to generate head-tail entity pairs. In order to further enhance the effect of the model, in addition to sentences and extended relations, a thought chain process is introduced into the input of the large model, that is, first require the model to generate head-tail entity pairs, then combine this output with the extended relationship, and finally require the model to evaluate whether the two entities show this extended relationship, and try to ensure that the model generates results that are consistent with real-world facts. The specific prompts are as follows:

[0050] 1. Prompt on Chinese dataset:

[0051] (1) Role definition: “You are currently a senior relation extraction expert.”

[0052] (2) Instructions: “Your task is to extract head and tail entity pairs that match the relationship type given the text and relationship. When generating, follow the following rules:

[0053] 1) Based on the given relationship and the context of the given text, extract the head and tail entity pairs that may have the given relationship;

[0054] 2) Generate judgment sentences based on the relationship list and keywords, and check whether each judgment sentence is correct, and only output "yes" or "no";

[0055] 3) Generate a list of relation triples based on the judgment sentence, (head entity, relation, tail entity), where the relation must be a given relation. "

[0056] (3) Example of a thought chain: “Here is an example of a thought chain to help you generate results step by step:

[0057] Input: 'Encephalitis@ should be treated with plasma exchange after consultation with a hematologist. Encephalitis@ is usually treated with 4-5 plasma exchanges every other day. 'Relationship: [Adjunctive therapy]

[0058] Head and tail entity pair: [(encephalitis, plasma exchange)]

[0059] Answer: 'Is plasma exchange an adjunctive treatment for encephalitis? 'Yes.

[0060] Generate a list of relation triplets:

[0061] ``

[0062] (Encephalitis, adjuvant therapy, plasma exchange)

[0063] ``”

[0064] 2. Prompt on English dataset:

[0065] (1) Role definition: “You are currently a senior expert in relation extraction.”

[0066] (2) Instruction: "Your objective is to extract head-tail entity pairs given atext and the specified relation, adhering to the following rules:

[0067] 1)Utilize the context of the given text with the specified relationto extract head-tail entity pairs that represent the relation.

[0068] 2)Verify the yes-no question sentence formed by concatenating the specified relation and the extracted head-tail entity pairs, and output only'yes'or'no'.

[0069] 3)Generate a list of triplet(head entity,relation,tail entity)based on the answer of yes-no question,where the relation must be the specifiedrelation.”

[0070] (3) Example of a chain of thought: “The following is an example of a chain of thought that guides you step by step to generate the results.

[0071] Input:'We propose and analyze a block minimization framework for datalarger than the memory size.'Relation:COMPARE

[0072] Head-tail entity pair:[(data,memory size)]

[0073] Answer:Is the relation between'data'and'memory size''COMPARE'? Yes

[0074] Generate a list of triplet:

[0075] ``

[0076] (data,COMPARE,memory size)

[0077] ``”

[0078] S22. If there is an overlap between the output of the model and the ground truth, the output of the model is considered correct. Here, the accuracy is defined as the ratio of the intersection count to the ground truth count. This method calculates the F1 score of the extended relation r′ of all samples related to r. If the F1 score is zero, then the extended relation r′ is removed from R. ′ r Delete the extended relation r′.

[0079] S3. In the presence of this phenomenon, the extended relationship set (m rRepresents the number of relations in R) into the predefined relation set R, create a new relation set R″, and use R″ to perform the relation extraction task on the test set to enhance the performance of the model in the relation extraction task.

[0080] S31, from R ′ r Select the top k valid extended relations with the highest F1 score for each r∈R, and merge R with the selected extended relations among all relations in R;

[0081] S32. Input each sentence and the merged relation set in the test set into the large model, and design role definitions, instructions, and examples to enhance the input, so as to improve the model performance. At the same time, design a thinking chain in the instructions to further guide the large model, that is, require the model to generate triples according to the following steps: first, identify a set of head entity pairs and tail entity pairs from the sentence; second, select a relation for each pair from the provided relation set to form a triple; finally, evaluate the rationality of each triple, and only retain triples that are judged to be reasonable. The specific prompts are as follows:

[0082] 1. Prompt on Chinese dataset:

[0083] (1) Role definition: “You are currently a senior relation extraction expert.”

[0084] (2) Instructions: “Your task is to extract triples from the given text and relation list, following the following rules during the extraction process:

[0085] 1) Generate a list of keyword pairs from a given text;

[0086] 2) Extract potential relationships of keyword pairs from a given relationship list;

[0087] 3) Generate judgment sentences based on the relationship list and keywords, and check whether each judgment sentence is correct, and only output "yes" or "no";

[0088] 4) Generate a list of triples (head entity, relation, tail entity) based on the answer to the judgment sentence, where the relation must be in the given relation list. "

[0089] (3) Example of a thought chain: “Here is an example of a thought chain to help you generate results step by step:

[0090] Type: 'Bacterial meningitis@ In older people with meningitis, confusion or altered mental status is usually the only sign present.'

[0091] Intermediate step keyword pair: [(bacterial meningitis, confusion), (bacterial meningitis, altered mental status)]

[0092] Intermediate step relationship: [Clinical manifestations]

[0093] answer:

[0094] Is confusion a clinical manifestation of bacterial meningitis?

[0095] Is altered mental status a clinical manifestation of bacterial meningitis? Yes

[0096] Generate a list of relation triplets:

[0097] ``

[0098] (Bacterial meningitis, clinical manifestations, confusion)

[0099] (Bacterial meningitis, clinical manifestations, altered mental status)

[0100] ``”

[0101] 2. Prompt on English dataset:

[0102] (1) Role definition: “You are currently a senior expert in relation extraction.”

[0103] (2) Instruction: "Your objective is to extract triplets given a text and alist of relation, adhering to the following rules:

[0104] 1)Generate a keyword pair list from the given text

[0105] 2)Extract potential relations for keyword pairs from the givenrelation list

[0106] 3)Verify the yes-no question sentence formed by concatenating the potential relations and the extracted keyword pairs, and output only'yes'or'no'.

[0107] 4) Generate a list of triplet (head entity, relation, tail entity) based on the answer of yes - no question, where the relation must be in the given relation list {extend_label}.

[0108] (3) Chain of thought example: “The following is an example of a chain of thought that guides you step by step to generate the results.

[0109] Input: We present a text mining method for finding synonymous expressions based on the distributional hypothesis in a set of corpora.

[0110] Intermediate_keyword pair: [(text mining method, synonymous expressions), (distributional hypothesis, text mining method)]

[0111] Intermediate_relation: [USED - FOR, EVALUATE - FOR]

[0112] Answer:

[0113] Is the relation between ‘text mining method’ and ‘synonymous expressions’ the ‘USED - FOR’?Yes

[0114] Is the relation between ‘distributional hypothesis’ and ‘text mining method’ the ‘USED - FOR’?Yes

[0115] Is the relation between'text mining method'and'synonymousexpressions'the'EVALUATE-FOR'? No

[0116] Is the relation between'distributional hypothesis'and'text miningmethod'the'EVALUATE-FOR'? No

[0117] Generate a list of triplet:

[0118] ``

[0119] (text mining method,USED-FOR,synonymous expressions)

[0120] (distributional hypothesis,USED-FOR,text mining method)

[0121] ``”

[0122] In the above, the specific embodiments of the present invention are described with reference to the accompanying drawings. However, those skilled in the art will appreciate that various changes and substitutions may be made to the specific embodiments of the present invention without departing from the spirit and scope of the present invention. These changes and substitutions are all within the scope defined by the claims of the present invention.

Claims

1. An information extraction algorithm based on large model extended label enhancement, Features The following steps are involved: S1. For the relation extraction subtask in information extraction, a large model is used to expand the predefined relations from the text and corresponding head and tail entity pairs in a given training set; S2, use the extended relationship in S1 to extract the head and tail entity pairs from the text in the verification data set. If the extracted entity pairs are consistent with the standard entity pairs, the corresponding extended relationship is retained; S3. Integrate the retained extended relations into the predefined relation set and apply it on the test set to enhance the performance of the large model on the relation extraction task. S4. For the named entity recognition subtask in information extraction, a large model is used to expand the predefined entity types from the text and corresponding entity spans in a given training set; S5, using the extended entity in S4 to extract entity spans from the text in the validation dataset, if the extracted entity span is consistent with the standard entity span, the corresponding extended entity is retained; S6. Integrate the retained extended entities into the predefined entity set and apply it on the test set to enhance the performance of the large model on the entity recognition task. S7. For the event detection subtask in information extraction, use the large model to expand the predefined event types from the text and corresponding event trigger words in the given training set; S8, using the extended events in S7 to extract event trigger words from the text in the verification data set, if the extracted trigger words are consistent with the standard trigger words, then retain the corresponding extended events; S9. Integrate the retained extended events into the predefined event set and apply it on the test set to enhance the performance of the large model on the event detection task.

2. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S1 specifically includes: S11. Select all samples with relation r from the training set. For each sample, connect the sentence with the corresponding head-tail entity pair. Then input these combinations into the large model to prompt the large model to generate new relations, so as to obtain an extended set of each relation r. S12. Delete the duplicate relations that appear in each expansion set of each r to ensure the uniqueness between the sets.

3. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S2 specifically includes: S21. Select all samples with relation r from the validation set. For each r′, connect the sentence in each selected sample with r′ and input it into the large model, prompting the large model to generate head-tail entity pairs. S22. The ratio of the overlapping counts of the head-tail entity pairs and the golden head-tail entity pairs obtained in S21 to the counts of the golden head-tail entity pairs is taken as the accuracy rate, and based on this, the F1 scores of the extended relationship r′ of all samples related to r are calculated. If the F1 score is zero, the extended relationship r′ is deleted therefrom.

4. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S3 specifically includes: S31, selecting the first k valid extended relations with the highest F1 score, and merging the extended relations with the original predefined relation set; S32. Input each sentence and the merged relation set in the test set into the large model, and design role definitions, instructions, and examples to enhance the input, so as to improve the model performance. At the same time, design a thinking chain in the instructions to further guide the large model, that is, require the model to generate triples in the following steps: first, identify a set of head and tail entity pairs from the sentence; second, select a relation for each pair from the provided relation set to form a triple; finally, evaluate the rationality of each triple, and only retain triples that are judged to be reasonable.

5. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S4 specifically includes: S41. Select all samples with entity type e from the training set. For each sample, connect the sentence with the corresponding entity span, and then input these combinations into the large model to prompt the large model to generate new entities, so as to obtain an extended set of each entity type e. S42. Delete the duplicate relationships that appear in each expansion set of each e to ensure the uniqueness between the sets.

6. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S5 specifically includes: S51, select all samples with entity type e from the validation set, for each extended entity type e′, connect the sentence in each selected sample with e′ and input it into the large model, prompting the large model to generate entity span; S52. The overlapping count of the entity span and the golden entity span obtained in S51 is taken as the ratio of the golden entity span count, and based on this, the F1 score of the extended entity e′ of all samples related to the entity type e is calculated. If the F1 score is zero, the extended entity type e′ is deleted therefrom.

7. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S6 specifically includes: S61, selecting the first k valid extended entity types each having the highest F1 score, and merging the extended entity types with the original predefined entity type set; S62. Input each sentence and the merged entity type set in the test set into the big model, and design role definitions, instructions, and examples to enhance the input, so as to improve the model performance. At the same time, design a thinking chain in the instructions to further guide the big model, that is, require the model to generate entity pairs according to the following steps: first, identify entity spans from sentences; second, select an entity type for each span from the provided entity set; finally, evaluate the rationality of each entity and the corresponding entity type, and only retain the entities and their entity types that are judged to be reasonable.

8. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S7 specifically includes: S71, select all samples with event type d from the training set, for each sample, connect the sentence with the corresponding event trigger word, and then input these combinations into the big model, prompting the big model to generate new event types, so as to obtain an extended event type set for each event type d; S72. Delete the repeated event types appearing in each extended set of each d to ensure the uniqueness between the sets.

9. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S8 specifically includes: S81, select all samples with event d from the validation set, and for each extended event type d′, connect the sentence in each selected sample with d′ and input it into the large model, prompting the large model to generate event trigger words; S82. The ratio of the overlapping counts of the event trigger words and the golden event trigger words obtained in S81 to the golden event trigger word count is taken as the accuracy rate, and based on this, the F1 scores of the extended event type d′ of all samples related to the event type d are calculated. If the F1 score is zero, the extended event type d′ is deleted therefrom.

10. According to claim 1, an information extraction algorithm based on large model extended label enhancement, It is characterized in that The step S9 specifically includes: S91, selecting the first k valid extended event types each having the highest F1 score, and merging the extended event types with the original predefined event type set; S92. Input each sentence and the combined event type set in the test set into the big model, and design role definitions, instructions, and examples to enhance the input, so as to improve the model performance. At the same time, design a thinking chain in the instructions to further guide the big model, that is, require the model to generate event trigger words according to the following steps: first, identify event trigger words from sentences; second, assign an event type to the event trigger word from the provided event type set; finally, evaluate the rationality of each event trigger word and its event type, and only retain the event trigger words and their event types that are judged to be reasonable.

Citation Information

Patent Citations

  • Method for training relationship extraction model and extracting nested named entity relationship

    CN112487196A

  • Knowledge graph and large language model-based relationship extraction method and system and medium

    CN117494806A

  • Open information extraction method based on dual tasks

    CN117875419A