An information extraction algorithm based on large model expansion label enhancement

Abstract

The application discloses an information extraction algorithm based on large model extended label enhancement. The application comprises the following steps: taking relation extraction as an example, first, using the text in the training data set and the corresponding head-tail entity pair to expand the predefined relation type; second, using the expanded relation to extract the head-tail entity pair from the text in the verification data set, if the extracted head-tail entity pair is consistent with the real entity pair, the corresponding expanded relation is retained; finally, integrating the retained expanded relation into the predefined relation set, using the large model to perform the relation extraction task on the test set, thereby enhancing the performance of the model on the relation extraction task. The application is suitable for information extraction tasks in the field of natural language processing using large models, by designing an information extraction algorithm based on large model extended label enhancement, it is proved that the large model has the phenomenon of "false association" when performing information extraction tasks, and further using the extended type label to improve the performance of the large model.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically to the field of text information extraction technology, and more specifically, to providing an information extraction algorithm based on large model extended label enhancement. Background Technology

[0002] Information extraction is an important natural language processing task that aims to extract predefined types of information from unstructured text sources. Information extraction tasks typically include relation extraction, named entity recognition, and event detection. Information extraction often faces challenges in scenarios with limited data, such as zero or few-sample settings, where traditional models may struggle to perform the task effectively.

[0003] Recently, large language models like ChatGPT have become a fundamental pillar in the field of natural language processing, demonstrating high performance with just a few example instructions and without the need to adjust a large number of parameters. Currently, methods for applying large models to information extraction tasks can be roughly divided into four categories: (1) directly using large models for reasoning; (2) merging large and small language models; (3) utilizing small language models that have undergone knowledge distillation from large models; and (4) fine-tuning large models with instructions.

[0004] However, when large models like ChatGPT are applied to information extraction tasks, a phenomenon known as "spurious associations" emerges. For example, in relation extraction tasks, these models often correctly identify entity pairs even when the presented relation types are semantically unrelated to predefined types. This phenomenon exists on both Chinese and English datasets. Similarly, this "spurious association" phenomenon also exists on Chinese and English datasets for named entity recognition and event detection tasks. Although the current trend is to use large models to perform information extraction tasks, there is still room for improvement in their performance.

[0005] In conclusion, this interdisciplinary field urgently needs to design a new information extraction algorithm to solve the above problems. Summary of the Invention

[0006] In view of this, this invention proposes an information extraction algorithm based on large model-based expanded label enhancement. The algorithm comprises three basic stages: forward label expansion, backward label verification, and application of expanded labels. To provide a more comprehensive explanation, a relation extraction task is used as an example: Stage 1, using text and corresponding head-and-tail entity pairs from the training dataset to expand predefined relation types; Stage 2, using the expanded relations from Stage 1 to extract head-and-tail entity pairs from the text in the verification dataset; if the extracted head-and-tail entity pairs match the real entity pairs, the corresponding expanded relations are retained; Stage 3, the retained expanded relations are integrated into a predefined relation set, and the large model is used to perform the relation extraction task on the test set, thereby enhancing the model's performance on the relation extraction task.

[0007] Taking relation extraction as an example, the specific steps of this invention are as follows:

[0008] S1. Use a large model to extend predefined relations from the text and corresponding head and tail entity pairs in a given training set;

[0009] S2. Use the extended relations in S1 to extract head and tail entity pairs from the text in the validation dataset. If the extracted entity pairs are consistent with the standard entity pairs, retain the corresponding extended relations.

[0010] S3. Integrate the retained extended relations into a predefined relation set and apply it to the test set to enhance the performance of large models on relation extraction tasks.

[0011] Further, step S1 includes:

[0012] S11. Select all samples with relation r from the training set. For each sample, concatenate the sentence with the corresponding head and tail entity pairs, and then input these combinations into the large model, prompting the large model to generate new relations, thereby obtaining an expanded set R′ for each relation r. r ;

[0013] S12. Considering that the extended relations for different r∈R may be the same, therefore, eliminate each extended set R′ for each r∈R. r Repeated relationships appearing in the data ensure the uniqueness of sets.

[0014] Further, step S2 includes:

[0015] S21. Select all samples with relation r from the validation set. For each r′, concatenate the sentence in each selected sample with r′ and input it into the large model. Prompt the large model to generate head and tail entity pairs.

[0016] S22. The ratio of the overlap count of entity pairs and standard entities obtained in S21 to the standard entity pair count is used as the accuracy. Based on this, the F1 score of the extended relation r′ for all samples related to r is calculated. If the F1 score is zero, then from R′... r Delete the extended relation r′.

[0017] Further, step S3 includes:

[0018] S31, from R′ r Select the top k valid extended relations with the highest F1 score for each r∈R, and merge R with the selected extended relations from all relations in R;

[0019] S32. Input each sentence and the merged relation set from the test set into the large model, and design a thought chain to guide the large model to generate triples through the following steps: First, identify a set of head entity pairs and tail entity pairs from the sentence; second, select a relation for each pair from the provided relation set to form a triple; finally, evaluate the reasonableness of each triple and retain only the triples that are judged to be reasonable.

[0020] After adopting the above strategy, the positive effects of the present invention are:

[0021] This invention addresses the challenge of traditional models failing to effectively perform information extraction tasks in scenarios with limited data in the field of information extraction. It proposes a novel information extraction algorithm based on large model extended label enhancement, which effectively demonstrates the "spurious association" phenomenon that exists when large models perform information extraction tasks. Furthermore, it utilizes extended type labels to improve the performance of large models. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the extended tag construction method and its application provided by the present invention;

[0023] Figure 2 This is a flowchart illustrating the information extraction algorithm based on large model expanded label enhancement according to the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0025] Please see Figure 2 , Figure 2This is a schematic diagram illustrating the principle of the information extraction algorithm based on large model expansion and label enhancement of this invention. The method includes (taking relation extraction as an example): First, using text and corresponding head-and-tail entity pairs from the training dataset to expand predefined relation types; second, using the expanded relations to extract head-and-tail entity pairs from the text in the validation dataset; if the extracted head-and-tail entity pairs match the real entity pairs, the corresponding expanded relations are retained; finally, the retained expanded relations are integrated into a predefined relation set, and the large model is used to perform the relation extraction task on the test set, thereby enhancing the model's performance on the relation extraction task.

[0026] The specific steps include:

[0027] S1. Using a large model, extend the existing relation r from the text and entity pairs (h,t) in the given training set to create an extended relation r′. After performing this extension on all samples related to relation r, the extended relation set R′ is obtained. r ;

[0028] S11. Select all samples with relation r from the training set. For each sample, concatenate the sentence with the corresponding head and tail entity pairs, and then input these combinations into the large model, prompting the large model to generate new relations, thereby obtaining an expanded set R′ for each relation r. r To improve the generation quality of large models during this process, a Prompt containing character definitions, instructions, and examples as input to the large model was designed. The specific content is as follows:

[0029] 1. Prompt on the Chinese dataset:

[0030] (1) Role definition: "Currently, you are a senior relationship generation expert."

[0031] (2) Instruction: "Your task is to generate relation types for the head and tail entity pairs in the given text, and follow these rules when generating:"

[0032] 1) Utilize head-to-tail pairs, for example (ischemic stroke, MRI);

[0033] 2) Combine the context of the given text;

[0034] 3) Generate possible relation types.

[0035] (3) Example: "Here is an example of an instruction to help you understand the task:

[0036] Input: 'Encephalitis@ Plasma exchange should be performed after consulting a hematologist. Encephalitis@ Usually, 4-5 plasma exchanges are performed every other day.' Head-tail entity pair: [(Encephalitis, Plasma Exchange)]

[0037] Output: "[Support Healing]"

[0038] 2. Prompts on the English dataset:

[0039] (1) Role definition: “You are currently a senior relation generation expert.”

[0040] (2) Instruction: “Your objective is to generate relation types for head - tail entity pairs within a given text, adhering to the following rules:

[0041] 1) Utilize head - tail entity pairs, e.g., (noisy - channel architecture, transformation model).

[0042] 2) Incorporate the context of the given text.

[0043] 3) Generate relation types.”

[0044] (3) Example: “The following is an example of instruction to help you understand the task:

[0045] Input: ‘We present a text mining method for finding synonymous expressions based on the distributional hypothesis in a set of coherent corpora.’ Head - tail entity pair: (text mining method, synonymous expressions)

[0046] Output: [USED - FOR]”

[0047] S12. The extended set R′ of the specific relation r obtained according to S11 rConsidering that the relations extended by different r∈R may be the same, we eliminate the duplicate relations that appear in each extension set of each r∈R to ensure the uniqueness between sets.

[0048] S2. Select samples with relation r from the validation set. For each selected sample and each r′∈R′ r The sample and r′ are input into the large model, and it is prompted to generate head and tail entity pairs from the sample. If the generated entity pairs cannot be aligned with the real entity pairs, then from R′ r Delete r′. If R′ r If the set remains non-empty and the relations within the set are semantically completely unrelated to r, then the phenomenon of "spurious association" holds true.

[0049] S21. Select all samples with relation r from the validation set. For each r′, concatenate the sentence in each selected sample with r′ and input it into the large model, prompting the large model to generate head-and-tail entity pairs. To further enhance the model's performance, in addition to sentences and extended relations, a thought chain process is introduced into the input of the large model. That is, the model is first asked to generate head-and-tail entity pairs, then this output is combined with the extended relation, and finally the model is asked to evaluate whether the two entities exhibit this extended relation, so as to ensure that the model generates results that are consistent with real-world facts. The specific prompt is as follows:

[0050] 1. Prompt on the Chinese dataset:

[0051] (1) Role definition: "Currently, you are a senior relationship extraction expert."

[0052] (2) Instruction: "Your task is to extract head and tail entity pairs that conform to the relation type, given text and relations, and follow these rules when generating:"

[0053] 1) Based on the given relationship, and in conjunction with the context of the given text, extract the head and tail entity pairs that may have the given relationship;

[0054] 2) Based on the relation list and keywords, generate judgment sentences and check whether each judgment sentence is correct, outputting only 'yes' or 'no';

[0055] 3) Generate a list of relational triples based on the conditional statement: (head entity, relation, tail entity), where the relation must be a given relation.

[0056] (3) Example of a mind chain: "The following is an example of a mind chain to help you generate results step by step: "

[0057] Input: 'Encephalitis@ Plasma exchange should be performed after consulting a hematologist. Encephalitis@ Usually requires 4-5 plasma exchanges every other day.' Relationship: [Adjunctive Therapy]

[0058] Head-to-tail body pair: [(Encephalitis, plasma exchange)]

[0059] Answer: 'Is plasma exchange an adjunct treatment for encephalitis?' Yes.

[0060] Generate a list of relation triples:

[0061] ``

[0062] (Encephalitis, adjunctive therapy, plasma exchange)

[0063] ``

[0064] 2. Prompt on the English dataset:

[0065] (1) Role definition: "You are currently a senior expert in relation extraction."

[0066] (2) Instruction: "Your objective is to extract head-tail entity pairs given atext and the specified relation, adhering to the following rules:

[0067] 1)Utilize the context of the given text with the specified relationto extract head-tail entity pairs that represent the relation.

[0068] 2)Verify the yes-no question sentence formed by concatenating the specified relation and the extracted head-tail entity pairs, and output only'yes'or'no'.

[0069] 3)Generate a list of triplet(head entity,relation,tail entity)based on the answer of yes-no question,where the relation must be the specifiedrelation.”

[0070] (3) Example of a chain of thought: “The following is an example of a chain of thought that guides you step by step to generate the results.

[0071] Input:'We propose and analyze a block minimization framework for datalarger than the memory size.'Relation:COMPARE

[0072] Head-tail entity pair:[(data,memory size)]

[0073] Answer:Is the relation between'data'and'memory size''COMPARE'? Yes

[0074] Generate a list of triplet:

[0075] ``

[0076] (data, COMPPARE, memory size)

[0077] ``

[0078] S22. If there is overlap between the model's output and the ground truth, the model's output is considered correct. Here, accuracy is defined as the ratio of crossover counts to ground truth counts. The F1 score of the extended relation r′ for all samples related to r is calculated in this way. If the F1 score is zero, then from R... ′ r Delete the extended relation r′.

[0079] S3. In the presence of this phenomenon, the extended relation set will be... (m rThe number of relations in R is merged into a predefined relation set R, creating a new relation set R″, and the relation extraction task is performed on the test set using R″, thereby improving the model's performance on relation extraction tasks.

[0080] S31, from R ′ r Select the top k valid extended relations with the highest F1 score for each r∈R, and merge R with the selected extended relations from all relations in R;

[0081] S32. Input each sentence and the merged relation set from the test set into the large model, and design role definitions, instructions, and examples to enhance the input and improve model performance. Simultaneously, design a thought chain in the instructions to further guide the large model, requiring the model to generate triples according to the following steps: First, identify a set of head entity pairs and tail entity pairs from the sentence; second, select one relation from the provided relation set for each pair to form a triple; finally, evaluate the reasonableness of each triple, retaining only the triples judged as reasonable. The specific prompt is as follows:

[0082] 1. Prompt on the Chinese dataset:

[0083] (1) Role definition: "Currently, you are a senior relationship extraction expert."

[0084] (2) Instructions: "Your task is to extract triples from the given text and list of relations, following these rules during the extraction process:"

[0085] 1) Generate a list of keyword pairs from a given text;

[0086] 2) Extract potential relations from a given list of relations using key pairs;

[0087] 3) Based on the relationship list and keywords, generate judgment sentences and check whether each judgment sentence is correct, outputting only 'yes' or 'no';

[0088] 4) Generate a list of triples (head entity, relation, tail entity) based on the answer to the judgment statement, where the relation must be from the given list of relations.

[0089] (3) Example of a mind chain: "The following is an example of a mind chain to help you generate results step by step: "

[0090] Input: 'Bacterial meningitis @ In elderly patients with meningitis, the only signs and symptoms are usually confusion or altered mental status.'

[0091] Intermediate step keyword pair: [(bacterial meningitis, confusion), (bacterial meningitis, altered mental status)]

[0092] Intermediate steps: [Clinical manifestations]

[0093] answer:

[0094] Is confusion a clinical manifestation of bacterial meningitis?

[0095] Is altered mental status a clinical manifestation of bacterial meningitis? Yes

[0096] Generate a list of relation triples:

[0097] ``

[0098] (Bacterial meningitis, clinical manifestations, confusion)

[0099] (Bacterial meningitis, clinical manifestations, altered mental status)

[0100] ``

[0101] 2. Prompt on the English dataset:

[0102] (1) Role definition: "You are currently a senior expert in relation extraction."

[0103] (2) Instruction: "Your objective is to extract triplets given a text and alist of relation, adhering to the following rules:

[0104] 1)Generate a keyword pair list from the given text

[0105] 2)Extract potential relations for keyword pairs from the givenrelation list

[0106] 3)Verify the yes-no question sentence formed by concatenating the potential relations and the extracted keyword pairs, and output only'yes'or'no'.

[0107] 4) Generate a list of triplet (head entity, relation, tail entity) based on the answer of yes - no question, where the relation must be in the given relation list {extend_label}.

[0108] (3) Chain of thought example: “The following is an example of a chain of thought that guides you step by step to generate the results.

[0109] Input: We present a text mining method for finding synonymous expressions based on the distributional hypothesis in a set of corpora.

[0110] Intermediate keyword pair: [(text mining method, synonymous expressions), (distributional hypothesis, text mining method)]

[0111] Intermediate relation: [USED - FOR, EVALUATE - FOR]

[0112] Answer:

[0113] Is the relation between ‘text mining method’ and ‘synonymous expressions’ the ‘USED - FOR’? Yes

[0114] Is the relation between ‘distributional hypothesis’ and ‘text mining method’ the ‘USED - FOR’? Yes

[0115] Is the relation between'text mining method'and'synonymousexpressions'the'EVALUATE-FOR'? No

[0116] Is the relation between'distributional hypothesis'and'text miningmethod'the'EVALUATE-FOR'? No

[0117] Generate a list of triplet:

[0118] ``

[0119] (text mining method,USED-FOR,synonymous expressions)

[0120] (distributional hypothesis,USED-FOR,text mining method)

[0121] ``

[0122] Specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that various modifications and substitutions can be made to the specific embodiments of the present invention without departing from the spirit and scope of the invention. All such modifications and substitutions fall within the scope defined by the claims of the present invention.

Claims

1. An information extraction algorithm based on large model expanded label enhancement, characterized in that... Includes the following steps: S1. For the relation extraction subtask in information extraction, a large model is used to expand predefined relations from the text and corresponding head and tail entity pairs in the given training set, specifically including: S11. Select all samples with relation r from the training set. For each sample, concatenate the sentence with the corresponding head and tail entity pairs. Then, input these combinations into the large model to prompt the large model to generate new relations, thereby obtaining an expanded set for each relation r. S12. Remove duplicate relations that appear in each extended set of each r to ensure the uniqueness between sets. S2. Use the extended relations from S1 to extract head and tail entity pairs from the text in the validation dataset. If the extracted entity pairs match the standard entity pairs, retain the corresponding extended relations, which include: S21. Select all samples with relation r from the validation set. For each r′, concatenate the sentence in each selected sample with r′ and input it into the large model. Prompt the large model to generate head and tail entity pairs. S22. Take the overlap count of the head and tail entity pairs and the golden head and tail entity pairs obtained in S21, and the ratio of the golden head and tail entity pair count to the golden head and tail entity pair count as the accuracy, and calculate the F1 score of the extended relation r′ of all samples related to r based on this. If the F1 score is zero, delete the extended relation r′. S3. Integrate the retained extended relations into a predefined relation set and apply it to the test set to enhance the performance of large models on relation extraction tasks. S4. For the named entity recognition subtask in information extraction, use a large model to expand predefined entity types from the text and corresponding entity span in the given training set. S5. Use the entity span in S4 to extend the predefined entity type. Extract the entity span from the text in the validation dataset. If the extracted entity span is consistent with the standard entity span, retain the corresponding extended entity. S6. Integrate the retained extended entities into a predefined entity set and apply them to the test set to enhance the performance of the large model on entity recognition tasks. S7. For the event detection subtask in information extraction, use a large model to expand predefined event types from the text in the given training set and the corresponding event trigger words; S8. Use the extended predefined event types in S7 to extract event trigger words from the text in the validation dataset. If the extracted trigger words are consistent with the standard trigger words, the corresponding extended events are retained. S9. Integrate the retained extended events into a predefined event set and apply them to the test set to enhance the performance of large models on event detection tasks.

2. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S3 specifically includes: S31. Select the top k valid extended relations with the highest F1 score from the set of extended relations and merge the above extended relations with the original predefined set of relations. S32. Input each sentence and the merged relation set from the test set into the large model, and design role definitions, instructions, and examples to enhance the input and improve model performance. At the same time, design a thought chain in the instructions to further guide the large model, that is, require the model to generate triples according to the following steps: First, identify a set of head and tail entity pairs from the sentence; second, select a relation for each pair from the provided relation set to form a triple; finally, evaluate the rationality of each triple and retain only the triples that are judged to be rational.

3. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S4 specifically includes: S41. Select all samples with entity type e from the training set. For each sample, concatenate the sentence with the corresponding entity span, and then input these combinations into the large model to prompt the large model to generate new entities, thereby obtaining an expanded set for each entity type e. S42. Delete the duplicate relationships that appear in each of the extended sets of each e to ensure the uniqueness between sets.

4. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S5 specifically includes: S51. Select all samples with entity type e from the validation set. For each extended entity type e′, concatenate the sentence in each selected sample with e′ and input it into the large model. Prompt the large model to generate entity span. S52. Take the overlap count of the entity span and the golden entity span obtained in S51, and use the ratio of the golden entity span count to the accuracy as the accuracy. Based on this, calculate the F1 score of the extended entity e′ of all samples related to entity type e. If the F1 score is zero, remove the extended entity type e′ from it.

5. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S6 specifically includes: S61. Select the top k valid extended entity types with the highest F1 score from them, and merge the above extended entity types with the original predefined entity type set; S62. Input each sentence and the merged entity type set from the test set into the large model, and design role definitions, instructions, and examples to enhance the input and improve model performance. At the same time, design a thought chain in the instructions to further guide the large model, that is, require the model to generate entity pairs according to the following steps: first, identify entity spans from sentences; second, select an entity type for each span from the provided entity set; finally, evaluate the rationality of each entity and its corresponding entity type, and only retain the entities and their entity types that are judged to be rational.

6. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S7 specifically includes: S71. Select all samples with event type d from the training set. For each sample, concatenate the sentence with the corresponding event trigger word, and then input these combinations into the large model to prompt the large model to generate new event types, thereby obtaining an extended event type set for each event type d. S72. Delete the duplicate event types that appear in each extended set of each d to ensure the uniqueness between sets.

7. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S8 specifically includes: S81. Select all samples with event d from the validation set. For each extended event type d′, concatenate the sentence in each selected sample with d′ and input it into the large model. Prompt the large model to generate event trigger words. S82. Take the overlap count of the event trigger words and the golden event trigger words obtained in S81, and use the ratio of the golden event trigger word count to the accuracy. Based on this, calculate the F1 score of the extended event type d′ of all samples related to event type d. If the F1 score is zero, remove the extended event type d′.

8. The information extraction algorithm based on large model expanded label enhancement according to claim 1, characterized in that, Step S9 specifically includes: S91. Select the top k valid extended event types with the highest F1 score from these extended event types and merge them with the original predefined set of event types. S92. Input each sentence and the merged set of event types from the test set into the large model, and design role definitions, instructions, and examples to enhance the input and improve model performance. At the same time, design a thought chain in the instructions to further guide the large model, that is, require the model to generate event trigger words in the following steps: first, identify event trigger words from the sentences; second, assign an event type to the event trigger words from the provided set of event types; finally, evaluate the rationality of each event trigger word and its event type, and only retain the event trigger words and their event types that are judged to be reasonable.

Images