Data enhancement entity relation triple extraction method based on big and small model collaboration

Through the data enhancement method of collaborative size and model, the complexity and low recall of multimodal data processing in supply chain management in the power industry is solved, and more efficient and accurate physical relationship extraction is achieved, suitable for power grid supply chain management .

CN119088982APending Publication Date: 2024-12-06DEZHOU POWER SUPPLY COMPANY OF STATE GRID SHANDONG ELECTRIC POWER +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411192540.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional information processing methods are difficult to cope with the complexity and accuracy requirements in supply chain management of the power industry, especially when processing multimodal data and complex sentence patterns, the recall rate is low and the accuracy is insufficient, making it difficult to effectively integrate multimodal information.

Method used

Using data-enhanced entity relationship triple extraction method based on size and model collaboration, the data augmentation and relationship extraction tasks are combined with the advantages of large language models and small models. Optimize entity relationship extraction performance.

Benefits of technology

It significantly improves the efficiency and accuracy of the extraction of multiple entity relationship triplets, enhances the robustness and adaptability of data processing, can better integrate multimodal information, and is suitable for the field of power grid supply chain management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088982A_ABST
    Figure CN119088982A_ABST
Patent Text Reader

Abstract

The invention provides a data enhancement entity relation triple extraction method based on big and small model cooperation. The method comprises the architecture design of four task units: a data preprocessing task, a training data enhancement task, a multivariate entity pair screening task and a relation extraction task; according to the method, multi-modal information of a document is fully utilized as sentence text supplement, and entity relationship triples are better extracted by combining respective advantages of large and small models, so that the training effect and generalization ability of an entity on a screening model are improved, the efficiency and accuracy of processing a complex sentence pattern containing a plurality of entity relationship triples are improved, and the efficiency and the accuracy of processing the complex sentence pattern are improved. The method does not depend on a single data source or a traditional information extraction technology, has high compatibility, and can be widely applied to various information extraction tasks in the field of power grid services; in addition, a dynamic routing selection strategy according to a preliminary extraction result of the large language model is introduced, so that the robustness of the large language model in processing a plurality of entity relationship triples is improved, and the large language model can more accurately deal with complex information in supply chain management.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of supply chain management technology, and in particular to a data-enhanced entity relationship triple extraction method based on collaboration between large and small models, which is used to improve the efficiency and accuracy of knowledge semantic feature reasoning and active defect discovery technology, and greatly improves the overall level of information processing in the field of supply chain management. Background Art

[0002] In the current field of power industry supply chain management, with the rapid increase in data volume and the diversification of information types, traditional information processing methods are unable to cope with the growing complexity and accuracy requirements. Especially in power grid supply chain management, identifying potential risk points such as inventory shortages, transportation delays, single-source dependence, and supplier misconduct requires more advanced management technologies.

[0003] Traditional methods that rely solely on large language models (LLMs) are often unable to parse complex sentences containing multiple entity relationship triples. This is because large language models are not always better than small models in information extraction tasks. Their performance is limited by key factors such as insufficient ability to process long texts, misalignment problems when generating structured outputs, and insufficient model stability.

[0004] In addition, since the training data sets of large language models are mainly composed of sentences with simple structures, they tend to ignore key information when processing complex sentences containing many triples; although fine-tuning can improve the performance of large language models on specific tasks, it faces the dual challenges of insufficient data and inefficient decoding mechanisms, resulting in the recall rate of large language models being lower than that of small models in scenarios with multiple entity relationship triple extraction tasks.

[0005] Although traditional small models can extract more triples, they do so at the expense of accuracy. In addition, in the field of modern knowledge services that highly integrate natural language processing and semantic understanding, pure text information extraction technology faces challenges, especially in the field of power grid business, which involves supplier archives, transportation information, inventory data, etc., which may exist in the form of multimodal data. Large language models may not extract comprehensive information when processing multimodal information due to lack of fusion capabilities, and fail to effectively integrate and utilize the rich multimodal information contained in the document, resulting in ambiguity and coreference resolution problems.

[0006] If entity extraction technology can be extended to the multimodal field, the data can be enhanced. However, the differences in the expression of the same entity in different media also bring new challenges. Therefore, exploring how to integrate the advantages of large and small models to improve the efficiency and accuracy of information extraction and exploring how to make full use of limited data for data enhancement and expansion to expand the robustness of information extraction have become the top priorities of current research and development. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a data-enhanced entity relationship triple extraction method based on large and small model collaboration, which overcomes the problem that in the prior art, it is difficult to extract all entity relationship triples when a sentence has multiple entity relationship triples in a real multimodal scenario. By combining the model collaboration method with the advantages of a small entity extraction model and a large language generation model, the situation where a sentence may contain multiple entity relationship pairs is extracted, which greatly improves the performance of extracting multiple entity relationship triples, optimizes the efficiency and accuracy of the power grid supply chain management field in processing complex sentences with multiple entity relationship triples, and realizes the effective processing and solution of the low recall rate problem of entity relationship triple extraction in complex sentences by a large language model.

[0008] A data-enhanced entity-relation triple extraction method based on collaboration between large and small models, including the architecture design of four task units: data preprocessing task, training data enhancement task, multi-entity pair screening task and relationship extraction task;

[0009] The data preprocessing tasks are:

[0010] An image description generation model is used to convert the image into a text description, and an OCR method is used to extract the text content in the image;

[0011] Specifically, images describing the same scene are converted into text descriptions, and integrated using a large language model to form a single text language description as input for the large language model to extract entity relationship triples;

[0012] As an example, the integration refers to combining the features in the image with the information in the text, and effectively utilizing the world knowledge of the large language model for coreference resolution.

[0013] The training data augmentation task:

[0014] Data enhancement for positive samples is performed in two ways;

[0015] First, entity pair expansion rewriting is to expand the training set of the multi-entity pair screening task by rewriting the corresponding entities in the original sentence and the given positive entity pair;

[0016] Secondly, the extraction of relational enhancement based on cross-sentence pairs is to enrich the training data by extracting relational templates in sentences, identifying mutually related templates, and rewriting them;

[0017] As an example, the two methods of data augmentation for positive samples make full use of the world knowledge contained in the large language model to expand and rewrite the source training data from multiple dimensions, thereby increasing the diversity and coverage of the training data, and further improving the generalization ability of the entity to the screening model.

[0018] As an example, by diversifying and enhancing the positive samples, the entity pair screening model can better capture potential relationship patterns and reduce dependence on specific contexts or expressions, thereby improving the performance of the entity pair screening model when processing unseen data.

[0019] The multi-entity pair screening task:

[0020] Aims to filter out erroneous extraction results and screen out correct entity pairs;

[0021] As an example, the entity pair screening model and entity pair extraction model architecture of the present invention adopt BERT as the backbone network, use an enhanced data set to train the multi-entity pair screening task, and use the binary cross entropy loss function as the optimization target to train the entity pair screening model; this method can improve the accuracy of entity pair screening, better capture the fine-grained semantic information in the text, thereby enhancing the robustness of the entity pair screening model in the entity pair screening task.

[0022] The relationship extraction task:

[0023] The merged event description is directly input into the fine-tuned large language model to extract relation triples; specifically, the fine-tuned large language model is called to perform relation extraction in two steps;

[0024] The first step is to directly input the merged event description into the large language model to extract preliminary relationship triples;

[0025] Step 2: According to the result of the relation extraction of the large language model in the first step, determine whether a second relation extraction is needed to extract additional relation triples;

[0026] As an example, the process architecture for determining whether the second relationship extraction is required includes two categories:

[0027] ① There is no need to extract multiple relation triples, in which case the result of the relation extraction is the final result;

[0028] ② When multiple relationship triples need to be screened, the final result is: the second relationship extraction result after enhancement of the screened entity pairs.

[0029] As an example, for the first step of relation extraction of the large language model, a dynamic retrieval sample library is used as a contextual prompt to enhance the relation extraction capability of the large language model.

[0030] As an example, in the case where multiple triples need to be screened, the screened entity pairs are used as prompt information and spliced ​​with the relationship extraction results of the first step of the large language model, and entity relationship triples are extracted again to guide the large language model to consider more potential entity pairs when extracting multiple entity relationship triplets and determine appropriate relationships for them.

[0031] As an example, by supplementing the prompts and fine-tuning the instructions of the entity pairs after screening, the large language model can effectively utilize the contextual learning ability of the large language model for the task of extracting entity relationship triples, thereby improving the precision and recall of the large language model in the task of extracting entity relationship triples, making the entire process more robust.

[0032] Beneficial effects of the present invention:

[0033] Make full use of the multimodal information of the document as a supplement to the sentence text, combine the respective advantages of large and small models to better extract entity relationship triples, thereby improving the training effect and generalization ability of the entity pair screening model, and processing the efficiency and accuracy of complex sentences containing multiple entity relationship triples.

[0034] The present invention starts from the perspective of multimodal data processing and does not rely on a single data source or traditional information extraction technology, which significantly enhances the adaptability and flexibility of the present invention in relationship extraction tasks;

[0035] Moreover, the method of the present invention is highly compatible and can be widely used in various information extraction tasks in the field of power grid business. Specifically, the multimodal data preprocessing module and the entity pair evaluation module are designed to guide the large language model to better integrate image and text information, and accurately screen entity relationships. In the fine-tuning stage, the LoRA fine-tuning technology is combined with contextual learning to enhance the large language model's understanding of complex sentences and its ability to extract entity relationship triples.

[0036] In addition, a dynamic routing selection strategy based on the preliminary extraction results of the large language model is introduced to improve the robustness of the large language model in processing multiple entity relationship triples, enabling it to more accurately cope with complex information in supply chain management.

[0037] Compared with the prior art, the method proposed in the present invention has good performance and generalization ability in processing multimodal and complex sentence structures, and has broad application prospects in the field of multimodal information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the overall process of the data-enhanced entity relationship triple extraction method based on the collaboration of large and small models of the present invention.

[0039] Figure 2 Schematic diagram of the overall module architecture of the data-enhanced entity relationship triple extraction method based on collaboration between large and small models of the present invention. DETAILED DESCRIPTION

[0040] Below, reference Figure 1 to Figure 2 As shown in the figure, the data-enhanced entity-relation triple extraction method based on the collaboration of large and small models includes the architecture design of four task units: data preprocessing task, training data enhancement task, multi-entity pair screening task and relationship extraction task;

[0041] The data preprocessing tasks are:

[0042] An image description generation model is used to convert the image into a text description, and an OCR method is used to extract the text content in the image;

[0043] Specifically, images describing the same scene are converted into text descriptions, and the large language model is used to integrate the images with the original text descriptions of the task to form a single text language description, which is used as the input for the large language model to extract entity relationship triples;

[0044] As an example, the OCR is: optical character recognition method.

[0045] As an example, this process combines features in the image with information in the text, effectively leveraging the world knowledge of the large language model for coreference resolution.

[0046] As an example, the tips for integrating the text description of the image and the text into events using the large language model are:

[0047] Prompt: The following content refers to the same event. The parts of the image description that may have the same reference as the text are unified based on the text. Multiple events are merged and summarized into one event. Only the merged event description is output. In this process, as much entity information as possible is retained.

[0048] The training data augmentation task:

[0049] Data enhancement for positive samples is performed in two ways;

[0050] First, entity pair expansion rewriting is: expand the training data in the entity pair screening task by rewriting the entities in the given positive entity pair and the corresponding entities in the original sentence; the rewriting prompt is constructed as follows: Prompt: Rewrite the content of the following json dataset data: rewrite the entities em1Text and em2Text mentioned in relationMentions, and rewrite the corresponding content in sensText. Note that the label key value field of the triple must not be changed after rewriting. The relationship name of the triple must be selected from the relationship given in the candidate relationship list, and other relationships are not considered. Only the rewritten json data is returned in the json format of the original data. Do not return anything other than the json data.

[0051] Secondly, based on the extraction of cross-sentence pairs, the relationship enhancement is to extract the relationship templates in the sentence, identify the related templates, and rewrite them to enrich the training data; the rewriting prompt is constructed as follows: Prompt: Rewrite the content of the following json dataset data: and rewrite the content describing the relationship in sensText. Be careful not to rewrite the listed entities, only rewrite the relationship description between the entity pairs, but do not change their meaning. Only return the rewritten json data in the json format of the original data, and do not return anything else except the json data:

[0052] As an example, the original sentence is: sensText; the given positive entity pair is: {"em1Text": "entity 1", "em2Text": "entity 2", "label": "relationship"}.

[0053] Secondly, the extraction of relational enhancement based on cross-sentence pairs is to enrich the training data by extracting relational templates in sentences, identifying mutually related templates, and rewriting them;

[0054] As an example, positive entity pairs across sentences are identified based on a core assumption, namely: if the positive entity pairs in two sentences show consistent relational text patterns, namely relation templates, then the two entity pairs essentially share the same relationship; based on this assumption, these entity pairs can be considered to have the same relational features and are therefore regarded as positive samples in the training set of the multi-entity pair screening task; first, relation templates that present the same relational patterns in different sample data are identified and captured; then, these relation templates are used to rewrite the relational descriptions in the original sentences to generate new training samples.

[0055] As an example, the two methods of performing data augmentation on the training data for the multi-entity pair screening task make full use of the world knowledge contained in the large language model to expand and rewrite the source training data from multiple dimensions, thereby increasing the diversity and coverage of the training data, and further improving the generalization ability of the entity pair screening model.

[0056] As an example, by diversifying and enhancing the training data of the entity pair extraction task, the entity pair screening model can better capture potential relationship patterns and reduce dependence on specific contexts or expressions, thereby improving the performance of the entity pair screening model when processing unseen data.

[0057] The multi-entity pair screening task:

[0058] Aims to filter out erroneous extraction results and screen out correct entity pairs;

[0059] As an example, the entity pair screening model architecture of the present invention adopts BERT as the backbone network, uses an enhanced data set for training, and uses a binary cross entropy loss function as the optimization target to train the screening model; this method can improve the accuracy of entity pair screening and better capture the fine-grained semantic information in the text, thereby enhancing the robustness of the entity pair screening model in the multi-entity pair screening task.

[0060] Specifically, when text is input into the network:

[0061] First, use the Bert pre-training model to pre-encode the text and obtain the word vector v of each word. i ;

[0062] Then, the encoder with the same architecture is used to encode the word into a two-dimensional word vector attention matrix; specifically, two linear layers are used to encode the encoded word vector v i Convert to two-dimensional word vector attention: q i and k i ;

[0063] Finally, calculate each word vector v i Cosine similarity and scaled dot product attention between As output, the association between entity pairs is judged;

[0064] The evaluation adopts the training loss as the joint optimization of binary cross entropy loss function and marginal loss function;

[0065] The binary cross entropy loss function is defined as follows:

[0066]

[0067] Where: y ijis a word pair (t i ,t j ) indicates whether the two words are positively correlated, σ is the sigmoid function; A ij is the scaled dot product attention between word vectors.

[0068] The marginal loss function is defined as follows:

[0069]

[0070] Among them, dist(a i r ,p i r ) represents sample a i r and its random positive sample p i r The cosine distance, dist(a i r ,n i r ) represents sample a i r and its random negative sample p i r The cosine distance of , γ is used as a margin parameter to control the minimum required gap between the scores of positive and negative samples.

[0071] The relationship extraction task:

[0072] The merged event description is directly input into the fine-tuned large language model for entity relation triple extraction;

[0073] This process calls the large language model to perform relationship extraction in two steps. In the first step, the merged event description is directly input into the large language model for multiple sampling to extract preliminary entity relationship triples.

[0074] In the second step, the relationship extraction results obtained by multiple sampling of the large language model in the first step are analyzed and the self-consistency of the large language model is analyzed to determine whether a second relationship extraction is needed to extract additional entity relationship triples. Based on the judgment, the task process is divided into two categories:

[0075] ① There is no need to extract multiple triples. In this case, the answer of the large language model is the final result;

[0076] ② And the situation where multiple triplets need to be screened;

[0077] As an example, for the first step of relationship extraction of the large language model, samples with semantics similar to those of the sentences in the current relationship extraction task are retrieved from the dynamic retrieval sample library as contextual prompts for the large language model to enhance the relationship extraction capability of the large language model.

[0078] As an example, in the case where multiple triples need to be screened, the screened entity pairs are used as prompt information and spliced ​​with the first step of the relation extraction result of the large language model, and entity relation triples are extracted again to guide the large language model to consider more potential entity pairs when extracting multiple entity relation triples and determine appropriate relations for them. The prompt is constructed as follows: Prompt: {First relation extraction problem and results}. Now we are required to judge that the entity pairs that may be related in the above sentence are {candidate entity pairs}. Please check these candidate entity pairs in the original text, determine whether they may exist in the relation list, add missing triples, discard wrong triples, and output the final result.

[0079] As an example, by supplementing the prompts and fine-tuning the instructions of the entity pairs after screening, the large language model can effectively utilize the contextual learning ability of the large language model for multiple entity relationship triple extraction tasks, thereby improving the precision and recall of the large language model in the entity relationship triple extraction task, making the entire process more robust.

[0080] The above are only preferred embodiments of the present invention. It should be understood that the description of the above embodiments is only used to help understand the method and core ideas of the present invention, and is not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the ideas and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data-enhanced entity relationship triple extraction method based on collaboration between large and small models, characterized in that: The architecture design includes four task units: data preprocessing task, training data enhancement task, multi-entity pair screening task and relationship extraction task; The data preprocessing tasks are: An image description generation model is used to convert the image into a text description, and an OCR method is used to extract the text content in the image; Specifically, images describing the same scene are converted into text descriptions, and integrated using a large language model to form a single text language description as input for the large language model to extract entity relationship triples; The training data augmentation task: Data enhancement for positive samples is performed in two ways; First, entity pair expansion and rewriting is to expand the multi-entity pair screening task training set by rewriting the corresponding entities in the original sentence and the given positive entity pair; Secondly, the extraction of relational enhancement based on cross-sentence pairs is to enrich the training data by extracting relational templates in sentences, identifying related templates, and rewriting them; The multi-entity pair screening task: Aims to filter out erroneous extraction results and screen out correct entity pairs; The relationship extraction task: The merged event description is directly input into the fine-tuned large language model to extract entity relationship triples; specifically, the fine-tuned large language model is called to perform relationship extraction in two steps; The first step is to directly input the merged event description into the large language model to extract preliminary entity relationship triples; The second step is to determine whether a second relationship extraction is needed to extract additional entity relationship triples based on the result of the relationship extraction in the first step.

2. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: The OCR is: optical character recognition method.

3. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: The integration refers to combining features in the image with information in the text, and effectively utilizing the world knowledge of the large language model to perform coreference resolution.

4. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: The two methods of data enhancement for positive samples make full use of the world knowledge contained in the large language model to expand and rewrite the source training data from multiple dimensions, thereby increasing the diversity and coverage of the training data, and further improving the generalization ability of the entity to the screening model.

5. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: By diversifying and enhancing the positive samples, the entity pair screening model can better capture potential relationship patterns and reduce dependence on specific contexts or expressions, thereby improving the performance of the entity pair screening model when processing unseen data.

6. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: The entity pair screening model and entity pair extraction model architecture adopt BERT as the backbone network, use an enhanced dataset to train the multi-entity pair screening task, and use the binary cross entropy loss function as the optimization target to train the entity pair screening model; this method can improve the accuracy of entity pair screening and better capture the fine-grained semantic information in the text, thereby improving the robustness of the entity pair screening model in the entity pair screening task.

7. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: The process architecture for determining whether a second relationship extraction is required includes two categories: ① When there is no need to extract multiple triples, the result of the relationship extraction is the final result; ② When multiple triples need to be screened, the final result is: the result of the second relationship extraction after the screening entity pairs are enhanced.

8. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: For the first step of extracting preliminary entity relationship triples, a dynamic retrieval sample library is used as contextual prompts to enhance the relationship extraction capability of the large language model.

9. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 1 is characterized in that: When multiple triplets need to be screened, the screened entity pairs are used as prompt information and spliced ​​with the relationship extraction results of the first step of the large language model, and relation triples are extracted again to guide the large language model to consider more potential entity pairs when extracting multiple entity relationship triplets and determine appropriate relationships for them.

10. The data-enhanced entity relationship triple extraction method based on large and small model collaboration according to claim 9 is characterized in that: By supplementing the prompts and fine-tuning the instructions of the entity pairs after screening, the large language model can effectively utilize the contextual learning ability of the large language model for the task of extracting entity relationship triples, thereby improving the precision and recall of the large language model in the task of extracting entity relationship triples, making the entire process more robust.

Citation Information

Patent Citations

  • Multi-modal knowledge graph verification method and device, electronic equipment and storage medium

    CN117725232A

  • System and method for searching media or data based on contextual weighted keywords

    US20240070188A1