Knowledge graph-based corpus entity extraction model adjustment method and system

Abstract

This application provides a method and system for adjusting a corpus-based entity extraction model based on knowledge graphs. It utilizes knowledge graph data to train a large-parameter general-purpose language model, and uses this data to compare and evaluate the accuracy of the training results. Subsequently, it generates corpora from the large-parameter general-purpose language model to adjust a small-parameter general-purpose language model. Furthermore, it leverages data derived from the knowledge graph to evaluate the adjusted small-parameter general-purpose language model, selecting high-quality models for deployment. This adjustment method enables the small-parameter general-purpose language model to possess high entity and relation extraction capabilities, while not relying on massive computing power, reducing data costs, and addressing the issues of poor model versatility and inadequate task completion quality.

Description

Technical Field

[0001] This invention relates to the field of large language models, and more particularly to a method and system for adjusting entity extraction models from corpora based on knowledge graphs. Background Technology

[0002] Knowledge graphs are a technological approach that uses graphical models to describe knowledge and model the relationships between everything in the world. Composed of nodes and edges, knowledge graphs describe concepts, entities, and their relationships in a structured form, expressing internet information in a way that more closely resembles human cognition. This provides a better ability to organize, manage, and understand the massive amounts of information on the internet. By leveraging the vast storage of facts and information through knowledge graphs, people can further improve computing speed with the help of computers, enabling them to respond effectively and promptly to scenarios requiring large amounts of computation.

[0003] With the development of the times, the amount of text accumulated in various fields is becoming increasingly massive. Therefore, language models have emerged to efficiently process or integrate text. Currently, based on the amount of data processed, language models can be divided into high-parameter and low-parameter language models. General-purpose high-parameter language models can perform text generation, text translation, text summarization, and intelligent question answering for various scenarios. However, in practical applications such as entity and relation extraction, high-parameter language models require significant computational power and consume substantial resources, while low-parameter language models require less computational power but require adjustments and interventions, and the quality of the output is not high. Summary of the Invention

[0004] One of the objectives of this invention is to address the shortcomings of existing technologies and provide a method for adjusting a corpus entity extraction model based on knowledge graphs. This method selects suitable large-parameter general-purpose language models for corpus production and then adjusts small-parameter general-purpose language models so that they can accurately extract entities and relationships with less computational power.

[0005] This invention provides a method for adjusting a corpus entity extraction model based on knowledge graphs, comprising the following steps:

[0006] S1. Derive triples from the knowledge graph;

[0007] S2. Input the triples into the large parameter general language model, and the large parameter general language model outputs entities and relations;

[0008] S3. The large-parameter general-purpose large language model outputs natural language text through entities and relations;

[0009] S4. Input the natural language text into a large parameter general-purpose large language model, and the large parameter general-purpose large language model outputs entities and relations;

[0010] S5. Evaluate the entities and relationships, and select the high-scoring generalized large language model with the largest parameters;

[0011] S6. Use the aforementioned large-parameter general-purpose large language model to produce a corpus;

[0012] S7. Use the corpus to adjust the small-parameter general large language model;

[0013] S8. Input natural language text into the adjusted small-parameter general large language model to output triples;

[0014] S9. Evaluate the triples and select the small-parameter general language model with the highest score.

[0015] In this scheme, triples derived from knowledge graphs are used to screen large-parameter general-purpose language models, and suitable large-parameter general-purpose language models are selected to produce corpora, providing a corpus for adjusting small-parameter general-purpose language models. Based on the evaluation, models that can extract entities and relations from natural language text with high quality can be selected, which solves the problem that small-parameter general-purpose language models still cannot complete text extraction tasks well even with external intervention. This saves computing power and improves the quality of text extraction.

[0016] In a preferred embodiment of the present invention, in step S1, the triples are in fact triple format, and the triples are output in a specific format. In this embodiment, the knowledge graph outputs fact triple files in a specific format, which are then input into a large-parameter general-purpose language model.

[0017] In a preferred embodiment of the present invention, in step S2, the triples and prompt words are input into a large-parameter general-purpose language model, and entities and relations are output in a certain format. In this embodiment, the triples are input into the large-parameter general-purpose language model, which extracts entities and relations and outputs them as files in formats such as JSON.

[0018] In a preferred embodiment of the present invention, in step S4, the natural language text, combined with prompt words, is input into several large-parameter general-purpose large language models, and the entities and relations are output in a certain format. In this embodiment, the natural language text generated by the large-parameter general-purpose large language model is input into the model, prompt words are added, and the model extracts the entities and relations from the text again, then outputs them as a file in a certain format.

[0019] In a preferred embodiment of the present invention, in steps S5 and S6, the entities and relations output by several large-parameter general-purpose language models are compared with the triples derived from the knowledge graph. The large-parameter general-purpose language model with the highest similarity is selected as the one with the highest score, and the large-parameter general-purpose language model with the highest score is used to produce corpus to form a corpus. In this embodiment, indicators such as ROUGH are used to compare the similarity between the entities and relations output by the large-parameter general-purpose language models and the entities and relations in the triples derived from the knowledge graph. The language model with the most similar results among multiple large-parameter general-purpose language models is selected as the language model for producing corpus, proving that its extraction accuracy is high and can be used to adjust the small-parameter general-purpose language models to produce corpus.

[0020] In a preferred embodiment of the present invention, in step S7, the corpus is input into several small-parameter general-purpose large language models in the AL PCA dataset format for adjustment. In this embodiment, the corpus generated by the selected large-parameter general-purpose large language models is used to train and adjust the small-parameter general-purpose language models, making them more suitable for certain specific text tasks.

[0021] In a preferred embodiment of the present invention, in step S8, the natural language text is input into several adjusted small-parameter general-purpose large language models to output JSON-formatted triples. The output JSON-formatted triples are used to evaluate the adjusted small-parameter general-purpose large language models to select the large language models with superior extraction capabilities.

[0022] In a preferred embodiment of the present invention, in step S9, the results output by several adjusted small-parameter general-purpose language models are compared using metrics such as ROUGH. The small-parameter general-purpose language model with the highest index is considered to have the highest score. In this embodiment, the entities in the triples output by the adjusted small-parameter general-purpose language model and the entities derived from the knowledge graph are compared using metrics such as ROUGH. The results are quantified by the metrics, and the small-parameter general-purpose language model with the highest index is selected based on the score. This model can then be used in various specific application scenarios.

[0023] The beneficial effects of the present invention include at least the following:

[0024] This invention provides a knowledge graph-based corpus entity extraction model adjustment method. It uses data from the knowledge graph to train a large-parameter general-purpose language model, and this data is used to compare and evaluate the accuracy of the training results. Subsequently, the large-parameter general-purpose language model generates corpora to adjust a small-parameter general-purpose language model. Furthermore, data derived from the knowledge graph is used a second time to evaluate the adjusted small-parameter general-purpose language model, selecting high-quality models for deployment. This adjustment method enables the small-parameter general-purpose language model to possess high entity and relation extraction capabilities, while not relying on massive computing power, reducing data costs, and improving model versatility and task completion quality.

[0025] The present invention also provides a system including an execution subject, wherein the execution subject is provided with a readable storage medium, and the readable storage medium stores a computer program. When the computer runs the computer program, it executes the knowledge graph-based corpus entity extraction model adjustment method as described above, which can be conveniently applied to various terminal devices, so that the devices execute the above adjustment method. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the steps of the knowledge graph-based corpus entity extraction model adjustment method provided in this application. Detailed Implementation

[0027] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0028] In the description of this invention, the terms "upper," "lower," "left," and "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and are not intended to require the invention to be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0029] The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” as used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0030] Example

[0031] See Figure 1 As shown, this invention provides a method for adjusting a corpus entity extraction model based on knowledge graphs, comprising the following steps:

[0032] S1. Derive triples from the knowledge graph;

[0033] S2. Input the triples into the large-parameter general-purpose language model, and the large-parameter general-purpose language model outputs entities and relations;

[0034] S3. The large-parameter general-purpose large language model outputs natural language text through entities and relations;

[0035] S4. Input the natural language text into the large parameter general-purpose large language model, and the large parameter general-purpose large language model outputs entities and relations;

[0036] S5. Evaluate entities and relationships, and select the high-scoring generalized large language model with the largest parameters;

[0037] S6. Use a large-parameter general-purpose language model to produce corpus to form a corpus;

[0038] S7. Use a corpus to adjust the general large language model with small parameters;

[0039] S8. Input natural language text into the adjusted small-parameter general large language model to output triples;

[0040] S9. Evaluate the triples and select the small-parameter general language model with the highest score.

[0041] Specifically, firstly, triples are derived from the knowledge graph, in the format of ("entity name", "relation name", "entity name") or [entity name": "entity name n", "attribute": ("attribute m", "attribute n"..., "attribute z")]. This triple data is input into a large-parameter general-purpose language model, which outputs a set of entities and relations. Language models such as Qwen-72B, Llama3-70B, and GLM4 can be used. Then, the large-parameter general-purpose language model is used again to further output the entity and relation set as natural language text. To test the accuracy of the extraction by the large-parameter general-purpose language model, the natural language text is returned to the large-parameter general-purpose language model for extraction, outputting entities and relations, which are then compared with the triples derived from the knowledge graph to confirm the similarity of entities and relations. In this way, the large-parameter general-purpose language model with the highest extraction accuracy can be selected as the language model for producing high-quality corpora.

[0042] A corpus is created using the selected large-parameter general-purpose language model to provide sufficient data for adjusting the small-parameter general-purpose language model. This data is then used to update some parameters of the small-parameter general-purpose language model, making it more suitable for specific text tasks. To evaluate the text extraction quality of the adjusted small-parameter general-purpose language model, natural language text is input into it to output triples. These triples are then evaluated to select the small-parameter general-purpose language models with higher extraction accuracy, which can then be put into further use.

[0043] This approach solves the problem that small-parameter general-purpose large language models still cannot perform text extraction tasks well even with external intervention, saving computing power and improving the quality of text extraction.

[0044] Furthermore, in step S1, the triples are in fact triple format and output in a specific format. In this scheme, the triples include entities, relations, and the attributes contained in both. The knowledge graph outputs fact triple files in a specific format, which are then input into a large-parameter general-purpose language model.

[0045] Furthermore, in step S2, the triples are input into a large-parameter general-purpose language model, and the entities and relations are output in a certain format. Specifically, the large-parameter general-purpose language model outputs the entities and relations in the triples in formats such as JSON and YAML.

[0046] Furthermore, in step S3, the triples are input into multiple large-parameter general-purpose language models (e.g., three or more), along with prompt words. The large-parameter general-purpose language models then output natural language text. Specifically, the input format for the prompt words and triples to the large-parameter general-purpose language model is: prompt word + one or more triples. The prompt words guide the model to generate text or perform specific tasks. For example, they tell the general-purpose language model that the user will input entities and relations, allowing the model to output natural language text; or they inform the model that there will be a piece of natural language text input, allowing the model to extract the entities and relations. The specific content of the prompt words is: "After ###, there are two entities and a relation description in JSON or YAML format, where 'Entity 1' represents the subject, 'Entity 2' represents the object, 'name' represents the name of this entity, 'attribute' is used to help describe this entity, and the 'predicate' in the 'relation' may be in English; please treat it as Chinese. Use this information to write a paragraph."

[0047] Furthermore, in step S4, the natural language text, combined with prompt words, is input into several large-parameter general-purpose language models, which then output entities and relations in a specific format. Specifically, the natural language text generated by the large-parameter general-purpose language models is returned to the input large-parameter general-purpose language models, and prompt words are added to extract entities and relations from the text again, which are then output as a file in a specific format. Therefore, the extraction accuracy of the large-parameter general-purpose language models can be determined by comparing the entities and relations generated by the large-parameter general-purpose language models with those exported from the knowledge graph. The prompt here could be: Extract entities and relations from the text following ###, and output the results in JSON format. Each entity is a separate entry, and relations are represented by triples, such as {"subject":"aphid","predicate":"belongs","object":"pesticide"}{"subject":"rice blast","predicate":"belongs","object":"disease"}. Entities and relations should be separated in the JSON". The main purpose of this prompt is to help the large-parameter general language model understand how to extract entities and relations from natural language text and what content format to use for the output results.

[0048] Furthermore, in steps S5 and S6, the entities and relations output by several large-parameter general-purpose language models are compared with the triples derived from the knowledge graph. The large-parameter general-purpose language model with the highest similarity is selected as the highest-scoring model. The high-scoring large-parameter general-purpose language model is used to produce corpus data to form a corpus, which is then used to adjust small-parameter general-purpose language models, serving as a data source for its adjustment. Specifically, the ROUGH method is used, comparing the similarity between the entities and relations output by the large-parameter general-purpose language model and the entities and relations extracted from the triples derived from the knowledge graph using ROUGH and other indicators. Based on the similarity level, the language model with the most similar results among multiple large-parameter general-purpose language models is selected as the language model for producing corpus data, demonstrating its high extraction accuracy and its ability to produce corpus data for adjusting small-parameter general-purpose language models and forming a corpus. ROUGH is a software package for automatically evaluating summary quality, which compares the computer-generated summary to be evaluated with a human-created ideal summary. ROUGH contains four metrics: ROUGE-N (n-gram co-occurrence statistics), ROUGE-L (longest common subsequence), ROUGE-W (weighted longest common subsequence), and ROUGE-S (skip word bigram co-occurrence statistics). ROUGE-L is calculated based on the longest common subsequence, which is defined as: if there exists a strictly increasing index sequence [i1, i2, ..., i...] k ], such that for all [j = 1, 2, ..., k], x ij =z i Then the sequence Z = [z1, z2] 2， …,zn ] is the sequence X = [x1, x2, ..., x3, ..., x4, ..., x5, ..., x6, ..., x7, ..., x8, ..., x9, ..., x1, ..., x1, ..., x9, ..., x1, ..., x1, ..., x2 ... 2， …,x m The longest common subsequence (LCS) of two sequences X and Y is the common subsequence with the largest length.

[0049] ROUGE will ultimately provide recall, precision, and F1 score. Recall is calculated based on sentence-level longest common subsequence (LSC). Precision and F1 score are calculated as follows:

[0050]

[0051] In formulas (2), (3), and (4), R lsc ,P lsc ,F lsc Let represent the recall, precision, and F1 score of sentence-level LSC, respectively. X and Y are two distinct sequences, and LSC(X,Y) is the longest common subsequence of X and Y. β is usually set to a very large number, and only R needs to be considered. lsc .

[0052] Since sentence-level LCS only considers word order matching between two sentences, ignoring the internal grammatical structure and semantic relationships, it cannot fully reflect the semantic similarity between two sentences. Furthermore, sentence-level LCS cannot be directly applied to the evaluation of multi-sentence summaries because it cannot effectively combine the LCS matching results between multiple sentences. Therefore, summary-level LCS is ultimately adopted, a method derived from sentence-level LCS.

[0053]

[0054] In formulas (5), (6), and (7), R lsc ,P lsc ,F lsc Recall, precision, and F1 score of the LSC at the abstract level are respectively the same as in formulas (6), (7), and (8), where β is set to a very large number by DUC, and only R lsc This needs to be considered. LCS U (ri,C) is the union of the sentence-level LCS scores of the reference summary ri and the candidate summary set C.

[0055] Further, in step S7, the corpus is organized in the Alpaca dataset format and input into several small-parameter general-purpose language models for adjustment. Specifically, the corpus generated by the selected large-parameter general-purpose language models is used to train and adjust the small-parameter general-purpose language models. By organizing the corpus into the Alpaca dataset format and combining it with prompt words, the LoRA method is used on three or more small-parameter general-purpose language models. This makes the small-parameter general-purpose language models more suitable for certain specific text tasks. LoRA (Low-Rank Adaptation) is an adjustment technique designed to better adapt a pre-trained language model to a specific task or domain. It introduces a low-rank matrix into the model's backbone, capturing the differences between the pre-trained model and the specific task, thereby adjusting the model. Compared to fully adjusting all parameters, LoRA only introduces and adjusts a small amount of data, offering advantages such as shorter adjustment time and higher efficiency. Understandably, in this embodiment, the prompt words should be simplified due to the long length of the input corpus data, reducing the overall length of the data input to the large language model. Therefore, the prompt words can be entered as: "You will perform entity extraction and relation extraction on the language text after ###, and the extraction results will be presented in JSON format ###".

[0056] Furthermore, in step S8, natural language text is input into several adjusted small-parameter general-purpose large language models to output triples. Specifically, the JSON-formatted triples are used to evaluate the adjusted small-parameter general-purpose large language models, comparing them with the triples exported from the knowledge graph, thereby selecting the large language model with better extraction capabilities.

[0057] Furthermore, in step S9, the results output by several adjusted small-parameter general-purpose language models are compared using metrics such as ROUGH. The small-parameter general-purpose language model with the highest index is the highest-scoring model. In this embodiment, the entities in the triples output by the adjusted small-parameter general-purpose language model and the entities derived from the knowledge graph are compared using metrics such as ROUGH. The results are quantified by the metrics, and the small-parameter general-purpose language model with the highest index is selected based on the score. This model can then be used in various specific application scenarios.

[0058] Specifically, using the ROUGH method, a small-parameter general-purpose large language model with adjusted test set input is used. The entities output by the small-parameter general-purpose large language model are compared with those output by the knowledge graph. The memory usage, GPU usage, and parsing success rate of the model output are also tested to determine the appropriate deployment method based on the parameter requirements of the application scenario.

[0059] The table below shows the language model adjusted in this embodiment and the performance comparison before and after the adjustment:

[0060]

[0061] Note: The small-parameter general-purpose large language models used include Qwen2-7B, yi1.5-9B, Lamma3-8B, etc. In the table, Llma3-Chinese-v2, Qwen2, and YI1.5 are model names, and the parameters 8B, 7B, and 9B after the colon are the number of parameters in the corresponding model. -ER indicates the adjusted model, and -ER-large indicates that the complete corpus was used during adjustment, i.e., a complete corpus of approximately 9000 data points, used to demonstrate the impact of using different corpus sizes. Inference time refers to the average time required for each data point in the test set to go from input to output. The JSON parsing success rate is expressed by the following formula:

[0062]

[0063] In the formula: JSON parsing success rate is an index that evaluates the model's understanding and execution capabilities of instructions, and assesses whether it can output in the expected format.

[0064] The knowledge graph-based corpus entity extraction model adjustment method provided in this embodiment can be stored in a readable storage medium containing a computer program. When the computer runs the program, it executes the knowledge graph-based corpus entity extraction model adjustment method described above. Specifically, ROM / RAM, magnetic disks, optical disks, etc., can all serve as carriers for the adjustment method, allowing it to be presented in software form based on this readable storage medium.

[0065] Specifically, the computer program executes on the computer device in the following steps:

[0066] Data containing entities and relationships is exported from the knowledge graph stored in a readable storage medium, and the data is summarized, valid values ​​are extracted and stored in the readable storage medium.

[0067] Read the data exported from the knowledge graph, configure prompt words for the general large language model, use the large parameter general large language model to convert the data exported from the knowledge graph into natural language text, and output the natural language text as a file and save it in a readable storage medium;

[0068] Read the natural language text output by the general large language model, configure prompt words for the general large language model, extract entities and relations from the natural language text using the large parameter general large language model, organize them into an entity and relation set, and then output and save the entity and relation set in a readable storage medium.

[0069] The ROUGH method is used to compare the entity and relation sets output by the large parameter general language model with the data initially exported from the knowledge graph. By comparing the entities and relations, the accuracy of the large parameter general language model in completing entity extraction and relation extraction tasks can be evaluated.

[0070] Select the high-performance general-purpose large language model with large parameters, and use its output corpus to adjust the low-parameter general-purpose large language model.

[0071] Use LoRA to tune a general large language model with small parameters;

[0072] Test a general large language model with small parameters and evaluate it using the ROUGH metric.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for adjusting a corpus-based entity extraction model based on knowledge graphs, characterized in that, Includes the following steps: S1. Derive triples from the knowledge graph; S2. Input the triples into a large-parameter general-purpose large language model. The large-parameter general-purpose large language model outputs entities and relations. Input the triples and prompt words into several large-parameter general-purpose large language models and output entities and relations in a certain format. S3. The large-parameter general-purpose large language model outputs natural language text through entities and relations; S4. Input the natural language text into a large parameter general language model. The large parameter general language model outputs entities and relations. Input the natural language text and prompt words into several large parameter general language models and output the entities and relations in a certain format. S5. Evaluate the entities and relationships, and select the high-scoring generalized large language model with the largest parameters; S6. Use the large parameter general language model to produce corpus to form a corpus, compare the entities and relations output by several large parameter general language models with the triples derived from the knowledge graph, the large parameter general language model with the highest similarity is the one with the highest score, and use the large parameter general language model with the highest score to produce corpus to form a corpus; S7. Use the corpus to adjust the small-parameter general large language model; S8. Input natural language text into the adjusted small-parameter general large language model to output triples; S9. Evaluate the triplet, select the small parameter general language model with the highest score, and compare the output results of several adjusted small parameter general language models using the ROUGH index. The small parameter general language model with the highest index is the highest scored.

2. The method for adjusting the corpus entity extraction model based on knowledge graphs according to claim 1, characterized in that: In step S1, the triplet is in fact triplet format, and the triplet is output in a certain format.

3. The method for adjusting the corpus entity extraction model based on knowledge graphs according to claim 1, characterized in that: In step S7, the corpus is input into a general large language model with several small parameters in the format of the alpaca dataset for adjustment.

4. The method for adjusting the corpus entity extraction model based on knowledge graphs according to claim 1, characterized in that: In step S8, the natural language text is input into a generalized large language model with several adjusted small parameters to output triples in JSON format.

5. A system for adjusting a corpus-based entity extraction model based on knowledge graphs, comprising an execution entity, wherein the execution entity is equipped with a readable storage medium storing a computer program, characterized in that: When the computer runs the computer program, it performs the knowledge graph-based corpus entity extraction model adjustment method as described in any one of claims 1-4.

Images