Knowledge graph-based corpus entity extraction model adjustment method and system

Abstract

According to the corpus entity extraction model adjustment method and system based on the knowledge graph, the knowledge graph is used for providing data to train a large-parameter general large language model, and the data is used for comparing and evaluating the accuracy degree of training results. And then corpora are produced through the large-parameter general large-language model to serve for adjusting the small-parameter general large-language model, and the adjusted small-parameter general large-language model is evaluated through secondary utilization of data exported from the knowledge graph, so that the high-quality small-parameter general large-language model is screened out to be put into use. By means of the adjusting method, the small-parameter general large language model has high entity and relation extraction capacity, meanwhile, the method does not depend on a large amount of computing power, the data cost can be reduced, and the problems that the model universality is poor and the task completion quality is poor are solved.

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a method and system for adjusting a corpus entity extraction model based on a knowledge graph. Background Art

[0002] Knowledge graph is a technical method that uses a graph model to describe knowledge and model the relationship between all things in the world. The knowledge graph consists of nodes and edges, and describes concepts, entities and their relationships in the objective world in a structured form. It expresses the information on the Internet in a form closer to the human cognitive world, and provides a better ability to organize, manage and understand the massive amount of information on the Internet. By using the knowledge graph to store a large amount of facts and information, people can further improve the computing speed with the help of computers, and can respond effectively and promptly when faced with scenarios that require large amounts of computing.

[0003] With the development of the times, the amount of text accumulated in various fields has become increasingly huge. Therefore, language models have emerged in order to efficiently process or integrate text. At present, language models can be divided into large-parameter and small-parameter language models according to the amount of data processed. Among them, the general large language model can be used for text generation, text translation, text induction, and intelligent question and answer for a variety of scenarios. However, in practical applications such as the extraction of entities and relationships, large-parameter language models require large computing power and consume more resources, while small-parameter language models require less computing power, but need to be adjusted and intervened, and the completion quality is not high. Summary of the invention

[0004] The present invention aims to address the defects in the prior art, and one of its objectives is to provide a corpus entity extraction model adjustment method based on a knowledge graph, to screen out a suitable large-parameter general-purpose large language model for producing corpus, and then to adjust the small-parameter general-purpose large language model so that the small-parameter general-purpose large language model can accurately extract entities and relationships with less computing power.

[0005] The present invention provides a corpus entity extraction model adjustment method based on knowledge graph, comprising the following steps:

[0006] S1. Derive triples from the knowledge graph;

[0007] S2. Input the triples into a large-parameter universal language model, and the large-parameter universal language model outputs entities and relationships;

[0008] S3. Large parameter general language model outputs natural language text through entities and relations;

[0009] S4. Inputting the natural language text into a large-parameter universal language model, and the large-parameter universal language model outputs entities and relationships;

[0010] S5. Evaluate the entities and relations, and select the large-parameter universal language model with the highest score;

[0011] S6. Using the large-parameter universal large language model to produce corpus to form a corpus;

[0012] S7. Using the corpus to adjust the small parameter universal large language model;

[0013] S8. Inputting the natural language text into the adjusted small parameter universal large language model to output triples;

[0014] S9. Evaluate the triples and select the small-parameter universal large language model with the highest score.

[0015] In this scheme, the large-parameter general large language model is screened through the triples exported from the knowledge graph, and the suitable large-parameter general large language model production corpus is selected to provide a corpus for adjusting the small-parameter general large language model. According to the evaluation, the model with high quality in extracting entities and relationships from natural language text can be screened out, which solves the problem that the small-parameter general large language model still cannot complete the text extraction task well due to the need for external intervention. It not only saves computing power but also improves the quality of completed text work.

[0016] In a preferred technical solution of the present invention, in step S1, the triple is in fact triple format, and the triple is output in a certain format. In this solution, the knowledge graph outputs the fact triple file in a certain format, and then inputs the large parameter universal large language model.

[0017] In a preferred technical solution of the present invention, in step S2, the triples and prompt words are input into a number of large-parameter universal language models, and entities and relations are output in a certain format. In this solution, the triples are input into the large-parameter universal language model, so that the large-parameter universal language model extracts entities and relations, and outputs them as files in a format such as json.

[0018] In a preferred technical solution of the present invention, in step S4, the natural language text is input into a plurality of large-parameter universal large language models in combination with prompt words, and the entities and relationships are output in a certain format. In this solution, the natural language text produced by the large-parameter universal large language model is input into the large-parameter universal large language model, prompt words are added, so that the entities and relationships in the text are extracted again, and output as a file in a certain format.

[0019] In a preferred technical solution of the present invention, in the steps S5 and S6, the entities and relations output by the large-parameter universal large language models are compared with the triples derived from the knowledge graph, and the large-parameter universal large language model with the highest similarity is the highest score, and the large-parameter universal large language model with the highest score is used to produce corpus to form a corpus. In this solution, ROUGH and other indicators are used to compare the similarity of the entities and relations output by the large-parameter universal large language model and the entities and relations in the triples derived from the knowledge graph, and the language model with the most similar results is selected from multiple large-parameter universal large language models as the language model for producing corpus, which proves that its extraction accuracy is high and can be used to adjust the small-parameter universal large language model to produce corpus to form a corpus.

[0020] In a preferred technical solution of the present invention, in step S7, the corpus is input into several small-parameter universal large language models in the Al Paca data set format for adjustment. In this solution, the corpus produced by the screened large-parameter universal large language model is used to train and adjust the small-parameter universal language model, so that the small-parameter universal language model is more suitable for certain specific text tasks.

[0021] In a preferred technical solution of the present invention, in step S8, the natural language text is input into a plurality of adjusted small-parameter universal large language models to output triples in json format. The triples output in json format are used to evaluate the adjusted small-parameter universal large language models to select large language models with better extraction capabilities.

[0022] In a preferred technical solution of the present invention, in step S9, the results output by several adjusted small-parameter universal large language models are compared by using indicators such as ROUGH, and the small-parameter universal large language model with the highest indicator index has the highest score. In this solution, the entities in the triples output by the adjusted small-parameter universal large language model and the entities derived from the knowledge graph are compared using indicators such as ROUGH, the results are quantified by indicators, and the small-parameter universal large language model with the largest index is selected according to the score, which can be used in various specific application scenarios.

[0023] The beneficial effects of the present invention include at least:

[0024] The present invention provides a method for adjusting a corpus entity extraction model based on a knowledge graph, which uses a knowledge graph to provide data for training a large-parameter general large language model, and uses this data to compare and evaluate the accuracy of the training results. Subsequently, the large-parameter general large language model is used to produce corpus to serve the adjustment of the small-parameter general large language model, and the data derived from the knowledge graph is used for secondary use to evaluate the adjusted small-parameter general large language model, so as to screen out high-quality small-parameter general large language models for use. This adjustment method enables the small-parameter general large language model to have a higher ability to extract entities and relationships, and at the same time does not rely on a large amount of computing power, which can reduce data costs and improve the model's versatility and task completion quality.

[0025] The present invention also provides a system, including an execution body, on which a readable storage medium is provided, and the readable storage medium stores a computer program. When the computer runs the computer program, the corpus entity extraction model adjustment method based on the knowledge graph as described above is executed, which can be conveniently used in various terminal devices so that the devices execute the above adjustment method. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the steps of the knowledge graph-based corpus entity extraction model adjustment method provided in this application. DETAILED DESCRIPTION

[0027] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0028] In the description of the present invention, the directions or positional relationships indicated by the terms "up", "down", "left" and "right" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention rather than requiring the present invention to be constructed and operated in a specific direction. Therefore, they should not be understood as limitations on the present invention.

[0029] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms of "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0030] Example

[0031] See also Figure 1 As shown, the present invention provides a corpus entity extraction model adjustment method based on knowledge graph, comprising the following steps:

[0032] S1. Derive triples from the knowledge graph;

[0033] S2. Input the triples into a large-parameter universal large language model, and the large-parameter universal large language model outputs entities and relations;

[0034] S3. Large parameter general language model outputs natural language text through entities and relations;

[0035] S4. inputting the natural language text into the large-parameter universal large language model, and the large-parameter universal large language model outputs entities and relationships;

[0036] S5. Evaluate entities and relations and select the large-parameter universal language model with the highest score;

[0037] S6. Use a large-parameter general language model to generate corpus to form a corpus;

[0038] S7. Use the corpus to adjust the small-parameter general large language model;

[0039] S8. Inputting the natural language text into the adjusted small parameter universal large language model to output triples;

[0040] S9. Evaluate the triples and select the small-parameter universal large language model with the highest score.

[0041] Specifically, first, a triple is exported from the knowledge graph, and the format can be: ("entity name", "relationship name", "entity name") or [entity name": "entity name n", "attribute": ("attribute m", "attribute n" ..., "attribute z")]. This triple data is input into a large-parameter universal large language model, and the large-parameter universal large language model outputs and forms an entity and relationship set. The above-mentioned language model can be selected from Qwen-72B, Llama3-70B, GLM4, etc. Then, the large-parameter universal large language model is used again to further output the above-mentioned entity and relationship set as natural language text. In order to detect whether the extraction of the large-parameter universal large language model used is accurate enough, the above-mentioned natural language text is returned to the large-parameter universal large language model for extraction, and the entities and relationships are output, and compared with the triples exported from the knowledge graph to confirm the similarity of the entities and relationships. In this way, the large-parameter universal large language model with the highest extraction accuracy can be screened out as the language model for producing high-quality corpus.

[0042] The selected large-parameter universal large language model is used to produce corpus to form a corpus, which provides sufficient adjustment data for adjusting the small-parameter universal large language model. The above adjustment data is used to update some parameters of the small-parameter universal large language model to make it more suitable for specific text tasks. In order to detect the completion quality of text extraction by the adjusted small-parameter universal large language model, natural language text is input into the adjusted small-parameter universal large language model to output triples, so that the triples can be evaluated and the small-parameter universal large language model with higher extraction accuracy can be selected, which can be further put into use.

[0043] In this way, the problem that a general large language model with small parameters still cannot complete the text extraction task well, such as requiring external intervention, is solved. This not only saves computing power but also improves the quality of completed text work.

[0044] Furthermore, in step S1, the triple is in fact triple format and is output in a certain format. In this solution, the triple contains entities, relations and the attributes contained in the two. The knowledge graph outputs the fact triple file in a certain format, and then inputs the large parameter universal language model.

[0045] Furthermore, in step S2, the triples are input into a number of large-parameter universal large language models, and the entities and relations are output in a certain format. Specifically, the large-parameter universal large language model outputs the entities and relations in the triples in a json, yaml or other format file.

[0046] Further, in step S3, the triples are input into multiple large-parameter universal language models, such as more than three, and prompt words are input, and natural language text is output through the large-parameter universal language model. Specifically, the input format of prompt words and triples for the large-parameter universal language model is: prompt words + more than one triple. The prompt words are used to guide the model to generate text or perform input text for a specific task, such as telling the universal language model that the user will input entities and relationships, and let the universal language model output natural language text, or telling the large language model that there will be a natural language text input, so that the universal language model can extract the entities and relationships therein. The specific content of the prompt words is: "After ### are two entities and a relationship description expressed in json or yaml format, where "entity 1" represents the subject, "entity 2" represents the object, "name" represents the name of this entity, "attribute" is used to assist in describing this entity, and the "predicate" in the "relationship" may be in English. Please treat it as Chinese and use this information to write a paragraph.

[0047] Furthermore, in step S4, the natural language text is input into several large-parameter universal large language models in combination with the prompt words, and the entities and relations are output in a certain format. Specifically, the natural language text produced by the large-parameter universal large language model is returned to the large-parameter universal large language model used for input, and the prompt words are added so that the entities and relations in the text are extracted again and output as a file in a certain format. To this end, the entities and relations generated by the large-parameter universal large language model can be compared with the entities and relations derived from the knowledge graph to obtain the extraction accuracy of the large-parameter universal large language model used. The prompt words here can be: perform entity extraction and relationship extraction on the paragraph after ###, and output it in Json format. Each entity is a separate entry, and the relationship is represented by a triplet to represent a group of relationships, for example, {"subject":"wheat aphid","predicate":"belongs to","object":"pest"}{"subject":"rice blast","predicate":"belongs to","object":"disease"}Entities and relationships should be separated in json". The main purpose of the prompt words shown is to let the large-parameter general language model understand how to extract entities and relationships from natural language text and in what content format the extraction results are output.

[0048] Furthermore, in steps S5 and S6, the entities and relations output by several large-parameter universal large language models are compared with the triples derived from the knowledge graph, and the large-parameter universal large language model with the highest similarity is the one with the highest score. The large-parameter universal large language model with the highest score is used to produce corpus to form a corpus, which is used to adjust the small-parameter universal large language model as a data source for its adjustment. Specifically, the ROUGH method is used to compare the similarity of the entities and relations output by the large-parameter universal large language model with the entities and relations extracted from the triples derived from the knowledge graph using ROUGH and other indicators. According to the similarity, the language model with the most similar results is selected from multiple large-parameter universal large language models as the language model for producing corpus, proving that its extraction accuracy is high and can produce corpus for adjusting the small-parameter universal large language model and form a corpus. ROUGH is a software package for automatically evaluating the quality of summaries, which is achieved by comparing the computer-generated summary to be evaluated with the ideal summary created manually. ROUGH contains four indicators, namely ROUGE-N (n-gram co-occurrence statistics), ROUGE-L (longest common subsequence), ROUGE-W (weighted longest common subsequence and ROUGE-S (skip-word bigram co-occurrence statistics). ROUGE-L is calculated based on the longest common subsequence. The longest common subsequence is defined as: if there is a strictly increasing index sequence [i1,i2,…,i k ], so that for all [j=1,2,…,k], there is x ij =z i , then the sequence Z=[z1,z 2， …,zn ] is the sequence X=[x1,x 2， …,x m ] is a subsequence (Common Subsequence, CS). Given two sequences X and Y, the longest common subsequence (LCS) of X and Y is the common subsequence with the maximum length.

[0049] ROUGE will eventually give the recall, precision and F1 value. The calculation method of the recall, precision and F1 value based on the sentence-level longest common subsequence (LSC) is as follows:

[0050]

[0051] In formulas (2), (3), and (4), R lsc ,P lsc ,F lsc They represent the recall, precision, and F1 value of LSC at the sentence level. X and Y are two different sequences. LSC(X,Y) is the longest common subsequence of X and Y. β is usually set to a large number. Only R needs to be considered. lsc .

[0052] Since sentence-level LCS only considers the word order matching between two sentences and ignores the grammatical structure and semantic relationship within the sentences, it cannot fully reflect the semantic similarity of the two sentences. Sentence-level LCS cannot be directly applied to the evaluation of multi-sentence summaries because it is impossible to effectively combine the LCS matching results between multiple sentences. Therefore, the summary-level LCS is finally used, which is derived from the sentence-level LCS:

[0053]

[0054] In formulas (5), (6), and (7), R lsc ,P lsc ,F lsc They represent the recall, precision, and F1 value of LSC at the summary level, respectively. They are the same as formulas (6), (7), and (8). β is set to a large number by DUC. Only R lsc This is something that needs to be considered. LCS U (ri,C) is the union of the sentence-level LCS scores of the reference summary ri and the candidate summary set C.

[0055] Furthermore, in step S7, the corpus is organized in the alpaca dataset format and several small-parameter universal large language models are input for adjustment. Specifically, the corpus produced by the screened large-parameter universal large language model is used to train and adjust the small-parameter universal language model, by organizing the corpus into the alpaca dataset format, combining the prompt words, and using the LoRA method for more than three small-parameter universal large language models. Make the small-parameter universal language model more suitable for certain specific text tasks. LoRA (Low-Rank Adaptation) is an adjustment technology that aims to make the pre-trained language model better adaptable to specific tasks or fields. It introduces a low-rank matrix on the backbone of the model, and captures the difference between the pre-trained model and the specific task through the low-rank matrix, thereby adjusting the model. Compared with fully adjusting all parameters, LoRA only introduces a small amount of data and only adjusts a small amount of data, which has the advantages of short adjustment time and high efficiency. It can be understood that in this embodiment, the prompt word should be simplified because the input corpus data is long, so as to reduce the overall length of the data input into the large language model. Therefore, the prompt word can be input as: "You will perform entity extraction and relationship extraction on the language text after ###, and the extraction results will be presented in json format###"

[0056] Furthermore, in step S8, the natural language text is input into several adjusted small parameter universal large language models to output triples. Specifically, the triples in json format are used to evaluate the adjusted small parameter universal large language models and compared with the triples derived from the knowledge graph, so as to select the large language model with better extraction ability.

[0057] Further, in step S9, the results output by several adjusted small-parameter universal large language models are compared by using indicators such as ROUGH, and the small-parameter universal large language model with the highest indicator index has the highest score. In this embodiment, the entities in the triples output by the adjusted small-parameter universal large language model and the entities derived from the knowledge graph are compared by using indicators such as ROUGH, the results are quantified by indicators, and the small-parameter universal large language model with the largest index is selected according to the score, which can be used in various specific application scenarios.

[0058] Specifically, the ROUGH method is used to input the adjusted small-parameter universal large language model with the test set, and the entities output by the small-parameter universal large language model are compared with the entities output by the knowledge graph. The video memory occupancy, GPU occupancy, and parsing success rate of the model output are tested for deployment according to the parameter requirements of the scenario.

[0059] The following table shows the language model adjusted in this embodiment and the performance comparison before and after the adjustment:

[0060]

[0061] Note: The small-parameter general-purpose large language models used include Qwen2-7B, yi1.5-9B, Lamma3-8B, etc. In the table, Llma3-Chinese-v2, Qwen2, and YI1.5 are the model names, and the parameters 8B, 7B, and 9B after the colon are the parameter quantities of the corresponding models. -ER indicates the adjusted model, and -ER-large indicates that the complete corpus, i.e., the complete corpus data of about 9,000 pieces, is used to show the impact of using corpora of different sizes. The inference time refers to the average time required from input to output for each piece of data in the test set. The json parsing success rate is expressed by the following formula:

[0062]

[0063] Where: JSON parsing success rate is an index to evaluate the model's understanding and execution capabilities of instructions, and whether it can output in the expected format.

[0064] The corpus entity extraction model adjustment method based on the knowledge graph provided in this embodiment can be stored in a readable storage medium, which stores a computer program. When the computer runs the computer program, the corpus entity extraction model adjustment method based on the knowledge graph as described above is executed. Specifically, ROM / RAM, magnetic disk, optical disk, etc. can be used as the carrier of the above adjustment method, so that it can be presented in the form of software based on such a readable storage medium.

[0065] Specifically, the computer program is executed on a computer device according to the following steps:

[0066] Exporting data containing entities and relationships from a knowledge graph stored in a readable storage medium, summarizing the data, extracting valid values ​​and storing them in the readable storage medium;

[0067] Read the data exported from the knowledge graph, configure prompt words for the general large language model, use the large-parameter general large language model to convert the data exported from the knowledge graph into natural language text, and output the natural language text as a file and save it in a readable storage medium;

[0068] Read the natural language text output by the general large language model, configure prompt words for the general large language model, extract entities and relationships in the natural language text using the large-parameter general large language model, organize them into entity and relationship sets, and then output and save the entity and relationship sets in a readable storage medium;

[0069] The ROUGH method is used to compare the entity and relationship sets output by the large-parameter general language model with the data originally exported from the above knowledge graph, and the entities and relationships therein are compared to evaluate the accuracy of the large-parameter language model in completing the entity extraction and relationship extraction tasks;

[0070] Select the large-parameter universal language model with the best performance, and use its output corpus to adjust the small-parameter universal language model;

[0071] Use LoRA to adjust the small-parameter universal large language model;

[0072] Test a general large language model with small parameters and evaluate it using the ROUGH metric.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A corpus entity extraction model adjustment method based on knowledge graph, characterized in that: The following steps are involved: S1. Derive triples from the knowledge graph; S2. Input the triples into a large-parameter universal language model, and the large-parameter universal language model outputs entities and relationships; S3. Large parameter general language model outputs natural language text through entities and relations; S4. Inputting the natural language text into a large-parameter universal language model, and the large-parameter universal language model outputs entities and relationships; S5. Evaluate the entities and relations, and select the large-parameter universal language model with the highest score; S6. Using the large-parameter universal large language model to produce corpus to form a corpus; S7. Using the corpus to adjust the small parameter universal large language model; S8. Inputting the natural language text into the adjusted small parameter universal large language model to output triples; S9. Evaluate the triples and select the small-parameter universal large language model with the highest score.

2. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In the step S1, the triple is in fact triple format, and the triple is output in a certain format.

3. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In the step S2, the triples and prompt words are input into a number of large-parameter general language models, and entities and relations are output in a certain format.

4. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In the step S4, the natural language text is combined with the prompt words and input into a plurality of large-parameter universal large language models, and the entities and relationships are output in a certain format.

5. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In the steps S5 and S6, the entities and relations output by the large-parameter universal large language models are compared with the triples derived from the knowledge graph, and the large-parameter universal large language model with the highest similarity is given the highest score, and the large-parameter universal large language model with the highest score is used to produce corpus to form a corpus.

6. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In step S7, the corpus is input into several small-parameter universal large language models in the alpaca dataset format for adjustment.

7. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In the step S8, the natural language text is input into a number of adjusted small-parameter universal large language models to output triples in json format.

8. The method for adjusting a corpus entity extraction model based on a knowledge graph according to claim 1, characterized in that: In step S9, the output results of several adjusted small-parameter universal large language models are compared by using indicators such as ROUGH, and the small-parameter universal large language model with the highest indicator index has the highest score.

9. A system, comprising an execution subject, wherein a readable storage medium is provided on the execution subject, and the readable storage medium stores a computer program, characterized in that: When the computer runs the computer program, the knowledge graph-based corpus entity extraction model adjustment method as described in claims 1-8 is executed.

Images