Knowledge graph auxiliary question and answer method suitable for electric power field

Abstract

The invention discloses a knowledge graph auxiliary question answering method suitable for the electric power field, relates to the field of large language models, and improves the question answering accuracy by combining structured knowledge of an electric power knowledge graph with the natural language understanding ability of a large language model. The method also provides deeper and more reliable power field knowledge for the user, further promotes the value of the knowledge graph in the practical application of the large language model, and can more effectively understand and answer professional questions, thereby improving the accuracy of answers and the confidence of evaluators, providing valuable opinions for further optimizing the model, and improving the user experience. And the direction of future work is indicated.

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a knowledge graph-assisted question-answering method applicable to the electric power field. Background Art

[0002] Large language models have shown remarkable results in many natural language processing tasks. However, when question-answering tasks involve factual questions, large language models often make factual errors in answers and reasoning due to their own hallucinations and other problems. In order to alleviate the hallucination problem of large models, research on using knowledge graphs to assist large language models in question-answering tasks has begun to emerge.

[0003] For example, in the Chinese patent literature, the publication number is CN118093820A, the publication date is May 28, 2024, and the name is a method and system for intelligent generation of knowledge query statements driven by large-model and few-sample context learning based on scene perception. By utilizing the context learning ability of a large language model, efficient knowledge query statement generation is achieved based on a small number of examples.

[0004] Although the above-mentioned existing technologies have a broad knowledge base and strong learning ability, they still face the challenge of insufficient understanding depth and accuracy in highly professional technical fields such as the power sector. The power sector involves complex technical terms, professional knowledge and strict operating procedures. It is difficult for general models to accurately grasp these professional contents and it is difficult to show good results in power triple extraction.

[0005] Moreover, as electricity is an important form of energy in today's society, research on the use of domain knowledge graphs to assist large language model question answering is very important for industry insiders and the public to understand information about electricity safety and operation. However, the power sector has high requirements for the recognition of professional terms, and it is still difficult to reconstruct the data set. Even though the entity relationship extraction performance of the large language model has been improved, the complexity of the power sector, especially the widespread synonyms of professional terms in the industry and the conceptual ambiguity caused by them, makes it difficult for existing research to be directly used in the power sector knowledge graph-assisted large language model question answering tasks, and cannot provide users with accurate and authoritative knowledge in the power sector. Summary of the invention

[0006] The purpose of the present invention is to solve the deficiencies in the above-mentioned prior art and to provide a knowledge graph-assisted question answering method suitable for the power field. This framework not only improves the performance of large language models in question answering tasks in the power field and improves the accuracy of question answering, but also provides users with more accurate and authoritative knowledge in the power field.

[0007] The present invention is achieved through the following technical solutions:

[0008] A knowledge graph-assisted question-answering method applicable to the electric power field, characterized in that it comprises the following steps:

[0009] S1. Clean and transform the data from the entity relationship annotation dataset NPICKG in the power industry to obtain the NPICKG instruction dataset suitable for fine-tuning the large language model;

[0010] S2. Based on the NPICKG instruction dataset in step S1, the large language model is fine-tuned through the low-rank adaptive PEFT method to enhance the information extraction and natural language understanding capabilities of the large language model in the power field;

[0011] S3. The fine-tuned large language model adopts the entity and relation extraction enhancement method based on contextual learning. By enhancing the sequence-annotated named entity recognition module, the user's question is used as the input text to perform the sequence-annotated named entity recognition task. Then, using the sequence-annotated named entities, the adaptive relation mapping module performs the relation triple extraction task in the form of multiple-choice questions, and obtains a knowledge graph consisting of a set of candidate triples consisting of entities, entity types, and relations.

[0012] S4. Through the semantic synonymy fusion module and the semantic similarity search and matching module, the candidate entity set and candidate relationship set most relevant to the question of the input text are retrieved from the knowledge graph;

[0013] S5. The candidate entity set and candidate relationship set in step S4 are processed through query construction and conditional recall algorithm, so that all triples most relevant to the question in the input text can be accurately retrieved in the knowledge graph, and input into the large language model together with the question in the input text to obtain the output answer.

[0014] Furthermore, the specific steps of the S1 step are: for each data in the NPICKG data set, a series of instruction data is constructed through code conversion, and each instruction data in the series of instruction data is divided into an instruction part and an output part, wherein the instruction part includes three fields: task description, mode and input.

[0015] Furthermore, the pattern field in the instruction part specifies in detail the type of SPO triples to be extracted. Each SPO triple consists of three parts: the head entity type, the relationship, and the tail entity type;

[0016] The task description field in the instruction part provides specific instructions to clearly guide the large language model to complete the task, that is, to instruct the large language model to extract SPO triples that match the predefined pattern from the given input text; for types not defined in the pattern, the large language model should return an empty list and ensure that its answer conforms to the format requirements of the JSON string;

[0017] The input field in the instruction part is the input text received by the large language model. The large language model needs to analyze and process the input text according to the above task description and mode;

[0018] The output part is the output generated by the model after processing the given input text, which contains the specific SPO triple information extracted from the text given in the input field.

[0019] Furthermore, the specific steps of fine-tuning in step S2 are:

[0020] S2.1. Decompose the original weight matrix W of the large language model into two low-rank matrices A and B, that is, W = AB, where the ranks of the low-rank matrix A and the low-rank matrix B are much smaller than the rank of the original weight matrix W;

[0021] S2.2. During the fine-tuning process, only the parameters of the low-rank matrix A and the low-rank matrix B are trained. The original weight matrix W is replaced by the product of AB, and the original weight matrix is ​​kept unchanged to achieve fine-tuning of the large language model.

[0022] Furthermore, the specific steps in step S3 are as follows: the enhanced sequence annotation entity recognition module first constructs a prompt prompt(*), takes the user's question as the input text I, and forms a prompt instruction specially designed for I, marked as Prompt(I); then, submits this prompt instruction as input to the large language model to generate a series of text sequences V = {v1, ..., v n}; Finally, the text sequence V is mapped into a series of entity recognition labels to obtain the final output O; the adaptive relationship mapping module transforms the relationship into a logically fluent natural language representation by designing a relationship mapping template, and combines the relationship converted by the relationship mapping template with the final output O to construct a set of multiple-choice questions. The generated multiple-choice questions are handed over to the large language model for processing, and the correct options selected by the large language model are mapped back to specific triples to obtain a set of candidate triples consisting of entities, entity types, and relationships.

[0023] Furthermore, the design of the Prompt (I) can be further decomposed into three components:

[0024] The first part of the task is a fixed sentence, which sets the role for the large language model and indicates the task; the second part of the task is a variable sentence, which is to mark all entities with entity type [entity type] in the given input text I to indicate the entity type to be extracted by the large language model. [Entity type] is a variable parameter, which represents the specific entity type to be extracted by the task; the third part of the task is a fixed sentence, that is, please use double brackets to mark the recognized entities on the input text I, and output the original input text I if it is not recognized;

[0025] Example, an example is a variable paragraph, consisting of a positive text pair and a negative text pair. By adding a small number of examples for context learning of the model, the expected output format of the large language model is effectively guided;

[0026] Input,The input part is a variable paragraph, which represents the text that needs to be recognized by named entities, that is, the input text I.

[0027] Furthermore, the specific steps in the S4 step are: the semantic synonymy fusion module constructs a comprehensive synonym mapping table by combing the synonyms of each entity in the knowledge graph, obtains a candidate entity relationship list formed by the entity directly mentioned in the question in the input text and other synonymous representations of the entity through the synonym mapping table, and the semantic similarity search and matching module calculates the similarity between each entity in the candidate entity relationship list and all entities in the knowledge graph and the similarity between each relationship in the candidate entity relationship list and all relationships in the knowledge graph, and obtains the most relevant candidate entity set and candidate relationship set according to descending order of similarity.

[0028] Furthermore, the calculation process of the semantic similarity search and matching module includes entity similarity calculation SimiEntities, relationship similarity calculation SimiRelations, entity sorting and screening SelecTopKEntities and relationship sorting and screening SelecTopKRelations. For each entity in the candidate entity relationship list, the function SimiEntities calculates the similarity between the entity and each entity in the knowledge graph. The function SelecTopKEntities(se, ke, te) sorts the retrieved entities according to the similarity and selects the top ke entities that are greater than the threshold te. Entity, get the final candidate entity set, where se represents the entity to be sorted, ke is the preset value, indicating the top ke of the entity in the sort, and te is the preset threshold; for each relationship in the candidate entity relationship list, the function SimiRelations calculates the similarity between the relationship and each relationship in the knowledge graph, and the function SelecTopKRelations(sr, kr, tr) sorts the retrieved entities according to the similarity, and selects the top kr relationships that are greater than the threshold tr to get the final candidate relationship set, where sr represents the relationship to be sorted, kr is the preset value, indicating the top kr of the entity in the sort, and tr is the preset threshold.

[0029] Furthermore, the processing process in step S5 is to construct an empty set T, and according to the candidate entity set and the candidate relationship set, each entity in the candidate entity set is used as a head entity and combined with each relationship in the candidate relationship set to find the tail entity, thereby forming a triple consisting of the head entity, the relationship and the tail entity, and all the found triples are added to T; each entity in the candidate entity set is used as a tail entity and combined with each relationship in the candidate relationship set to find the head entity, thereby forming a triple consisting of the head entity, the relationship and the tail entity, and all the found triples are added to T; if the above two methods fail to find relevant triples, each entity in the candidate entity set, that is, all single-hop relationships and entities related to the entity, is retrieved, and the found triples are added to T. According to the above three methods, all triples closely related to the input text problem are obtained.

[0030] Furthermore, in order to effectively integrate the triples in step S5 into the question-answering process, a knowledge navigation prompt template is designed that can convert the retrieved triples into contextual information that can be processed by a large language model, and can combine this information with the question of the input text to form an information-rich and structured input.

[0031] The beneficial effects of the present invention are as follows:

[0032] By combining the structured knowledge of the power knowledge graph with the natural language understanding ability of the large language model, the present invention not only improves the accuracy of question answering, but also provides users with more in-depth and reliable knowledge in the power field, further promoting the value of the knowledge graph in the practical application of the large language model, and can more effectively understand and answer professional questions, thereby improving the accuracy of the answers and the confidence of the evaluators. At the same time, it also provides valuable insights for further optimizing the model and points out the direction of future work. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a knowledge graph-assisted question-answering method applicable to the electric power field according to the present invention;

[0034] Figure 2 This is an example of the output format of a large language model under the ESLER method;

[0035] Figure 3 Example of template for ESLER prompt;

[0036] Figure 4 It is the ARM relationship mapping template;

[0037] Figure 5 Prompt template example for ARM;

[0038] Figure 6 It is the entity synonym mapping table of the power field of the present invention;

[0039] Figure 7 A knowledge navigation prompt template for the original large language model method of the present invention;

[0040] Figure 8 This is a knowledge navigation prompt template for the large language model + power field knowledge graph assisted question and answer of the present invention. DETAILED DESCRIPTION

[0041] Example 1

[0042] like Figure 1 As shown, a knowledge graph-assisted question answering method applicable to the power field includes the following steps:

[0043] S1. Clean and transform the entity relationship annotation dataset NPICKG from the power industry to obtain the NPICKG instruction dataset suitable for fine-tuning the large language model;

[0044] S2. Based on the NPICKG instruction dataset in step S1, the large language model is fine-tuned through the low-rank adaptive PEFT method to enhance the information extraction and natural language understanding capabilities of the large language model in the power field;

[0045] S3. The fine-tuned large language model adopts the entity and relation extraction enhancement method based on contextual learning. By enhancing the sequence-annotated named entity recognition module, the user's question is used as the input text to perform the sequence-annotated named entity recognition task. Then, using the sequence-annotated named entities, the adaptive relation mapping module performs the relation triple extraction task in the form of multiple-choice questions, and obtains a knowledge graph consisting of a set of candidate triples consisting of entities, entity types, and relations.

[0046] S4. Through the semantic synonymy fusion module and the semantic similarity search and matching module, the candidate entity set and candidate relationship set most relevant to the question of the input text are retrieved from the knowledge graph;

[0047] S5. The candidate entity set and candidate relationship set in step S4 are processed through query construction and conditional recall algorithm, so that all triples most relevant to the question in the input text can be accurately retrieved in the knowledge graph, and input into the large language model together with the question in the input text to obtain the output answer.

[0048] In this embodiment, the NPICKG instruction dataset is derived from the entity relationship annotation dataset NPICKG within the power industry. The dataset is based on a wide range of documents related to the power field, including design instructions, equipment specifications, operating manuals, and maintenance records, etc. These documents provide rich domain knowledge and detailed technical information. Through systematic collection, preprocessing, and fine annotation, the NPICKG dataset not only contains a large number of entities and relationships, but also reflects the complex interactions and operating procedures between power equipment components, providing a foundation for the large language model to deeply understand and learn knowledge in the power field. By following a strict data processing process and combining the in-depth insights of experts in the power field, the NPICKG dataset is cleaned and transformed to obtain the NPICKG instruction dataset for fine-tuning.

[0049] The PEFT method based on Low-Rank Adaptation (LoRA) is used to fine-tune the large language model to enhance the model's information extraction and natural language understanding capabilities in the power field. The core idea of ​​LoRA technology is to adjust and optimize the model's self-attention mechanism by introducing a low-rank matrix. Compared with complete retraining or large-scale parameter adjustment, this method can fine-tune the model more efficiently and accurately. By using the LoRA fine-tuning strategy, we can achieve effective customization of the model without significantly increasing the complexity of the model, so that it can better adapt to data and tasks in specific fields.

[0050] This example is mainly implemented using the open source machine learning library PyTorch developed by Facebook's artificial intelligence research team. The experimental server uses the Red Hat Enterprise Linux 8 operating system and the GPU uses a single-card NVIDIA-A100-80G GPU. CUDA 11.7 is used to accelerate deep learning operations, and Python 3.10 is selected as the programming language to ensure efficient execution of the experiment and full display of the performance of the large language model.

[0051] Example 2

[0052] This embodiment further elaborates and supplements the implementation mode of the present invention on the basis of Embodiment 1.

[0053] The specific steps of the S1 step are: for each data of the NPICKG dataset, a series of instruction data is converted and constructed through code, as shown in Table 1. Each instruction data in the series of instruction data is divided into an instruction part and an output part, wherein the instruction part includes three fields: task description, mode and input.

[0054] Table 1. Data example of NPICKG instruction data set

[0055]

[0056] The pattern field in the instruction part specifies the type of SPO triples to be extracted. Each SPO triple consists of a head entity type (subject_type), a relation (relation), and a tail entity type (subject_type). For example, the pattern may define three types of SPO triples: (power equipment, component, power component), (power station, located, geographical location), (power station, expert, person).

[0057] Considering that the long context length of a large language model will affect the performance of the model, the pattern field in each JSON string is designed to contain at most 5 types of SPO triples. Considering that the NPICKG dataset predefines 20 types of SPO triples, this means that each data in the NPICKG dataset will generate 4 instruction training data. This meticulous design not only takes into account the actual technical limitations, but also ensures the efficiency of the training process and the accuracy of the target.

[0058] The task description field in the instruction part provides specific instructions to clearly guide the large language model to complete the task. For example, the instruction may require the model to act as an expert focusing on the extraction of SPO triples of power equipment, that is, to instruct the large language model to extract SPO triples that match the predefined pattern from the given input text; for types not defined in the pattern, the large language model should return an empty list and ensure that its answer conforms to the format requirements of the JSON string;

[0059] The input field in the instruction part is the input text received by the large language model. The large language model needs to analyze and process the input text according to the above task description and pattern. For example, the input text is "The steam generator is an important component of the power station and is composed of parts such as tube sheets". The large language model needs to extract the SPO triples that meet the above pattern definition based on this text, as shown in Table 1.

[0060] The output part is the output generated by the model after processing the given input text, which includes the specific SPO triple information extracted from the text given in the input field, as shown in Table 1.

[0061] Example 3

[0062] This embodiment further elaborates and supplements the implementation mode of the present invention on the basis of Embodiment 1 or Embodiment 2.

[0063] The specific steps of fine-tuning in step S2 are:

[0064] S2.1. Decompose the original weight matrix W of the large language model into two low-rank matrices A and B, that is, W = AB, where the ranks of the low-rank matrix A and the low-rank matrix B are much smaller than the rank of the original weight matrix W;

[0065] S2.2. During the fine-tuning process, only the parameters of the low-rank matrix A and the low-rank matrix B are trained. The original weight matrix W is replaced by the product of AB, and the original weight matrix is ​​kept unchanged to achieve fine-tuning of the large language model.

[0066] Example 4

[0067] This embodiment further elaborates and supplements the implementation manner of the present invention on the basis of Embodiment 1, Embodiment 2 or Embodiment 3.

[0068] The specific steps in step S3 are as follows: the Enhanced Sequence Labeling Entity Recognition module (ESLER) first constructs a prompt prompt(*), takes the user's question as the input text I, and forms a prompt instruction specially designed for I, marked as Prompt(I); then, submits this prompt instruction as input to the large language model to generate a series of text sequences V = {v1, ..., v n}; Finally, map this text sequence V into a series of entity recognition labels to get the final output O. Specifically, by matching the entity type in the "task" and the entity content in the double brackets, we get zero, one or more entity pairs, thereby obtaining the final output O; The way to annotate the text sequence V is crucial. The annotation method must clearly separate the boundaries of the entities, and the entities must be easy to extract. At the same time, the annotated text sequence V should be easy to generate by a large language model, thereby improving the accuracy of the named entity recognition task. To meet this requirement, a new large language model output format is introduced. Figure 2 As shown in the figure, for an input sequence containing at least one entity, the identified entity should be surrounded by specific tags "((" and "))"; if the input sequence does not involve any relevant entities, the output V will be the same as the input I. This strategy converts the generation task of the large language model into a sequence labeling task. The large language model only needs to mark the entity position and copy the rest, which greatly reduces the complexity of fully encoding the label information in the generated text.

[0069] like Figure 4As shown in the figure, Adaptive Relationship Mapping (ARM) is a module that converts relations into logically fluent natural language representations by designing relation mapping templates, such as converting "place of occurrence" into "{entity i} occurs in {entity j} region". When more than two entities are identified in the text, this model will traverse all possible entity pair combinations and list all potential options. When the number of relation types is m and the number of entities is n (n>2), the number of options generated for each text is at least m×n!, which may reduce the accuracy of answering questions with large language models. To solve this problem, a pattern constraint method for refining relation types is introduced to reduce the number of candidate relation types by retrieving schema types in the NPICKG dataset. For example, when the entity types are "nuclear incident" and "text", we retrieved patterns including "nuclear incident_cause_text", "nuclear incident_leads to_text" and "nuclear incident_handling measures_text", thereby establishing "cause", "leads to" and "handling measures" as candidate relationships, and then only converting these three relationships in the relationship mapping template, thereby reducing the number of irrelevant options generated.

[0070] The relations converted by the relation mapping template are combined with the final output O to construct a set of multiple-choice questions. This method not only simplifies the complexity of the relation extraction task, but also makes full use of the advantages of the large language model in processing question-answering tasks, and improves the accuracy and efficiency of relation recognition. Figure 5 As shown, the generated multiple-choice questions are handed over to the large language model for processing, and the correct options selected by the large language model are mapped back to specific triples to obtain a set of candidate triples consisting of entities, entity types, and relations.

[0071] like Figure 3 As shown, the design of Prompt (I) can be further decomposed into three components:

[0072] The first part of the task is a fixed sentence, namely "You are an expert in named entity recognition of power stations", which sets the role for the large language model and indicates the task; the second part of the task is a variable sentence, which is to mark all entities with entity type [entity type] in the given input text I to indicate the entity type to be extracted by the large language model. [Entity type] is a variable parameter, representing the specific entity type to be extracted by the task; the third part of the task is a fixed sentence, namely, please use double brackets to mark the recognized entities on the original text. If not recognized, output the original input text I, which can ensure that the large language model outputs the answer according to our labeling method.

[0073] Example, the example part is a variable paragraph, consisting of a positive text pair and a negative text pair. By adding a small number of examples to conduct contextual learning of the model, the expected output format of the large language model is effectively guided; according to the idea of ​​contrastive learning, when the entity type is "power station component", we give an extraction example of a text containing a power station component as a positive example, and at the same time give an extraction example of a text that does not contain a power station component as a negative example. As shown in Table 2, that is, "{Input: The cooling system is one of the key components for the safe operation of the power station, which includes parts such as cooling pumps. Output: ((Cooling system)) is one of the key components for the safe operation of the power station, which includes parts such as cooling pumps.}{Input: Daya Bay Power Station is located on the Dapeng Peninsula. Output: Daya Bay Power Station is located on the Dapeng Peninsula.}". This small number of examples method can effectively guide the expected output format of the large language model.

[0074] In order to optimize the example selection strategy so that a small number of examples can more accurately reflect the specific entity types in the task, a sample template library for named entity recognition is constructed, as shown in Table 2. For each entity type, a text entry containing the entity type is selected from the NPICKG dataset as input, and its output is adjusted according to the format of double-bracket entity annotation. Such input and output pairs are used as positive examples; for each entity type, text that does not contain the entity type is selected from the NPICKG dataset as input, and the same text is copied as output. Such input and output pairs form negative examples.

[0075] Table 2. ESLER example template library

[0076]

[0077]

[0078] Input: The input part is a variable paragraph, which represents the text that needs to be recognized by named entities, that is, the input text I.

[0079] In order to compare and evaluate the entity relationship extraction capabilities of large language models, the confusion matrix is ​​used to follow the standard evaluation indicators to evaluate the entity relationship extraction performance of large language models through precision, recall and F1 score.

[0080] 1. Fine-tuning experiment

[0081] In this paper, the large language model power field triple extraction task is defined as an autoregressive generation task guided by clear instructions. This task first requires the model to interpret the instructions and identify the intention behind them. Then, the model extracts specific triple information related to the power field from the given text based on the information contained in the instructions.

[0082] Table 3. Triple extraction results of the open source large language model before and after Lora fine-tuning

[0083]

[0084] The model fine-tuning results are shown in Table 3. The results show that the LoRA fine-tuning strategy greatly improves the adaptability and effectiveness of the model for complex information extraction tasks in the power field. After fine-tuning LoRA based on the power instruction dataset, all major language models have shown significant performance improvements. Specifically, the F1 value indicators of all models have improved. For example, after fine-tuning, the F1 value of the ChatGLM-6B model increased from 4.12 to 48.09, while the F1 value of the Baichuan-13B model jumped from 12.06 to 65.79. At the same time, the F1 value of the Qwen-14B model increased by 6 times after fine-tuning.

[0085] 2. Ablation experiment of entity and relation extraction enhancement method based on context learning

[0086] The entity and relation extraction enhancement methods based on contextual learning can be applied to the pre-fine-tuning model and the post-fine-tuning model respectively, so we conduct ablation experiments on the pre-fine-tuning model and the post-fine-tuning model respectively. This experimental strategy can test the relative independence of the context method and the fine-tuning method, while allowing us to accurately evaluate the impact of each module on the model performance and the synergistic effect when they are used together. Specifically, we still use ChatGLM-6B, Baichuan-7B, Baichuan-13B, and Qwen-14B as the baseline large language model before fine-tuning. For these models, we tested three different settings: adding only the ESLER module and using the normal mode for relation extraction, and adding the ESLER module and the ARM module at the same time. In this way, we can analyze the contribution of each component separately, as well as the impact of their combination on model performance.

[0087] Table 4. Ablation experiments of ESLER and ARM large language models before fine-tuning

[0088]

[0089] The ablation experiment of the large language model before fine-tuning is shown in Table 4. By introducing ESLER and ARM to these large language models to fuse contextual information, their F1 values ​​have been improved to a certain extent, showing the enhancement of the large language model's ability to understand tasks and extract triple information. However, due to the lack of electricity knowledge and other reasons, the large language model without fine-tuning has poor performance in triple extraction tasks, with a maximum F1 value of only about 16%, and cannot be directly applied to actual extraction work. However, even after the enhancement of these modules, the performance of the large language model in complex triple information extraction tasks is still limited. This shows that on billion- and 10-billion-level large language models, methods that rely solely on contextual learning have limitations and cannot effectively solve the triple extraction problem in large language models in the power field.

[0090] Table 5. Ablation experiments of ESLER and ARM large language models after fine-tuning

[0091]

[0092] The ablation experiment of the large language model after fine-tuning is shown in Table 5. The knowledge extraction ability of the large language model after fine-tuning is significantly improved. When the ESLER and ARM modules are added to the large language model, although the exact value of the gain varies with the large language model and the module, the trend is consistent, and the F1 score is improved, which shows that these modules are effective in improving the performance of information extraction tasks. After the large language model fine-tuned by LoRA in the power field is enhanced by the ESLER+ARM module, the best two 10 billion parameter models obtain an F1 value greater than 70%, which is 7% higher than the large language model after fine-tuning, and about 60 points higher than the base large language model before fine-tuning.

[0093] Example 5

[0094] This embodiment further elaborates and supplements the implementation manner of the present invention on the basis of Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4.

[0095] The specific steps in step S4 are as follows: the semantic synonymy fusion module combs the synonyms of each entity in the knowledge graph, such as Figure 6 As shown, a comprehensive synonym mapping table is constructed, and a candidate entity relationship list formed by the entity directly mentioned in the question in the input text and other synonymous representations of the entity is obtained through the synonym mapping table. The semantic similarity search and matching module calculates the similarity between each entity in the candidate entity relationship list and all entities in the knowledge graph, and the similarity between each relationship in the candidate entity relationship list and all relationships in the knowledge graph. The most relevant candidate entity set and candidate relationship set are obtained by arranging them in descending order according to the similarity.

[0096] The calculation process of the semantic similarity search and matching module includes entity similarity calculation SimiEntities, relationship similarity calculation SimiRelations, entity sorting and screening SelecTopKEntities and relationship sorting and screening SelecTopKRelations. For each entity in the candidate entity relationship list, the function SimiEntities calculates the similarity between the entity and each entity in the knowledge graph. The function SelecTopKEntities(se, ke, te) sorts the retrieved entities according to the similarity and selects the entities with the top ke and greater than the threshold te. , get the final candidate entity set, where se represents the entity to be sorted, ke is the preset value, indicating the top ke of the entity in the sorting, and te is the preset threshold; for each relationship in the candidate entity relationship list, the function SimiRelations calculates the similarity between the relationship and each relationship in the knowledge graph, and the function SelecTopKRelations(sr, kr, tr) sorts the retrieved entities according to the similarity, and selects the top kr relationships that are greater than the threshold tr to get the final candidate relationship set, where sr represents the relationship to be sorted, kr is the preset value, indicating the top kr of the entity in the sorting, and tr is the preset threshold.

[0097] Example 6

[0098] This embodiment further elaborates and supplements the implementation manner of the present invention on the basis of Embodiment 1, Embodiment 2, Embodiment 3, Embodiment 4 or Embodiment 5.

[0099] The processing process in the S5 step is to construct an empty set T, and according to the candidate entity set and the candidate relationship set, combine each entity in the candidate entity set as the head entity with each relationship in the candidate relationship set to find the tail entity, thereby forming a triple consisting of the head entity, the relationship and the tail entity, and add all the found triples to T; combine each entity in the candidate entity set as the tail entity with each relationship in the candidate relationship set to find the head entity, thereby forming a triple consisting of the head entity, the relationship and the tail entity, and add all the found triples to T; if the above two methods fail to find relevant triples, retrieve each entity in the candidate entity set, that is, all single-hop relationships and entities related to the entity, and add the found triples to T. According to the above three methods, all triples closely related to the input text problem are obtained.

[0100] As an example, the candidate entity set contains two entities: "entity A" and "entity B", and the candidate relationship set contains one relationship: "relationship X". According to the logic of the algorithm, the combination of "entity A" and "relationship X" is operated first. The algorithm tries three queries: first, it tries to find the corresponding tail entity based on "entity A" as the head entity and "relationship X"; second, it tries to find the head entity with "entity A" as the tail entity through "relationship X"; finally, if the first two steps do not successfully retrieve any results, the algorithm will perform the third query, that is, explore all single-hop relationships and entities related to "entity A". After completing the query for the combination of "entity A" and "relationship X", the algorithm then processes the combination of "entity B" and "relationship X" and executes the same three-step query logic. This includes first trying to find the tail entity related to "relationship X" with "entity B" as the head entity, then trying to find the head entity that can be linked to "relationship X" with "entity B" as the tail entity, and finally, if none of these attempts yields results, explore all single-hop relationships and entities directly related to "entity B".

[0101] In order to effectively integrate the triples in step S5 into the question-answering process, the following Figure 8 The knowledge navigation prompt template shown in FIG. 1 can convert the retrieved triples into context information that can be processed by a large language model, and can combine this information with the question of the input text to form an information-rich and structured input. Figure 7 Compared with the knowledge navigation prompt template of the original large language model method shown in Figure 1, the knowledge navigation prompt template of the knowledge graph-assisted large language model in the power field is more complex to design. It not only requires the large language model to answer the user's question, but also requires the large language model to first confirm whether the triples it retrieved from the knowledge graph are directly related to the user's question. In this strategy, the large language model is guided to ignore information that is considered irrelevant to the question and integrate relevant and useful content when giving answers, rather than simply citing the retrieved triples. In addition, if the large language model cannot determine the user's intention or cannot provide an accurate answer, it is also required to clearly explain this to the user.

[0102] In order to comprehensively evaluate the retrieval performance of the present invention, a series of retrieval evaluation experiments were designed. We selected Baichuan-13B as the experimental model, kept other parts unchanged, and conducted comparative experiments on the semantic synonymy fusion technology (SSFT) and semantic similarity search and matching technology proposed in step S4.

[0103] First, as shown in Tables 6 and 7, we conducted a comparative experiment to evaluate the impact of semantic synonymy fusion technology (SSFT). The experiment was divided into two groups: one group used SSFT technology and the other group did not, with all other aspects unchanged. Through this comparison, we aim to clarify the role of SSFT technology in improving the relevance of retrieval results. The experiment shows that the recall rate of 6B and 13B level large language models with SSFT is 6%-7% higher than that without SSFT. The SSFT method effectively improves the ability to retrieve triples in the power field.

[0104] Table 6 Comparative experiment of SSFT technology under ChatGLM-6B model

[0105]

[0106] Table 7 Comparative experiment of SSFT technology under Baichuan-13B model

[0107]

[0108] In terms of semantic overlap metrics, to evaluate the semantic overlap between the generated answer and the real answer, we use the BERTScore scoring method to obtain the F1 score. In addition, we calculate the similarity SIM between the embedding of the baseline answer and the embedding of the generated answer.

[0109] This experiment compares the performance of three methods: a pure large language model (LLM), a large language model combined with keywords (LLM+Keyword), and our proposed large language model combined with knowledge graph assisted question answering (LLM+NKAQA) in the question and answer task in the power field.

[0110] As shown in Tables 8 and 9, the evaluation results of the three methods of LLM, LLM+keyword, and LLM+NKAQA under the ChatGLM-6B and Baichuan-7B models all showed an increasing trend. As shown in Tables 10 and 11, the LLM+NKAQA method also achieved the highest score under the Baichuan-14B and Qwen-14B models, among which the performance of the LLM+NKAQA method was 20 percentage points better than the pure LLM method on average. Tables 8-11 show the evaluation of BERT-Score and SIM of four open source large language models under different methods, and the LLM+NKAQA method achieved the highest score.

[0111] Table 8 Evaluation of semantic overlap metrics based on the ChatGLM-6B large language model

[0112]

[0113] Table 9 Evaluation of semantic overlap metrics based on the Baichuan-7B large language model

[0114]

[0115] Table 10 Evaluation of semantic overlap metrics based on the Baichuan-14B large language model

[0116]

[0117] Table 11 Evaluation of semantic overlap metrics based on the Qwen-14B large language model

[0118]

[0119] To ensure the accuracy and reliability of the evaluation, we specially invited three experts with expertise in the power field and familiar with the knowledge graph in the power field to conduct independent evaluation. The main task of the evaluator is to rank each answer based on the similarity between the generated answer and the benchmark answer. This ranking mechanism allows multiple answers to get the same ranking, and we allow the evaluator to select the "uncertain" option as an answer, indicating that it is not certain which answer is better, to determine the evaluator's confidence.

[0120] In terms of evaluation metrics, we used three key indicators: average ranking, preference rate, and uncertainty rate. Average ranking reflects the average ranking of the answers generated by each method among all evaluators; preference rate refers to the percentage of methods selected as the best answer by evaluators in all test samples; uncertainty rate measures the frequency of evaluators selecting the "uncertain" option, thus providing insights into the evaluator's confidence level. Through the comprehensive use of these indicators, we aim to comprehensively evaluate and compare the performance of different methods on question-answering tasks in the power sector, while considering the evaluator's confidence in the accuracy and reliability of the generated answers.

[0121] The accuracy and reliability of the answers generated by the pure large language model (LLM), the large language model combined with keywords (LLM+Keyword), and the large language model combined with knowledge graph assisted question answering (LLM+NKAQA) of the present invention were analyzed. The specific evaluation results are summarized in Tables 12 and 13, showing that when using the Baichuan-7B or Baichuan-13B model, at least two evaluators reached a consensus when comparing and evaluating the three methods.

[0122] Table 12. Results of manual evaluation of different methods based on the Baichuan-7B large language model

[0123]

[0124] Table 13. Results of manual evaluation of different methods based on the Baichuan-13B large language model

[0125]

[0126] Regardless of which model version is used, the present invention significantly outperforms the pure LLM method and the LLM+Keyword method in manual evaluation. This shows that when combined with the information of the knowledge graph, the generated answers are not only more accurate, but also more able to meet the expectations of expert evaluators, thereby obtaining the highest preference rate and the best average ranking. At the same time, the LLM+NKAQA method shows the lowest uncertainty rate, which means that the evaluator has a higher confidence in the answers generated by this method and can correctly distinguish whether the answers are right or wrong. The emergence of low uncertainty rates shows that the quality of the answers generated by the model for professional field questions has been significantly improved, and the evaluators can more easily identify whether they are correct or not, reducing the ambiguity in the evaluation process.

[0127] The manual evaluation results highlight the advantages of the proposed method in handling professional and complex questions. By integrating the structured information of the knowledge graph, the method can more effectively understand and answer professional questions, thereby improving the accuracy of the answers and the confidence of the evaluators.

Claims

1. A knowledge graph-assisted question answering method applicable to the electric power field, characterized by: The steps include: S1. Clean and transform the data from the entity relationship annotation dataset NPICKG in the power industry to obtain the NPICKG instruction dataset suitable for fine-tuning the large language model; S2. Based on the NPICKG instruction dataset in step S1, the large language model is fine-tuned through the low-rank adaptive PEFT method to enhance the information extraction and natural language understanding capabilities of the large language model in the power field; S3. The fine-tuned large language model adopts the entity and relation extraction enhancement method based on contextual learning. By enhancing the sequence-annotated named entity recognition module, the user's question is used as the input text to perform the sequence-annotated named entity recognition task. Then, using the sequence-annotated named entities, the adaptive relation mapping module performs the relation triple extraction task in the form of multiple-choice questions, and obtains a knowledge graph consisting of a set of candidate triples consisting of entities, entity types, and relations. S4. Through the semantic synonymy fusion module and the semantic similarity search and matching module, the candidate entity set and candidate relationship set most relevant to the question of the input text are retrieved from the knowledge graph; S5. The candidate entity set and candidate relationship set in step S4 are processed through query construction and conditional recall algorithm, so that all triples most relevant to the question in the input text can be accurately retrieved in the knowledge graph, and input into the large language model together with the question in the input text to obtain the output answer.

2. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 1, characterized in that: The specific steps of step S1 are: for each data of the NPICKG data set, a series of instruction data is constructed through code conversion, and each instruction data in the series of instruction data is divided into an instruction part and an output part, wherein the instruction part includes three fields: task description, mode and input.

3. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 2, characterized in that: The pattern field in the instruction part specifies in detail the type of SPO triples to be extracted. Each SPO triple consists of three parts: the head entity type, the relationship, and the tail entity type. The task description field in the instruction part provides specific instructions to clearly guide the large language model to complete the task, that is, to instruct the large language model to extract SPO triples that match the predefined pattern from the given input text; for types not defined in the pattern, the large language model should return an empty list and ensure that its answer conforms to the format requirements of the JSON string; The input field in the instruction part is the input text received by the large language model. The large language model needs to analyze and process the input text according to the above task description and mode; The output part is the output generated by the model after processing the given input text, which contains the specific SPO triple information extracted from the text given in the input field.

4. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 1, characterized in that: The specific steps of fine-tuning in step S2 are: S2.

1. Decompose the original weight matrix W of the large language model into two low-rank matrices A and B, that is, W=AB, where the ranks of the low-rank matrices A and B are much smaller than the rank of the original weight matrix W; S2.

2. During the fine-tuning process, only the parameters of the low-rank matrix A and the low-rank matrix B are trained. The original weight matrix W is replaced by the product of AB, and the original weight matrix is ​​kept unchanged to achieve fine-tuning of the large language model.

5. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 1, characterized in that: The specific steps of step S3 are as follows: the enhanced sequence annotation entity recognition module first constructs a prompt prompt(*), takes the user's question as the input text I, and forms a prompt instruction specially designed for I, marked as Prompt(I); then, submits this prompt instruction as input to the large language model to generate a series of text sequences V={v1,…,v n }; Finally, map the text sequence V into a series of entity recognition labels to obtain the final output O; The adaptive relationship mapping module transforms the relationship into a logically fluent natural language representation by designing a relationship mapping template, combines the relationship converted by the relationship mapping template with the final output O to construct a set of multiple-choice questions, and submits the generated multiple-choice questions to the large language model for processing. The correct options selected by the large language model are mapped back to specific triples to obtain a set of candidate triples consisting of entities, entity types, and relationships.

6. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 5, characterized in that: The design of Prompt (I) can be further decomposed into three components: The first part of the task is a fixed sentence, which sets the role for the large language model and indicates the task; the second part of the task is a variable sentence, which is to mark all entities with entity type [entity type] in the given input text I to indicate the entity type to be extracted by the large language model. [Entity type] is a variable parameter, which represents the specific entity type to be extracted by the task; the third part of the task is a fixed sentence, that is, please use double brackets to mark the recognized entities on the input text I, and output the original input text I if it is not recognized; Example, an example is a variable paragraph, consisting of a positive text pair and a negative text pair. By adding a small number of examples for context learning of the model, the expected output format of the large language model is effectively guided; Input,The input part is a variable paragraph, which represents the text that needs to be recognized by named entities, that is, the input text I.

7. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 1, characterized in that: The specific steps in the S4 step are: the semantic synonymy fusion module constructs a comprehensive synonym mapping table by combing the synonyms of each entity in the knowledge graph, obtains a candidate entity relationship list formed by the entity directly mentioned in the question in the input text and other synonymous representations of the entity through the synonym mapping table, and the semantic similarity search and matching module calculates the similarity between each entity in the candidate entity relationship list and all entities in the knowledge graph and the similarity between each relationship in the candidate entity relationship list and all relationships in the knowledge graph, and obtains the most relevant candidate entity set and candidate relationship set in descending order according to the similarity.

8. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 7, characterized in that: The calculation process of the semantic similarity search and matching module includes entity similarity calculation SimiEntities, relationship similarity calculation SimiRelations, entity sorting and screening SelecTopKEntities and relationship sorting and screening SelecTopKRelations. For each entity in the candidate entity relationship list, the function SimiEntities calculates the similarity between the entity and each entity in the knowledge graph. The function SelecTopKEntities (se, ke, te) sorts the retrieved entities according to the similarity and selects the entities with the top ke and greater than the threshold te. , the final candidate entity set is obtained, where se represents the entity to be sorted, ke is a preset value, indicating the top ke of the entity in the sorting, and te is a preset threshold; for each relationship in the candidate entity relationship list, the function SimiRelations calculates the similarity between the relationship and each relationship in the knowledge graph, and the function SelecTopKRelations (sr, kr, tr) sorts the retrieved entities according to the similarity, and selects the top kr relationships that are greater than the threshold tr to obtain the final candidate relationship set, where sr represents the relationship to be sorted, kr is a preset value, indicating the top kr of the entity in the sorting, and tr is a preset threshold.

9. The knowledge graph-assisted question-answering method applicable to the electric power field according to claim 1, characterized in that: The processing process in step S5 is to construct an empty set T, and according to the candidate entity set and the candidate relationship set, each entity in the candidate entity set is used as a head entity and combined with each relationship in the candidate relationship set to find the tail entity, thereby forming a triple consisting of the head entity, the relationship and the tail entity, and all the found triples are added to T; Each entity in the candidate entity set is taken as the tail entity and combined with each relation in the candidate relation set to find the head entity, thereby forming a triple consisting of the head entity, the relation and the tail entity, and all the found triples are added to T; if the above two methods fail to find relevant triples, each entity in the candidate entity set, that is, all single-hop relations and entities related to the entity, is retrieved, and the found triples are added to T. According to the above three methods, all triples closely related to the input text problem are obtained.

10. A knowledge graph-assisted question-answering method applicable to the electric power field as claimed in claim 9, characterized in that: In order to effectively integrate the triples in step S5 into the question-answering process, a knowledge navigation prompt template is designed to convert the retrieved triples into contextual information that can be processed by a large language model, and to combine this information with the question of the input text to form an information-rich and structured input.

Images