Method for using triple verification mechanism to improve large-model knowledge question answering

Abstract

The present application relates to the technical field of knowledge question answering. Provided is a method for using a triple verification mechanism to improve large-model knowledge question answering. The method comprises: converting an original training set into a training set including knowledge graph triple information, so as to fine-tune a large model; and using the fine-tuned pre-trained large model to extract entities and relationships from a question of a user, inputting a knowledge graph to generate second knowledge graph triple information, inputting the second knowledge graph triple information into the large model to obtain an initial answer, so as to form third knowledge graph triple information, comparing the second knowledge graph triple information with the third knowledge graph triple information, and if the second knowledge graph triple information is similar to the third knowledge graph triple information, outputting a final answer, and if the second knowledge graph triple information is not similar to the third knowledge graph triple information, outputting a final answer after manual verification.

Description

Method for improving large model knowledge question answering by applying triple correction mechanism

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202411252361.5, filed on September 9, 2024, and entitled "Method for improving large model knowledge question answering by applying triple correction mechanism", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application belongs to the technical field of knowledge question answering, and specifically relates to a method for improving large model knowledge question answering by applying a triple correction mechanism. BACKGROUND

[0004] High-precision fields such as medicine and law have very high requirements for the accuracy and reliability of information acquisition. When conducting knowledge question answering in the above-mentioned fields, it is necessary to ensure that the answers are accurate and traceable. However, large models are black box models with poor interpretability, and cannot guarantee that the answers obtained are real and reliable. At the same time, in some specific professional fields or classified fields, due to the lack of data available for pre-training, the professionalism of large models is greatly compromised, and the quality of answers is not satisfactory. Although the knowledge graph based on structured knowledge base can improve the accuracy and referenceability of answers, its knowledge coverage is limited and it lacks creativity and completeness.

[0005] Therefore, there are certain limitations in using large models or knowledge graphs alone for knowledge question answering. Although there are many studies on the integration of knowledge graphs and large models for knowledge question answering, simply integrating the two still cannot guarantee the accuracy of knowledge question answering. SUMMARY

[0006] The present application provides a method for improving large model knowledge question answering by applying a triple correction mechanism to solve the problem that existing large models or knowledge graphs cannot guarantee the accuracy of knowledge answering.

[0007] The technical solution adopted by the present application is as follows:

[0008] The present application provides a method for improving large model knowledge question answering by applying a triple correction mechanism, which includes:

[0009] Constructing a knowledge graph according to the extraction results of expert prior knowledge, selecting a pre-trained large model to construct a training set question and answer pair for the knowledge graph, and fine-tuning the pre-trained large model to form a joint knowledge extraction model; forming a thought chain according to the first knowledge graph triple information extracted by the pre-trained large model, and using it to construct a large model; converting the original training set into a training set containing knowledge graph triple information to fine-tune the large model;

[0010] The fine-tuned pre-training large model is used to extract entities and relationships in the user question, and the knowledge graph is input to generate second knowledge graph triple information. The second knowledge graph triple information is input into the large model to obtain an initial answer, so as to form third knowledge graph triple information. The second knowledge graph triple information is compared with the third knowledge graph triple information. If the second knowledge graph triple information is similar to the third knowledge graph triple information, the final answer is output. If the second knowledge graph triple information is not similar to the third knowledge graph triple information, the final answer is output after artificial correction.

[0011] The application provides a method for improving large model knowledge question and answer by using a triple correction mechanism, which further comprises the following additional technical features:

[0012] According to the extraction result of the expert prior knowledge, a knowledge graph is constructed, a pre-training large model is selected to construct a training set of question and answer pairs for the knowledge graph, and the pre-training large model is fine-tuned to form a joint knowledge extraction model, specifically:

[0013] According to the expert prior knowledge in the field, the entities in the knowledge graph, the relationships between the entities and the related attributes are determined as the original triples of the knowledge graph;

[0014] A pre-training large model is selected to enhance the recognition ability of non-Chinese characters, and the vocabulary table is modified to enhance the recognition ability of special text;

[0015] A series of question and answer pairs are constructed according to the content of the knowledge graph, and the question and answer pairs are used for subsequent pre-model training;

[0016] The pre-training large model is fine-tuned using the constructed question and answer pairs. The fine-tuning is performed in a supervised learning manner, and labeled training data is used to optimize the model parameters.

[0017] According to an embodiment of the application, the first knowledge graph triple information extracted according to the pre-training large model forms a thought chain, which is used to construct a large model, specifically:

[0018] The fine-tuned pre-training large model is used to extract first knowledge graph triples from the knowledge graph to form a thought chain, and the training set containing knowledge graph triple information is converted into a prompt training set, and the large model is fine-tuned using the prompt training set.

[0019] According to an embodiment of the application, the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model, specifically:

[0020] Knowledge triples are extracted from the knowledge graph using a pre-training large model;

[0021] According to the extracted triple information, a prompt for the large model is constructed; the prompt is used to guide the large model to generate an answer related to the knowledge graph;

[0022] The original training set is converted into a prompt training set containing knowledge graph triple information, and the prompt training set will be used to fine-tune the large model;

[0023] A large model based on the Transformer architecture is selected as the base model; the weights of the pre-trained large model are loaded as the starting point for fine-tuning;

[0024] The token sequence format input into the large model is defined; the target function of named entity recognition is defined, the hyperparameters are set, and gradient accumulation is used to cope with the memory limit.

[0025] According to an embodiment of the present application, the pre-trained large model after fine-tuning is used to extract entities and relationships in the user's question, and the knowledge graph is input to generate second knowledge graph triple information, which is input into the large model to obtain an initial answer to form third knowledge graph triple information, specifically:

[0026] The pre-trained large model after fine-tuning is used to extract entities and relationships from the user's question; according to the extracted entities and relationships, relevant information is searched in the knowledge graph, and knowledge graph triples are generated; these knowledge graph triples are arranged as prompts and input into the large model; the large model generates a preliminary answer according to the input prompts; entities and relationships are extracted from the preliminary answer.

[0027] According to an embodiment of the present application, the second knowledge graph triple information and the third knowledge graph triple information are compared, if the second knowledge graph triple information is similar to the third knowledge graph triple information, the final answer is output, if the second knowledge graph triple information is not similar to the third knowledge graph triple information, the final answer is output after artificial correction, specifically: entities and relationships are extracted from the preliminary answer generated by the large model, and new triples are formed; the triples of the preliminary answer are compared with the triples of the user's question; if the difference is large, it is considered that the preliminary answer may be incorrect, the knowledge graph retrieval enhancement generation technology is used for verification, and the model is optimized or artificially corrected.

[0028] According to an embodiment of the present application, the hyperparameters include but are not limited to learning rate, batch size and training round number.

[0029] According to an embodiment of the present application, the target function includes but is not limited to cross-entropy loss function.

[0030] A computer non-volatile readable storage medium, having stored thereon a program, the program being executed by a processor to implement steps of improving a large model knowledge question and answer method by applying a triple correction mechanism.

[0031] An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, the processor implementing steps of improving a large model knowledge question and answer method by applying a triple correction mechanism when executing the program.

[0032] As the above technical solutions are adopted, the application has the following beneficial effects:

[0033] By jointly pre-training a knowledge extraction model to extract entities and relationships contained in the user question, mining subgraphs from the knowledge graph, constructing knowledge graph triples, and organizing training data in the form of

instruction-knowledge graph triple-question

[0034] The accompanying drawings described herein are used to provide further understanding of the application, and form a part of the application. The illustrative embodiments of the application and their descriptions serve to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0035] Fig. 1 is a model training flowchart of a large model knowledge question and answer method improved by applying a triple correction mechanism according to an embodiment of the application;

[0036] Fig. 2 is a method flowchart of a large model knowledge question and answer method improved by applying a triple correction mechanism according to an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to more clearly explain the overall concept of the application, the following will be described in detail with reference to the accompanying drawings.

[0038] In the following description, many specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the application and the features in each embodiment can be combined with each other without conflict.

[0039] In this application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the description of the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0040] As shown in FIGS. 1-2, a method for improving large model knowledge question answering by applying a triple correction mechanism includes:

[0041] According to the extraction result of expert prior knowledge, a knowledge graph is constructed; a pre-trained large model is selected to construct a training set question and answer pair for the knowledge graph, and the pre-trained large model is fine-tuned to form a joint knowledge extraction model; according to the first knowledge graph triple information extracted by the pre-trained large model, a thinking chain is formed to construct a large model; the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model.

[0042] Specifically, according to the extraction result of expert prior knowledge, a knowledge graph is constructed: In this step, the expert prior knowledge extraction result in the field (prior knowledge refers to the knowledge formed according to the professional and experience of experts) is combined to construct a knowledge graph. This graph contains entities, relationships between entities and related attributes in a specific field. Such a graph has high professionalism and accuracy because it is constructed based on expert knowledge rather than simply extracted from online text data on the Internet.

[0043] Select a pre-trained large model to construct a training set question and answer pair for the knowledge graph, and fine-tune the pre-trained large model: Select a pre-trained large model, such as BERT (Bidirectional Encoder Representations from Transformers, pre-trained language model based on Transformer architecture), which has been pre-trained on a large amount of general data and has good language understanding ability. Next, based on the content in the knowledge graph, a task training set specifically for question answering is constructed, which contains question and answer pairs for entities and relationships in the knowledge graph. Then, the pre-trained large model is fine-tuned using this training set, with the goal of enabling the model to better perform named entity recognition and entity relationship extraction tasks, preparing for subsequent knowledge graph triple extraction.

[0044] According to the first knowledge graph triple information extracted from the pre-trained large model, a thinking chain is formed to build a large model: using the pre-trained large model that has been fine-tuned, entities and their relationships are extracted from the knowledge graph, and a knowledge graph triple of "entity-relation-entity" is formed. Then, according to these triple information, a prompt of a large model is constructed according to the thinking chain of

knowledge graph triple-instruction-question

[0045] Convert the original training set into a training set containing knowledge graph triple information to fine-tune the large model: the original training set usually contains standard data sets of natural language processing tasks. On this basis, add triple information extracted from the knowledge graph, and convert the original training set into a new training set containing these triples. This new training set is used to further fine-tune the large model, so that the model can better understand and generate high-quality text related to a specific field.

[0046] Form a joint knowledge extraction model: integrate the fine-tuned pre-trained model, knowledge graph, and related question and answer pairs to form a joint knowledge extraction model. This model not only has the powerful language understanding ability of the pre-trained model, but also can use the structured knowledge in the knowledge graph for more accurate entity recognition and relationship extraction. Apply the joint knowledge extraction model to actual knowledge question and answer tasks to ensure that the model can accurately extract relevant information from the knowledge graph according to the user's question and generate correct answers.

[0047] For example, construct a knowledge graph according to the extraction results of expert prior knowledge: assume that a team of experts in the legal field is responsible for constructing a legal knowledge graph. This graph includes legal provisions, cases, relationships between regulations, and legal terminology entities. For example, entities can include "Contract Law", "Liability for Infringement", and relationships can include "apply to", "include", etc.

[0048] Select a pre-trained large model to construct a training set of question and answer pairs for the knowledge graph, and fine-tune the pre-trained large model: choose BERT as the pre-trained large model. Next, construct question and answer pairs for entities and relationships in the knowledge graph, such as "What situations does the Contract Law apply to?". Then use these question and answer pairs to fine-tune the BERT model so that it can better understand legal terminology and relationships, complete the task of named entity recognition and relationship extraction.

[0049] According to the first knowledge graph triple information extracted from the pre-trained large model, the thinking chain is formed to build the large model: using the fine-tuned BERT model to extract entities and their relationships from the knowledge graph to form a triple of "entity-relation-entity", such as "contract law-applicable to-commercial transaction". Then, according to these triple information, according to the thinking chain of

knowledge graph triple-instruction-question

[0050] Convert the original training set into a training set containing knowledge graph triple information to fine-tune the large model: Finally, the original training set is converted into a new training set containing the above knowledge graph triple information. This new training set not only contains ordinary question and answer pairs, but also adds triple information specific to the legal field. Using this new training set to fine-tune the large model based on the Transformer architecture makes the large model generate more accurate and professional legal question answers.

[0051] For example, according to the extraction results of expert prior knowledge, a knowledge graph is constructed

[0052] Example:

[0053] Domain expert: medical expert

[0054] Entity: pneumonia, cough, antibiotic, fever

[0055] Relationship: cause, treatment

[0056] Attribute: none

[0057] The constructed knowledge graph contains the following triples:

[0058] (pneumonia, cause, cough)

[0059] (pneumonia, cause, fever)

[0060] (pneumonia, treatment, antibiotic)

[0061] Select a pre-trained large model and build a training set question and answer pair

[0062] Example:

[0063] Pre-trained model: BERT Base

[0064] Question and answer pair: based on the above knowledge graph, design the question and answer pair in the training set.

[0065] Example question and answer pair:

[0066] Q: What symptoms can pneumonia cause? A: Pneumonia can cause cough.

[0067] Q: What can treat pneumonia? A: Antibiotics can treat pneumonia.

[0068] Fine-tuning a pre-trained large model

[0069] Example:

[0070] Fine-tuning process: The BERT Base model is fine-tuned using the above question-answer pair training set. This may include adjusting learning rate, batch size, and other hyperparameters, and running multiple epochs to optimize the model.

[0071] Forming a joint knowledge extraction model

[0072] Example:

[0073] Model integration: The fine-tuned BERT model is combined with the previously constructed knowledge graph to form a joint knowledge extraction model. When a user asks a question about pneumonia, the model first tries to find relevant information from the knowledge graph and uses the fine-tuned BERT model to understand and generate answers.

[0074] Practical application example:

[0075] User question: "If I have pneumonia, what symptoms might I have?"

[0076] The joint knowledge extraction model identifies the entity "pneumonia" in the question and finds the associated symptom entities in the knowledge graph. The model finds the entities "cough" and "fever" and knows that they have a "cause" relationship with pneumonia.

[0077] The model-generated answer might be: "If you have pneumonia, you may have symptoms of cough and fever."

[0078] It is also possible to consider introducing a dynamic update mechanism so that the knowledge graph can be updated in real time according to the latest research results and industry standards.

[0079] It is also possible to explore the use of more advanced model architectures, such as DeBERTa or RoBERTa, etc. In addition to simple question-answer pairs, complex query training samples such as multi-hop reasoning problems can be added to improve the model's performance in dealing with complex problems.

[0080] It is also possible to try to build higher-order relationships, such as four-tuples or even more complex graph patterns, which can help capture deeper connections between entities. In addition, the thought chain can be designed to be more flexible, allowing dynamic adjustments based on the context of the question, so that the model can more intelligently generate answers.

[0081] Additional metadata such as timestamps, confidence scores, etc. can also be added so that the model can learn not only the relationships between entities but also the timeliness and reliability of these relationships during training. In addition, an attention mechanism can be introduced to highlight the importance of certain triples, allowing the model to focus more on these key information when generating answers.

[0082] Using the fine-tuned pre-trained large model, the entities and relationships in the user's question are extracted, and the knowledge graph is input to generate second knowledge graph triple information. The second knowledge graph triple information is input into the large model to obtain an initial answer, forming third knowledge graph triple information. The second knowledge graph triple information and the third knowledge graph triple information are compared. If the second knowledge graph triple information is similar to the third knowledge graph triple information, the final answer is output. If the second knowledge graph triple information is not similar to the third knowledge graph triple information, the final answer is output after manual correction.

[0083] Specifically, the fine-tuned pre-trained large model is used to extract entities and relationships in the user's question: the user asks a question to the system, and the system first uses the fine-tuned pre-trained large model (such as the BERT model) to extract entities and their relationships in the question. This process involves named entity recognition (NER) and relation extraction (RE). To better identify entities, the present application uses the BIOE (Begin, Inside, Outside, End) entity labeling method, which labels the complete entity boundary and provides more information. At the same time, in order to enhance the model's ability to recognize non-Chinese characters, the bert-base-multilingual model is used, and the vocabulary is modified according to the special text in the field.

[0084] Generating second knowledge graph triple information: the extracted entities and relationships are used to generate "entity-relation-entity" triple information in the knowledge graph. This step selects 2-degree relationship entities related to the entities mentioned in the question, forms a subgraph, and generates multiple triples from it as part of the knowledge graph triple information.

[0085] Inputting the second knowledge graph triple information into the large model: the generated triple information will be organized into a prompt that conforms to the

knowledge graph triple-instruction-question

[0086] Obtain initial answer: The large model receives the refined prompt information and generates a preliminary answer. This process involves converting the text information into a string and vectorizing the input into the large model, which generates text based on the input.

[0087] Form third knowledge graph triple information: The preliminary answer is converted into third knowledge graph triple information, that is, entities and relationships are extracted from the answer text to form new triple information.

[0088] Compare second knowledge graph triple information with third knowledge graph triple information: Compare the triple information generated by the user question (second knowledge graph triple) with the triple information generated by the preliminary answer (third knowledge graph triple). If they are similar, the answer is considered correct; if they are not similar, there is an error or bias that needs to be further corrected.

[0089] Output final answer: If the two triple information is similar, output the preliminary answer as the final answer; if not, the answer needs to be corrected by manual correction or the model needs to be optimized, and then the final answer is output.

[0090] For example, using the fine-tuned pre-trained large model to extract entities and relationships in the user's question: The user asks, "What are the common symptoms of pancreatitis?" The fine-tuned pre-trained large model (such as the BERT model) identifies "pancreatitis" as a disease entity and "symptoms" as an attribute, and determines the relationship between them.

[0091] Generate second knowledge graph triple information: Based on the identified entities and relationships, extract relevant subgraphs from the medical knowledge graph to generate triple information such as "pancreatitis - has symptoms - abdominal pain" and "pancreatitis - has symptoms - nausea".

[0092] Input second knowledge graph triple information into large model: Organize these triple information into a prompt and input it into the large model based on the Transformer architecture. This prompt may be as follows: "Please list the symptoms of pancreatitis: pancreatitis - has symptoms - abdominal pain, pancreatitis - has symptoms - nausea."

[0093] Obtain initial answer: The large model generates a preliminary answer based on the input prompt, for example: "The common symptoms of pancreatitis include abdominal pain, nausea, vomiting, and fever."

[0094] Form third knowledge graph triple information: Extract entities and relationships from the preliminary answer to form "pancreatitis - has symptoms - abdominal pain", "pancreatitis - has symptoms - nausea", "pancreatitis - has symptoms - vomiting", and "pancreatitis - has symptoms - fever".

[0095] Compare the second knowledge graph triple information with the third knowledge graph triple information: Compare the two triple sets and check for inconsistencies. In this example, the preliminary answer contains additional mentions of "vomiting" and "fever" beyond the "abdominal pain" and "nausea" already confirmed in the knowledge graph. If the two triple sets are sufficiently close, then the preliminary answer is considered reasonable.

[0096] Output the final answer: Since the preliminary answer is largely consistent with the information in the knowledge graph, the system can directly output the final answer: "Common symptoms of pancreatitis include abdominal pain, nausea, vomiting, and fever." If the preliminary answer differs significantly from the information in the knowledge graph, it needs to be manually corrected to ensure the accuracy of the information before outputting the final answer.

[0097] Further adjustments can be made to the pre-trained large model to better adapt to the language style and professional terminology of specific domains. For example, more domain-specific corpora can be introduced to fine-tune the model, enabling it to better identify and understand domain-specific entities and relationships.

[0098] A system can also be designed to continuously update and calibrate the knowledge graph triples during user-system interactions, making the final answer more accurate.

[0099] The knowledge graph can also have real-time or periodic updating functions to reflect the latest research findings or changes. In this way, when new information is added to the knowledge graph, the system can automatically adjust its answer strategy.

[0100] When extracting entities and relationships from user questions, not only the current question but also the previous dialogue history should be considered to better understand the background and intent of the question.

[0101] In addition to the knowledge graph, other types of data sources such as literature databases and expert opinions can be introduced to enrich the triple information, making the generated answer more comprehensive.

[0102] When the second knowledge graph triple information and the third knowledge graph triple information are found to be not similar, in addition to manual correction, an automated process can also be designed to optimize the model, such as through reinforcement learning, to allow the model to learn from mistakes and improve future answers.

[0103] For each generated answer, the system can calculate a trust score based on factors such as the source of information in the knowledge graph, update frequency, etc., to help users judge the reliability of the answer.

[0104] The application provides an application of a triple correction mechanism to improve a large model knowledge question and answer method, which further includes the following additional technical features:

[0105] According to the extraction results of expert prior knowledge, a knowledge graph is constructed; a pre-trained large model is selected to construct a training set of question and answer pairs, and the pre-trained large model is fine-tuned, specifically:

[0106] According to the extraction of expert prior knowledge in the field, the entities, relationships between entities and related attributes in the knowledge graph are determined as the original triples of the knowledge graph;

[0107] A pre-trained large model is selected to enhance the recognition ability of non-Chinese characters, and the vocabulary is modified to enhance the recognition ability of special text;

[0108] According to the content of the knowledge graph, a series of question and answer pairs are constructed, which are used for subsequent pre-model training;

[0109] The constructed question and answer pairs are used to fine-tune the pre-trained large model, and the fine-tuning is performed through supervised learning, using labeled training data to optimize model parameters.

[0110] Specifically, the knowledge graph is constructed: this step is completed by experts in the field based on their professional experience and knowledge. Experts will define entities, relationships between entities and associated attributes in the knowledge graph. The construction of the knowledge graph is based on the accurate extraction of the professional knowledge defined by experts in the field, which means that it contains the information points considered important by experts, which can represent the core concepts and connections of the field.

[0111] Select a pre-trained large model: after selecting the knowledge graph, the next step is to select a pre-trained large model. The pre-trained large model mentioned here should be able to enhance the recognition ability of non-Chinese characters, which means that when selecting the model, it is inclined to use a multilingual version of the model, such as bert-base-multilingual model, rather than a single language model such as bert-base-Chinese. In addition, the content of the vocabulary (vocab) will be modified to enhance the model's recognition ability of special text in specific fields.

[0112] Construct question and answer pairs: next, a series of question and answer pairs are created according to the content of the knowledge graph. These question and answer pairs will be used to train the pre-trained large model, so that the model can better understand and process questions and answers in a specific field. The design of question and answer pairs is based on the information contained in the knowledge graph, aiming to cover as many knowledge points and application scenarios as possible.

[0113] Fine-tuning of pre-trained large models: The pre-trained large model is fine-tuned using the constructed question and answer pairs. This fine-tuning process is done through supervised learning, which uses data with correct answer labels to train the model, aiming to optimize the model's parameters so that it can more accurately identify entities, relationships, and answer questions. Supervised learning guides the model's learning process by providing known correct input-output pairs, thereby improving the model's performance on specific tasks.

[0114] In some embodiments of the present application, the first knowledge graph triple information extracted from the pre-trained large model is used to form a thinking chain to build a large model, specifically:

[0115] The first knowledge graph triple is extracted from the knowledge graph using the fine-tuned pre-trained large model, and the thinking chain is formed. The training set containing knowledge graph triple information is converted into a prompt training set, and the large model is fine-tuned using the prompt training set.

[0116] Specifically, the first knowledge graph triple is extracted from the knowledge graph using the fine-tuned pre-trained large model

[0117] Specific steps: Select a pre-trained large model: Use a pre-trained large model that has been fine-tuned for a specific domain (such as the BERT model), which has been optimized to better identify and extract entities and relationships in that specific domain.

[0118] Extract entities and relationships: Use the fine-tuned pre-trained large model to extract entities and their relationships from the constructed knowledge graph. For example, in the medical field, the model may extract the relationship between "Drug A" and "treatment".

[0119] Form knowledge graph triples: Form the extracted entities and their relationships into the form of "entity-relation-entity", i.e. knowledge graph triples. For example, "Drug A-treatment-disease B".

[0120] Form a thinking chain, specific steps:

[0121] Build a thinking chain: According to the extracted first knowledge graph triple information, build a thinking chain, i.e.

knowledge graph triple-instruction-question

[0122] Example: If the question is "What disease does Drug A treat?", the thinking chain may be "Drug A-treatment-disease B-please answer what disease does Drug A treat".

[0123] Convert the training set containing knowledge graph triple information into a prompt training set

[0124] Specific steps:

[0125] Convert the training set: Convert the training set containing knowledge graph triple information into prompt training set. The purpose of this step is to enable the large model to better understand and generate answers related to the knowledge graph.

[0126] Example: Convert "Drug A - Treatment - Disease B" to "Drug A - Treatment - Disease B - Please answer what disease is treated by Drug A".

[0127] Fine-tune the large model using the prompt training set

[0128] Specific steps:

[0129] Fine-tune the large model: Fine-tune the large model using the converted prompt training set. The large model is based on the Transformer architecture, which means it can handle long sequences of text and performs well in generating text.

[0130] Example: Train the large model using the prompt training set containing "Drug A - Treatment - Disease B - Please answer what disease is treated by Drug A" to enable it to generate the correct answer according to the prompt.

[0131] In some embodiments of the present application, the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model, specifically:

[0132] Extract knowledge triples from the knowledge graph using a pre-trained large model;

[0133] According to the extracted triple information, construct a prompt for the large model; the prompt is used to guide the large model to generate answers related to the knowledge graph;

[0134] Convert the original training set into a prompt training set containing knowledge graph triple information, which will be used to fine-tune the large model;

[0135] Select a large model based on the Transformer architecture as the base model; load the pre-trained large model weights as the starting point for fine-tuning;

[0136] Define the token sequence format input into the large model; define the target function of named entity recognition, set the hyperparameters, and use gradient accumulation to cope with memory limitations.

[0137] Specifically, extract knowledge triples from the knowledge graph using a pre-trained large model

[0138] First, select a suitable pre-trained large model, such as the BERT model, which has been pre-trained to handle natural language processing tasks.

[0139] Fine-tune the pre-trained large model to recognize entities and relationships between entities in a specific domain.

[0140] Use the fine-tuned model to extract entities and their relationships from the knowledge graph, forming knowledge graph triples (i.e., in the form of "entity -- relationship -- entity").

[0141] Based on the extracted triple information, construct a prompt for the large model

[0142] According to the extracted knowledge graph triple information, design a prompt for guiding the large model to generate answers related to the knowledge graph.

[0143] This prompt should contain triple information and be able to construct a thought chain according to specific instructions and questions to ensure that the large model can generate answers consistent with the content of the knowledge graph.

[0144] Convert the original training set into a prompt training set containing knowledge graph triple information

[0145] Convert the original training data into a prompt format containing knowledge graph triple information.

[0146] These prompts not only contain questions, but also contain relevant knowledge graph triple information, so that the large model can learn how to generate correct answers based on this information.

[0147] The prompt training set will be used to fine-tune the large model

[0148] The created prompt training set is used to further fine-tune the large model to make it better at generating high-quality answers related to the knowledge graph.

[0149] Select a large model based on the Transformer architecture as the base model

[0150] Select a large model based on the Transformer architecture as the base model, such as GPT or other similar architecture.

[0151] Load the pre-trained large model weights as the starting point for fine-tuning

[0152] Before fine-tuning, load the pre-trained large model weights as the starting point, so that you can use the general language understanding ability learned by the pre-trained large model.

[0153] Define the token sequence format for input into the large model

[0154] Determining how the text input into a large model should be segmented into a sequence of tokens, which often involves tokenization and converting the text into numerical vectors that the model can understand.

[0155] Defining an objective function for named entity recognition, setting hyperparameters

[0156] Designing an objective function to guide the task of named entity recognition, which measures the difference between the model's predicted entity labels and the true labels.

[0157] Setting various hyperparameters, such as learning rate, batch size, etc., which will affect the learning process of the model.

[0158] Using gradient accumulation to deal with memory constraints

[0159] During the training process, due to memory constraints, it may not be possible to process a large amount of data at once. Gradient accumulation is a technique that allows the model to accumulate gradients on multiple small batches and then update the model parameters, effectively utilizing limited memory resources.

[0160] In some embodiments of the present application, the pre-trained large model is fine-tuned to extract entities and relationships from user questions, and the knowledge graph is input to generate second knowledge graph triple information, and the second knowledge graph triple information is input into the large model to obtain an initial answer, to form third knowledge graph triple information, specifically:

[0161] Using the fine-tuned pre-trained large model to extract entities and relationships from the user's question; according to the extracted entities and relationships, find related information in the knowledge graph, and generate knowledge graph triples; organize these knowledge graph triples into prompts and input them into the large model; the large model generates a preliminary answer according to the input prompt; extract entities and relationships from the preliminary answer.

[0162] Specifically, the fine-tuned pre-trained large model is used to extract entities and relationships from the user's question:

[0163] First, use the pre-trained large model (such as BERT) that has been fine-tuned to analyze the sentence asked by the user.

[0164] The model will identify entities in the sentence (such as names, places, organizations, etc.) and the relationships between these entities (such as belonging to, located in, caused by, etc.).

[0165] Entity recognition uses the BIOE format, which is a format for marking entity boundaries, where B represents the beginning of an entity, I represents the middle part of an entity, O represents a non-entity part, and E represents the end of an entity.

[0166] Based on the extracted entities and relationships, relevant information is searched in the knowledge graph, and knowledge graph triples are generated:

[0167] Based on the entities and relationships extracted from the user's question, the knowledge graph is queried to obtain relevant triple information.

[0168] Triple usually refers to the form of "entity-relation-entity".

[0169] The results of the search may include direct entities and their relationships related to the user's question, and may even include secondary relationship entities to ensure the comprehensiveness of the information.

[0170] These knowledge graph triples are sorted into prompts and input into the large model:

[0171] The triple information extracted from the knowledge graph will be sorted into a structured prompt, which is constructed in the pattern of

knowledge graph triple - instruction - question

[0172] The prompt will be input into the large model, with the purpose of guiding the large model to generate answers based on the provided knowledge graph information.

[0173] The large model generates a preliminary answer based on the input prompt:

[0174] The large model receives the prompt containing knowledge graph triple information and generates a preliminary answer based on it.

[0175] This process involves converting text information into token sequences and then vectorizing them into the large model.

[0176] Entities and relationships are extracted from the preliminary answer again:

[0177] After the large model generates a preliminary answer, the preliminary answer needs to be analyzed and entities and relationships are extracted from it again.

[0178] The extracted entities and relationships will be restructured into triple form, becoming the third knowledge graph triple information.

[0179] In some embodiments of the present application, the second knowledge graph triple information is compared with the third knowledge graph triple information. If the second knowledge graph triple information is similar to the third knowledge graph triple information, the final answer is output. If the second knowledge graph triple information is not similar to the third knowledge graph triple information, the final answer is output after artificial correction. Specifically, entities and relationships are extracted from the preliminary answer generated by the large model, and new triples are formed. The triples of the preliminary answer are compared with the triples of the user question. If the two are significantly different, it is considered that the preliminary answer may be incorrect, and the knowledge graph retrieval augmented generation technology is used for verification, and the model is optimized or artificially corrected.

[0180] Specifically, entities and relationships are extracted from the preliminary answer generated by the large model, and new triples are formed:

[0181] After the large model generates a preliminary answer based on the knowledge graph triples, further analysis of this preliminary answer is needed. This involves using a pre-trained model to identify entities (such as names of people, places, etc.) and relationships between these entities (such as affiliation, location, etc.) from the preliminary answer. The identified entities and relationships will be organized into triples in the form of "entity-relation-entity".

[0182] The triples of the preliminary answer are compared with the triples of the user question:

[0183] After the preliminary answer is generated, the triples extracted from the answer will be compared with the triples extracted from the user question previously. This comparison is to ensure that the answer generated by the large model is consistent with the original question asked by the user in terms of content.

[0184] If the two are significantly different, it is considered that the preliminary answer may be incorrect, and the knowledge graph retrieval augmented generation technology is used for verification, and the model is optimized or artificially corrected:

[0185] If there is a significant difference between the triples of the preliminary answer and the triples of the user question, it means that the large model may not have correctly understood or answered the user's question. At this time, the Knowledge Graph Retrieval Augmented Generation (KG-RAG) technology is used to verify the accuracy of the answer. KG-RAG technology can help confirm whether the entities and relationships in the preliminary answer are correct. If errors or inconsistencies are found, the answer can be corrected through model optimization or manual intervention to ensure that the final answer is both accurate and reliable.

[0186] In some embodiments of the present application, the hyperparameters include but are not limited to learning rate, batch size, and number of training rounds.

[0187] Specifically, Learning Rate: This refers to the speed at which model parameters are updated during training. A higher learning rate means that the parameters change more dramatically with each gradient descent, potentially causing the model to overshoot the minimum point when searching for the global minimum. Conversely, a lower learning rate may allow the model to more accurately approach the minimum, but the training time will be longer. Choosing the right learning rate is crucial for both the convergence speed and the final performance of the model.

[0188] Batch Size: When training neural networks, data is usually fed into the model in batches. Batch size refers to the number of samples in each batch. Larger batch sizes allow the model to iterate faster, but due to computational resource constraints, it is not possible to increase indefinitely. Smaller batch sizes can make the training process more stable, as the noise in gradient estimates is greater, helping to escape local optima, but can also lead to unstable training.

[0189] Number of Epochs: An epoch is a process of performing a forward pass and a backward pass on the entire training dataset. The number of epochs determines how many times the model will iterate over the entire dataset. Too many epochs can lead to overfitting, where the model performs well on training data but has poor generalization ability on new data; too few epochs can lead to underfitting, where the model fails to fully learn the patterns in the data.

[0190] In some embodiments of the present application, the objective function includes but is not limited to the cross-entropy loss function.

[0191] Specifically, Cross-Entropy Loss Function: This is a loss function used to evaluate the difference between the model's predicted results and the actual labels. It is widely used in classification tasks, especially in multi-class classification problems. In binary or multi-class classification problems, the cross-entropy loss function can effectively measure the difference between the model's predicted probability distribution and the actual labels.

[0192] The first step is to perform the pre-step of knowledge question answering, i.e. the construction of the knowledge graph, the pre-training of the large model and the fine-tuning of the large model. The implementation roadmap of this step is shown in Figure 1.

[0193] First, select the domain knowledge graph constructed from the extraction results of expert prior knowledge.

[0194] Then, select a suitable pre-trained large model (such as the BERT model), construct a training set of question and answer pairs for the content involved in the knowledge graph, and fine-tune the pre-trained large model so that it can complete the tasks of named entity recognition and entity relationship extraction, preparing for the original knowledge graph triple extraction.

[0195] Subsequently, according to the knowledge graph triple information extracted by the pre-trained large model, a large model prompt is constructed according to the thought chain of

knowledge graph triple-instruction-question

[0196] The original training set is converted into a prompt training set containing knowledge graph triple information, and the base large model is fine-tuned using the prompt training set. The large model is based on the Transformer architecture.

[0197] The second step is for the user to use the model proposed in the present application to perform domain knowledge question answering. The implementation roadmap of this step is shown in Figure 2, and the specific method of the model implementation technical route is as follows:

[0198] The pre-trained large model extracts entities and relationships: using the fine-tuned pre-trained large model, the entities and relationships in the user's question are extracted, and then input into the knowledge graph to generate corresponding "entity-relation-entity" knowledge graph triples.

[0199] Among them, the base model processing is greatly improved: the entity annotation method adopts BIOE format (B represents the beginning of the entity, I represents inside the entity, O represents non-entity, and E represents the end of the entity), which is more informative than the conventional BIO format, and is beneficial to model recognition and downstream task execution; Considering that the content of the knowledge graph is mainly in Chinese, supplemented by English, the bottom model weight no longer adopts the commonly used bert-base-Chinese model, but adopts the bert-base-multilingual model to enhance the model's recognition ability for non-Chinese characters; For special text in the field, the content of the vocabulary vocab is modified to enhance the model's recognition ability for special text.

[0200] When training and calling the model, the text is divided by characters, and the input token sequence x = [x0, …, x n ]

[0201] Among them, x0 and x n are the tokens of [CLS] and [SEP], respectively, and [x1, …, x n-1 ] is the text sequence token. The corresponding output sequence is H = [H0, …, H n ]

[0202] Then the objective function of named entity recognition is

[0203] And

[0204] where softmax is a function that converts a set of real numbers into a probability distribution; represents the sequence of the k-th entity predicted by the model; h k is the hidden state of the special token [CLS] in the pre-trained large model, which is used as the feature representation for entity classification; W s is a weight matrix used to map the hidden state of the model to the score of the entity label; b s is a bias term, which adds a constant offset to the score of each entity label.

[0205] Therefore, the overall goal of the model is: based on statistical analysis, for a given input sequence x, select the entity label sequence that results in the maximum conditional probability value as the optimal answer.

[0206] Generate knowledge graph triple prompt: To ensure the comprehensiveness of the information, select the entity and its 2-degree relationship entity to form a subgraph, and generate several triples containing the entities and relationships in the subgraph.

[0207] Since the large model cannot generate effective answers based on structured knowledge graph triples, a prompt is constructed according to the thought chain of

knowledge graph triple - instruction - question

[0208] Large model generates preliminary answer: After inputting the organized prompt information, the text information is converted into a string through a word segmentation tool, and then vectorized and input into the large model. The model can perform text generation, decision support, etc. Define the input token sequence as

[0209] where, is the token of the prompt containing the knowledge graph triple information. The objective function of the model's prediction result is

[0210] Therefore, similar to the pre-trained large model, the overall goal of the large model is: for the input sequence Calculate and get the y corresponding to the maximum value of the conditional probability P(y|x), and then generate the corresponding answer.

[0211] Knowledge graph traceability correction: Considering the unexplainability of the large model in the prediction process, the statistical parameters cannot accurately measure whether the result is accurate.

[0212] Therefore, after the large model generates a preliminary answer to the question raised by the user, the knowledge graph retrieval augmentation generation (KG-RAG) technology is used to verify the answer content, that is, the entities and relationships in the answer text are extracted and composed into triples, and the triples are compared with the triples extracted from the initial question. If the two sets of information are far apart, the model is optimized or manually corrected. Through knowledge graph tracing correction, the illusion problem of the large model can be effectively alleviated, and the accuracy of the answer can be improved.

[0213] A computer non-volatile readable storage medium, having stored thereon a program, the program being executed by a processor to implement the steps of the application triple proofreading mechanism to improve the large model knowledge question and answer method.

[0214] An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, the processor executing the program to implement the steps of the application triple proofreading mechanism to improve the large model knowledge question and answer method.

[0215] The places not described in the present application can be realized by using or referring to the existing technology.

[0216] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments.

[0217] The above is only an embodiment of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. An improved method for large model knowledge question answering by applying a triple correction mechanism, characterized in that, The method comprises the following steps: According to the extraction result of the expert prior knowledge, a knowledge graph is constructed, a pre-trained large model is selected to construct a training set of question and answer pairs for the knowledge graph, and the pre-trained large model is fine-tuned to form a joint knowledge extraction model. The first knowledge graph triple information extracted by the pre-trained large model is used to form a thinking chain to construct a large model; The original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model; The pre-trained large model is used to extract entities and relationships in the user's question, and the knowledge graph is input to generate second knowledge graph triple information. The second knowledge graph triple information is input into the large model to obtain an initial answer, and a third knowledge graph triple information is formed. The second knowledge graph triple information and the third knowledge graph triple information are compared. If the second knowledge graph triple information is similar to the third knowledge graph triple information, the final answer is output. If the second knowledge graph triple information is not similar to the third knowledge graph triple information, the final answer is output after artificial correction.

2. The method of claim 1, wherein, The method of constructing a knowledge graph according to the extraction result of expert prior knowledge, selecting a pre-trained large model to construct a training set of question and answer pairs for the knowledge graph, and fine-tuning the pre-trained large model to form a joint knowledge extraction model is as follows: According to the expert prior knowledge in the field, the entities, relationships between entities and related attributes in the knowledge graph are determined as the original triples of the knowledge graph; A pre-trained large model is selected to enhance the recognition ability of non-Chinese characters and modify the vocabulary table to enhance the recognition ability of special text; A series of question and answer pairs are constructed according to the content of the knowledge graph. The question and answer pairs are configured for the subsequent pre-trained large model; The pre-trained large model is fine-tuned using the constructed question and answer pairs. The fine-tuning is performed by supervised learning. The training data with labels is used to optimize the model parameters.

3. The method of claim 2, wherein, The first knowledge graph triple information extracted by the pre-trained large model is used to form a thinking chain to construct a large model. Specifically: The pre-trained large model is used to extract the first knowledge graph triple from the knowledge graph to form a thinking chain. The training set containing knowledge graph triple information is converted into a prompt training set. The prompt training set is used to fine-tune the large model.

4. The method of claim 3, wherein, The original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model. Specifically: The pre-trained large model is used to extract knowledge triples from the knowledge graph; According to the extracted triple information, a prompt configured for the large model is constructed. The prompt is configured to guide the large model to generate answers related to the knowledge graph; The original training set is converted into a prompt training set containing knowledge graph triple information. The prompt training set is configured to fine-tune the large model; A large model based on a Transformer architecture is selected as a base model; the weights of the pre-trained large model are loaded as the starting point for fine-tuning; The token sequence format input into the large model is defined; the objective function for named entity recognition is defined, the hyperparameters are set, and gradient accumulation is used to cope with memory limitations.

5. The method of claim 4, wherein, The pre-trained large model is fine-tuned to extract entities and relationships from user questions, and the knowledge graph is input to generate second knowledge graph triple information, which is input into the large model to obtain an initial answer to form third knowledge graph triple information, specifically: The pre-trained large model is fine-tuned to extract entities and relationships from user questions, and the knowledge graph is input to generate second knowledge graph triple information, which is input into the large model to obtain an initial answer to form third knowledge graph triple information, specifically:

6. The method of claim 5, wherein, The second knowledge graph triple information and the third knowledge graph triple information are compared, and if they are similar, the final answer is output, if they are not similar, the final answer is output after artificial correction, specifically: entities and relationships are extracted from the preliminary answer generated by the large model, and new triples are formed; compare the triples of the preliminary answer with the triples of the user question; If the difference is large, it is considered that the preliminary answer is incorrect, the knowledge graph retrieval enhancement generation technology is used for verification, and the model is optimized or manually corrected.

7. The method of claim 4, wherein, The hyperparameters include learning rate, batch size, and training rounds.

8. The method of claim 4, wherein, The objective function includes a cross-entropy loss function.

9. The method of claim 1, wherein, The training set question and answer pairs include question and answer pairs for entities and relationships in the knowledge graph.

10. The method of claim 1, wherein, The pre-trained large model is BERT.

11. The method of claim 1, wherein, The knowledge graph is updated based on research results and industry standards.

12. The method of claim 1, wherein, The thought chain is adjusted based on the context of the question.

13. The method of claim 5, wherein, The large model generates a preliminary answer based on the input prompt, including converting the prompt to a string and vectorizing the input to the large model to generate a preliminary answer.

14. The method of claim 1, wherein, The pre-trained large model is a multilingual version of the model.

15. The method of claim 1, wherein, The entities include names, place names, or organization names.

16. The method of claim 1, wherein, The relationships between entities include belonging, being located, or causing.

17. The method of claim 1, wherein, The entity recognition uses BIOE format, which is a format for marking entity boundaries, where B represents the beginning of the entity, I represents the middle part of the entity, O represents the non-entity part, and E represents the end of the entity.

18. The method of claim 6, wherein, The knowledge graph retrieval enhancement generation technology is used to confirm whether the entities and relationships in the preliminary answer are correct.

19. A computer non-transitory readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps of the application triple proofreading mechanism to improve the large model knowledge question and answer method according to any one of claims 1-18.

20. An electronic device comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized by The processor implements the steps in the method for improving large model knowledge question answering by applying the triple collation mechanism as claimed in any one of claims 1-18 when executing the program.

Images