A method for improving large model knowledge question answering using triple proofreading mechanism

Abstract

The present application provides a method for improving large-model knowledge question-answering by applying a triple proofreading mechanism, which relates to the field of knowledge question-answering technology, including: converting an original training set into a training set containing knowledge graph triple information to fine-tune the large model; using the fine-tuned pre-trained large model to extract entities and relationships in user questions, and input the knowledge graph to generate second knowledge graph triple information, input the second knowledge graph triple information into the large model to obtain the initial answer to form third knowledge graph triple information, compare the second knowledge graph triple information with the third knowledge graph triple information, if they are similar, output the final answer, if they are not similar, output the final answer after manual proofreading.

Description

Technical Field

[0001] The present application belongs to the technical field of knowledge question answering, and specifically relates to a method for improving large-model knowledge question answering by applying a triple proofreading mechanism. Background Art

[0002] High-tech fields (such as medicine and law) have extremely high requirements for the accuracy and reliability of information acquisition. When conducting knowledge questions and answers in the above fields, it is necessary to ensure that the answers are accurate and traceable. However, as a black box model, the big model has poor interpretability and cannot guarantee that the answers obtained are true and reliable. At the same time, in some specific professional fields or confidential fields, due to the lack of data available for pre-training, the professionalism of the big model is greatly reduced, and the quality of the answers is unsatisfactory. Although the knowledge graph based on the structured knowledge base can improve the accuracy and testability of the answers, its knowledge coverage is limited, and its creativity and completeness are insufficient.

[0003] Therefore, whether using large models or knowledge graphs alone for knowledge question answering, there are certain limitations. There are also many studies on the fusion of knowledge graphs and large models to achieve knowledge question answering, but it is still difficult to guarantee the accuracy of knowledge question answering by only doing graph-model fusion work. Summary of the invention

[0004] The present application provides a method for improving large-model knowledge question answering by applying a triple proofreading mechanism, so as to solve the problem that existing large models or knowledge graphs are difficult to ensure the accuracy of knowledge answers.

[0005] The technical solution adopted in this application is:

[0006] The present application embodiment provides a method for improving large model knowledge question answering by applying a triplet proofreading mechanism, including:

[0007] A knowledge graph is constructed based on the results of the extraction of expert prior knowledge, a pre-trained large model is selected to construct a training set question-answer pair for the knowledge graph, and the pre-trained large model is fine-tuned to form a joint knowledge extraction model; a thought chain is formed based on the first knowledge graph triple information extracted by the pre-trained large model to construct the large model; the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model;

[0008] Using the fine-tuned pre-trained large model, entities and relationships in the user's question are extracted, and the knowledge graph is input to generate a second knowledge graph triple information, and the second knowledge graph triple information is input into the large model to obtain an initial answer to form a third knowledge graph triple information, and the second knowledge graph triple information is compared with the third knowledge graph triple information. If they are similar, the final answer is output; if they are not similar, the final answer is output after manual proofreading.

[0009] The present application provides a method for improving large-model knowledge question answering by applying a triplet proofreading mechanism, which also includes the following additional technical features:

[0010] The method of constructing a knowledge graph based on the extraction results of expert prior knowledge, selecting a pre-trained large model to construct a training set question-answer pair for the knowledge graph, and fine-tuning the pre-trained large model to form a joint knowledge extraction model is as follows:

[0011] Extracting and determining entities, relationships between entities, and related attributes in the knowledge graph as original triples of the knowledge graph based on prior knowledge of experts in the field;

[0012] Select a pre-trained large model to enhance the recognition of non-Chinese characters and modify the vocabulary to enhance the recognition of special texts;

[0013] Constructing a series of question-answer pairs according to the content of the knowledge graph, wherein the question-answer pairs are used for subsequent pre-model training;

[0014] The pre-trained large model is fine-tuned using the constructed question-answer pairs, and the fine-tuning is performed by supervised learning, using labeled training data to optimize model parameters.

[0015] According to an embodiment of the present application, the first knowledge graph triple information extracted from the pre-trained large model forms a thought chain to construct a large model, specifically:

[0016] Use the fine-tuned pre-trained large model to extract the first knowledge graph triples from the knowledge graph to form a thought chain, convert the training set containing the knowledge graph triple information into a prompt training set, and use the prompt training set to fine-tune the large model.

[0017] According to one embodiment of the present application, the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model, specifically:

[0018] Extract knowledge triples from the knowledge graph using a pre-trained large model;

[0019] Based on the extracted triple information, a prompt for the large model is constructed; the prompt is used to guide the large model to generate answers related to the knowledge graph;

[0020] Convert the original training set into a prompt training set containing knowledge graph triple information, which will be used to fine-tune the large model;

[0021] Select a large model based on the Transformer architecture as the base model; load the weights of the pre-trained large model as the starting point for fine-tuning;

[0022] Define the token sequence format for input into the large model; define the objective function for named entity recognition, set hyperparameters, and use gradient accumulation to cope with video memory limitations.

[0023] According to an embodiment of the present application, the pre-trained large model after fine-tuning is used to extract entities and relationships in the user's question, and the knowledge graph is input to generate second knowledge graph triple information, and the second knowledge graph triple information is input into the large model to obtain an initial answer to form third knowledge graph triple information, specifically:

[0024] Use the fine-tuned pre-trained large model to extract entities and relations from the user's questions; based on the extracted entities and relations, search for relevant information in the knowledge graph and generate knowledge graph triples; organize these knowledge graph triples into prompts and input them into the large model; the large model generates a preliminary answer based on the input prompt; extract entities and relations again from the preliminary answer.

[0025] According to one embodiment of the present application, the second knowledge graph triple information is compared with the third knowledge graph triple information, and if they are similar, the final answer is output; if they are not similar, the final answer is output after manual proofreading, specifically: entities and relationships are extracted again from the preliminary answer generated by the large model to form new triples; the triples of the preliminary answer are compared with the triples of the user's question; if the difference between the two is large, it is considered that the preliminary answer may be wrong, and the knowledge graph retrieval enhancement generation technology is used for verification, and model optimization or manual correction is performed.

[0026] According to one embodiment of the present application, the hyperparameters include, but are not limited to, learning rate, batch size, and number of training rounds.

[0027] According to one embodiment of the present application, the objective function includes but is not limited to a cross entropy loss function.

[0028] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in the method for improving large-model knowledge question-answering by applying a triplet proofreading mechanism.

[0029] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for improving large-model knowledge question-answering by applying a triplet proofreading mechanism are implemented.

[0030] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0031] By jointly pre-training the knowledge extraction model, we extract the entities and relationships contained in the user's questions, mine subgraphs from the knowledge graph, construct knowledge graph triples, and organize them into training data of [instructions - knowledge graph triples - questions], which are input into the big model for training. This ensures the interpretability and accuracy of the initial answers given by the domain big model, and further converts the initial answers into triples, which are proofread and matched with the question triples, so as to strictly ensure that the answers are the same as the questions asked, thereby improving the accuracy of knowledge questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0033] Figure 1 A model training flow chart for improving a large model knowledge question answering method using a triplet proofreading mechanism provided in an embodiment of the present application;

[0034] Figure 2 A flow chart of a method for improving a large-model knowledge question-answering method by applying a triple proofreading mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features in each embodiment may be combined with each other without conflict.

[0037] In the present application, unless otherwise clearly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0038] Example 1

[0039] like Figure 1 to Figure 2 As shown, a method for improving large model knowledge question answering by applying a triplet proofreading mechanism includes:

[0040] Construct a knowledge graph based on the extraction results of expert prior knowledge; select a pre-trained large model to construct a training set question-answer pair for the knowledge graph, and fine-tune the pre-trained large model to form a joint knowledge extraction model; form a thinking chain based on the first knowledge graph triple information extracted by the pre-trained large model to construct the large model; convert the original training set into a training set containing knowledge graph triple information to fine-tune the large model.

[0041] Specifically, construct a knowledge graph based on the results of expert prior knowledge extraction: In this step, the results of expert prior knowledge extraction in the field (prior knowledge refers to the knowledge formed based on the expertise and experience of experts) are combined to construct a knowledge graph. This graph contains entities in a specific field, relationships between entities, and related attributes. This graph is highly professional and accurate because it is built based on expert knowledge rather than simply extracted from online text on the Internet.

[0042] Select a pre-trained large model to build a training set of question-answer pairs for the knowledge graph, and fine-tune the pre-trained large model: Select a pre-trained large model, such as BERT, which has been pre-trained on a large amount of general data and has good language understanding capabilities. Next, based on the content in the knowledge graph, build a task training set specifically for question answering, which contains question-answer pairs for entities and relations in the knowledge graph. Then, use this training set to fine-tune the pre-trained large model, with the goal of enabling the model to better complete the tasks of named entity recognition and extraction of relations between entities, and prepare for the subsequent knowledge graph triple extraction.

[0043] Based on the first knowledge graph triple information extracted by the pre-trained large model, a thought chain is formed to build the large model: using the fine-tuned pre-trained large model, entities and their relationships are extracted from the knowledge graph, and a knowledge graph triple of "entity-relationship-entity" is formed. Then, based on these triple information, a prompt of a large model is constructed according to the thought chain of [knowledge graph triple-instruction-question]. This prompt will help the large model understand the context of the question and generate a more accurate answer based on it.

[0044] Convert the original training set into a training set containing knowledge graph triple information to fine-tune the large model: The original training set usually contains a standard dataset for natural language processing tasks. On this basis, the triple information extracted from the knowledge graph is added to convert the original training set into a new training set containing these triples. This new training set is used to further fine-tune the large model so that the model can better understand and generate high-quality text related to a specific field.

[0045] Forming a joint knowledge extraction model: Integrate the fine-tuned pre-trained model, knowledge graph, and related question-answer pairs to form a joint knowledge extraction model. This model not only has the powerful language understanding ability of the pre-trained model, but also can use the structured knowledge in the knowledge graph for more accurate entity recognition and relationship extraction. Applying the joint knowledge extraction model to actual knowledge question-answering tasks ensures that the model can accurately extract relevant information from the knowledge graph based on the user's questions and generate the correct answer.

[0046] For example, constructing a knowledge graph based on the results of extracting expert prior knowledge: Assume that a team of experts in the legal field is responsible for constructing a legal knowledge graph. This graph includes legal clauses, cases, relationships between regulations, and entities such as legal terms. For example, entities may include "contract law", "tort liability", etc., and relationships may include "applicable to", "includes", etc.

[0047] Select a pre-trained large model to construct training set question-answer pairs for the knowledge graph and fine-tune the pre-trained large model: Select BERT as the pre-trained large model. Next, construct question-answer pairs for entities and relationships in the knowledge graph, such as "What situations does contract law apply to?" These question-answer pairs are then used to fine-tune the BERT model so that it can better understand legal terms and relationships and complete the tasks of named entity recognition and relationship extraction.

[0048] According to the first knowledge graph triple information extracted by the pre-trained large model, a thought chain is formed to build the large model: the fine-tuned BERT model is used to extract entities and their relationships from the knowledge graph to form "entity-relationship-entity" triples, such as: "Contract law-applicable to-commercial transactions". Then, based on these triple information, prompts are constructed according to the thought chain of [knowledge graph triple-instruction-question]. For example, for a question "How does contract law regulate commercial transactions?" a prompt containing relevant triple information will be constructed so that the large model can better understand the context of the question.

[0049] Convert the original training set into a training set containing knowledge graph triple information to fine-tune the large model: Finally, convert the original training set into a new training set containing the above knowledge graph triple information. This new training set not only contains common question-answer pairs, but also adds triple information unique to the legal field. Use this new training set to fine-tune the large model based on the Transformer architecture, so that the large model can generate more accurate and professional answers to legal questions.

[0050] For example, constructing a knowledge graph based on the extraction results of expert prior knowledge

[0051] Example:

[0052] Field experts: Medical experts

[0053] Entity: pneumonia, cough, antibiotics, fever

[0054] Relationship: Cause, Treat

[0055] Attributes: None

[0056] The constructed knowledge graph contains the following triples:

[0057] (pneumonia, causes, cough)

[0058] (pneumonia, causing, fever)

[0059] (pneumonia, treatment, antibiotics)

[0060] Select a pre-trained large model and build a training set of question-answer pairs

[0061] Example:

[0062] Pre-trained model: BERT Base

[0063] Question-answer pairs: Based on the knowledge graph above, design question-answer pairs in the training set.

[0064] Example question-answer pair:

[0065] Q: What symptoms does pneumonia cause? A: Pneumonia can cause coughing.

[0066] Q: What can treat pneumonia? A: Antibiotics can treat pneumonia.

[0067] Fine-tuning a large pre-trained model

[0068] Example:

[0069] Fine-tuning process: Fine-tune the BERT Base model using the above question-answer pair training set. This may include adjusting hyperparameters such as learning rate, batch size, and running multiple epochs to optimize the model.

[0070] Forming a joint knowledge extraction model

[0071] Example:

[0072] Model integration: The fine-tuned BERT model is combined with the previously constructed knowledge graph to form a joint knowledge extraction model. When a user asks a question about pneumonia, the model first tries to find relevant information from the knowledge graph and uses the fine-tuned BERT model to understand and generate answers.

[0073] Practical application examples:

[0074] User asked: "If I get pneumonia, what symptoms might I have?"

[0075] The joint knowledge extraction model identifies the entity "pneumonia" in the question and searches for the associated symptom entity in the knowledge graph. The model finds the entities "cough" and "fever" and knows that they have a "cause" relationship with pneumonia.

[0076] A model-generated response might be: “If you have pneumonia, you’d probably have a cough and a fever.”

[0077] It is also possible to consider introducing a dynamic update mechanism so that the knowledge graph can be updated in real time based on the latest research results and industry standards.

[0078] You can also explore the use of more advanced model architectures, such as DeBERTa or RoBERTa, etc. In addition, in addition to simple question-answer pairs, you can also increase the training samples of complex queries, such as multi-hop reasoning problems, to improve the performance of the model when facing complex problems.

[0079] You can also try to build higher-order relationships, such as quadruples or even more complex graph patterns, which help capture deeper connections between entities. In addition, the thought chain can be designed to be more flexible, allowing dynamic adjustments based on the context of the question, so that the model can generate answers more intelligently.

[0080] You can also add additional metadata, such as timestamps, confidence scores, etc., so that during training the model can not only learn the relationships between entities, but also understand the timeliness and reliability of these relationships. In addition, you can introduce an attention mechanism to highlight the importance of certain triplets, so that the model pays more attention to these key information when generating answers.

[0081] Using the fine-tuned pre-trained large model, entities and relations in the user's question are extracted, and the knowledge graph is input to generate a second knowledge graph triple information. The second knowledge graph triple information is input into the large model to obtain an initial answer to form a third knowledge graph triple information. The second knowledge graph triple information is compared with the third knowledge graph triple information. If they are similar, the final answer is output. If they are not similar, the final answer is output after manual proofreading.

[0082] Specifically, the entities and relations in the user's question are extracted using a fine-tuned pre-trained large model: the user asks the system a question, and the system first uses a fine-tuned pre-trained large model (such as the BERT model) to extract the entities in the question and the relations between them. This process involves named entity recognition (NER) and relation extraction (RE). In order to better identify entities, the present invention adopts the BIOE (Begin, Inside, Outside, End) entity annotation method, which annotates the complete entity boundaries and is more informative. At the same time, in order to enhance the model's recognition ability for non-Chinese characters, the bert-base-multilingual model is used, and the vocabulary is modified according to the domain-specific text.

[0083] Generate the second knowledge graph triple information: The extracted entities and relations are used to generate "entity-relationship-entity" triple information in the knowledge graph. This step selects the second-degree relationship entities related to the entities mentioned in the question to form a subgraph, and generates multiple triples from it as part of the knowledge graph triple information.

[0084] Input the second knowledge graph triple information into the big model: The generated triple information will be organized into a prompt that conforms to the [knowledge graph triple - instruction - question] thinking chain and input into the big model. Since it is difficult for the big model itself to directly generate effective answers through structured triple information, a prompt is constructed to guide the big model to understand and generate answers.

[0085] Get the initial answer: After receiving the sorted prompt information, the big model generates a preliminary answer. This process involves converting the text information into a string and vectorizing it into the big model, which then generates the text.

[0086] Forming the third knowledge graph triple information: The preliminary answer is converted into the third knowledge graph triple information, that is, entities and relations are extracted from the answer text to form new triple information.

[0087] Compare the triple information of the second knowledge graph with the triple information of the third knowledge graph: compare the triple information generated by the user question (the triple of the second knowledge graph) with the triple information generated by the preliminary answer (the triple of the third knowledge graph). If the two are similar, the answer is considered correct; if they are not similar, it indicates that there is an error or deviation, which requires further correction.

[0088] Output the final answer: If the information of the two triples is similar, the preliminary answer is output as the final answer; if they are not similar, it is necessary to correct the answer through manual proofreading, or optimize the model, and then output the final answer.

[0089] For example, a fine-tuned pre-trained large model is used to extract entities and relationships in user questions: The user asks: "What are the common symptoms of pancreatitis?" The fine-tuned pre-trained large model (such as the BERT model) recognizes that "pancreatitis" is a disease entity, "symptoms" is an attribute, and determines the relationship between the two.

[0090] Generate triple information of the second knowledge graph: Based on the identified entities and relationships, extract relevant subgraphs from the medical knowledge graph to generate triple information such as "pancreatitis-symptoms-abdominal pain" and "pancreatitis-symptoms-nausea".

[0091] Input the second knowledge graph triple information into the big model: Organize these triple information into a prompt and input it into the big model based on the Transformer architecture. This prompt may be like this: "Please list the symptoms of pancreatitis: pancreatitis-symptoms-abdominal pain, pancreatitis-symptoms-nausea."

[0092] Get an initial answer: The large model generates a preliminary answer based on the input prompt, for example: "Common symptoms of pancreatitis include abdominal pain, nausea, vomiting, and fever."

[0093] Forming the third knowledge graph triple information: extract entities and relations from the preliminary answer again to form "pancreatitis-symptoms-abdominal pain", "pancreatitis-symptoms-nausea", "pancreatitis-symptoms-vomiting", and "pancreatitis-symptoms-fever".

[0094] Compare the triple information of the second knowledge graph with the triple information of the third knowledge graph: Compare the two sets of triples to check for inconsistencies. In this example, the preliminary answer includes "vomiting" and "fever" in addition to "abdominal pain" and "nausea" that have been confirmed in the knowledge graph. If the two sets of triples are close enough, then the preliminary answer is considered reasonable.

[0095] Output the final answer: Since the preliminary answer is basically consistent with the information in the knowledge graph, the system can directly output the final answer: "Common symptoms of pancreatitis include abdominal pain, nausea, vomiting, and fever." If the preliminary answer is significantly different from the information in the knowledge graph, manual proofreading is required to ensure the accuracy of the information before outputting the final answer.

[0096] The pre-trained large model can also be further adjusted to make it more suitable for the language style and professional terminology of a specific field. For example, more in-field corpus can be introduced to fine-tune the model so that it can better recognize and understand entities and relationships in the field.

[0097] It is also possible to design a system that can continuously update and calibrate knowledge graph triples during the interaction between the user and the system, making the final answer more accurate.

[0098] The knowledge graph can also be updated in real time or regularly to reflect the latest research results or changes. In this way, when new information is added to the knowledge graph, the system can automatically adjust its answer strategy.

[0099] When extracting entities and relations in user questions, we can also consider not only the current question but also the previous conversation history to better understand the context and intent of the question.

[0100] In addition to knowledge graphs, other types of data sources can be introduced, such as literature databases, expert opinions, etc., to enrich the information of triples and make the generated answers more comprehensive.

[0101] When it is found that the triple information of the second knowledge graph is not similar to the triple information of the third knowledge graph, in addition to manual proofreading, an automated process can also be designed to optimize the model, such as through reinforcement learning, so that the model can learn from mistakes and improve future answers.

[0102] For each generated answer, the system can calculate a trust score based on factors such as the source of information in the knowledge graph and update frequency, to help users judge the reliability of the answer.

[0103] The present application provides a method for improving large-model knowledge question answering by applying a triplet proofreading mechanism, which also includes the following additional technical features:

[0104] Build a knowledge graph based on the results of the expert prior knowledge extraction; select a pre-trained large model to build a training set of question-answer pairs for the knowledge graph, and fine-tune the pre-trained large model, specifically:

[0105] Based on the prior knowledge of experts in the field, entities, relationships between entities, and related attributes in the knowledge graph are extracted and determined as the original triples of the knowledge graph;

[0106] Select a pre-trained large model to enhance the recognition of non-Chinese characters and modify the vocabulary to enhance the recognition of special texts;

[0107] Construct a series of question-answer pairs based on the content of the knowledge graph, which are used for subsequent pre-model training;

[0108] Use the constructed question-answer pairs to fine-tune the pre-trained large model. The fine-tuning is performed through supervised learning, using labeled training data to optimize the model parameters.

[0109] Specifically, constructing a knowledge graph: This step is completed by experts in the field based on their professional experience and knowledge. Experts will define the entities in the knowledge graph, the relationships between these entities, and the associated attributes. The construction of the knowledge graph is based on the precise extraction of professional knowledge in the field defined by the experts, which means that it contains information points that the experts consider important, which can represent the core concepts and connections in the field.

[0110] Select a pre-trained large model: After selecting the knowledge graph, the next step is to select a pre-trained large model. The pre-trained large model mentioned here should be able to enhance the recognition ability of non-Chinese characters, which means that when selecting a model, a multilingual version of the model, such as the bert-base-multilingual model, is preferred over a single language model such as bert-base-Chinese. In addition, the content of the vocabulary (vocab) will be modified to enhance the model's recognition ability for special texts in specific fields.

[0111] Build question-answer pairs: Next, create a series of question-answer pairs based on the content of the knowledge graph. These question-answer pairs will be used to train the pre-trained large model so that the model can better understand and process questions and answers in a specific field. The design of the question-answer pairs is based on the information contained in the knowledge graph, aiming to cover as many knowledge points and application scenarios as possible.

[0112] Fine-tuning of the pre-trained large model: Fine-tune the pre-trained large model using the question-answer pairs constructed above. This fine-tuning process is done through supervised learning, that is, using data with correct answer labels to train the model, with the goal of optimizing the model's parameters so that it can more accurately identify entities, relationships, and answer questions. Supervised learning guides the model's learning process by providing known correct input-output pairs, thereby improving the model's performance on specific tasks.

[0113] In some embodiments of the present application, a thought chain is formed based on the first knowledge graph triple information extracted from the pre-trained large model to construct the large model, specifically:

[0114] Use the fine-tuned pre-trained large model to extract the first knowledge graph triple from the knowledge graph to form a thought chain, convert the training set containing the knowledge graph triple information into a prompt training set, and use the prompt training set to fine-tune the large model.

[0115] Specifically, the first knowledge graph triples are extracted from the knowledge graph using the fine-tuned pre-trained large model.

[0116] Specific steps: Choose a pre-trained large model: Use a pre-trained large model that has been fine-tuned in a specific domain (such as the BERT model). This model has been optimized to better recognize and extract entities and relationships in a specific domain.

[0117] Extract entities and relationships: Use the fine-tuned pre-trained large model to extract entities and their relationships from the constructed knowledge graph. For example, in the medical field, the model may extract the relationship between "drug A" and "treatment".

[0118] Forming knowledge graph triples: The extracted entities and their relationships are organized into the form of "entity-relationship-entity", i.e., knowledge graph triples. For example, "drug A-treatment-disease B".

[0119] Form a thinking chain, specific steps:

[0120] Construct a thinking chain: Based on the extracted first knowledge graph triple information, construct a thinking chain, namely [knowledge graph triple - instruction - question]. This thinking chain can guide the large model on how to generate answers based on specific instructions and questions.

[0121] Example: If the question is “What disease does drug A treat?” then the thought chain might be “Drug A – Treatment – ​​Disease B – Please answer what disease does drug A treat.”

[0122] Convert the training set containing knowledge graph triple information into prompt training set

[0123] Specific steps:

[0124] Convert training set: Convert the training set containing knowledge graph triple information into prompt training set. The purpose of this step is to enable the large model to better understand and generate answers related to the knowledge graph.

[0125] Example: Convert "Drug A - Treatment - Disease B" to "Drug A - Treatment - Disease B - Please answer what disease does drug A treat".

[0126] Fine-tune the large model using the prompt training set

[0127] Specific steps:

[0128] Fine-tune the large model: Fine-tune the large model using the transformed prompt training set. The large model is based on the Transformer architecture, which means it is able to process long sequences of text and performs well in generating text.

[0129] Example: Use a prompt training set containing "Drug A - Treatment - Disease B - Please answer what disease does drug A treat" to train a large model so that it can generate correct answers based on the prompt.

[0130] In some embodiments of the present application, the original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model, specifically:

[0131] Extract knowledge triples from the knowledge graph using a pre-trained large model;

[0132] Based on the extracted triple information, a prompt for the large model is constructed; the prompt is used to guide the large model to generate answers related to the knowledge graph;

[0133] Convert the original training set into a prompt training set containing knowledge graph triple information. The prompt training set will be used to fine-tune the large model.

[0134] Select a large model based on the Transformer architecture as the base model; load the weights of the pre-trained large model as the starting point for fine-tuning;

[0135] Define the token sequence format for input into the large model; define the objective function for named entity recognition, set hyperparameters, and use gradient accumulation to cope with video memory limitations.

[0136] Specifically, a pre-trained large model is used to extract knowledge triples from the knowledge graph.

[0137] First, choose a suitable pre-trained large model, such as the BERT model, which has been pre-trained to handle natural language processing tasks.

[0138] Fine-tune the pre-trained large model to enable it to recognize entities in a specific domain and the relationships between entities.

[0139] Use the fine-tuned model to extract entities and their relationships from the knowledge graph to form knowledge graph triples (i.e., in the form of "entity-relationship-entity").

[0140] Based on the extracted triple information, construct a prompt for the large model

[0141] Based on the extracted knowledge graph triple information, design a prompt to guide the large model to generate answers related to the knowledge graph.

[0142] This prompt should contain triple information and be able to build a chain of thinking based on specific instructions and questions to ensure that the large model can generate answers that are consistent with the content of the knowledge graph.

[0143] Convert the original training set into a prompt training set containing knowledge graph triple information

[0144] Convert the original training data into prompt format containing knowledge graph triple information.

[0145] These prompts contain not only questions but also relevant knowledge graph triple information so that the large model can learn how to generate the correct answer based on this information.

[0146] The prompt training set will be used to fine-tune the large model

[0147] The created prompt training set is used to further fine-tune the large model to make it better at generating high-quality answers related to the knowledge graph.

[0148] Choose a large model based on the Transformer architecture as the base model

[0149] Choose a base model based on the Transformer architecture, such as GPT or other large-scale language models with similar architectures.

[0150] Load the weights of a pre-trained large model as a starting point for fine-tuning

[0151] Before fine-tuning, load the weights of a large pre-trained model as a starting point, so that you can take advantage of the general language understanding capabilities learned by the large pre-trained model.

[0152] Define the token sequence format for input into the large model

[0153] Determine how text input to a large model should be segmented into sequences of tokens, which usually involves tokenizing the text and converting it into numeric vectors that the model can understand.

[0154] Define the objective function of named entity recognition and set hyperparameters

[0155] Design an objective function to guide the task of named entity recognition, which measures the gap between the entity labels predicted by the model and the true labels.

[0156] Set various hyperparameters like learning rate, batch size, etc. which will affect the learning process of the model.

[0157] Using gradient accumulation to deal with video memory limitations

[0158] During the training process, due to the limitation of video memory, it may not be possible to process a large amount of data at one time. Gradient accumulation is a technique that allows the model to accumulate gradients over multiple mini-batches before updating the model parameters, thereby effectively utilizing limited video memory resources.

[0159] In some embodiments of the present application, the pre-trained large model after fine-tuning is used to extract entities and relationships in the user's question, and the knowledge graph is input to generate the second knowledge graph triple information, and the second knowledge graph triple information is input into the large model to obtain an initial answer to form the third knowledge graph triple information, specifically:

[0160] Use the fine-tuned pre-trained large model to extract entities and relations from the user's questions; based on the extracted entities and relations, search for relevant information in the knowledge graph and generate knowledge graph triples; organize these knowledge graph triples into prompts and input them into the large model; the large model generates a preliminary answer based on the input prompt; extract entities and relations again from the preliminary answer.

[0161] Specifically, the fine-tuned pre-trained large model is used to extract entities and relations from the user's question:

[0162] First, use a pre-trained large model that has been fine-tuned (such as BERT) to analyze the sentences asked by users.

[0163] The model will identify entities in a sentence (such as names of people, places, organizations, etc.) and the relationships between these entities (such as belongs to, is located in, caused by, etc.).

[0164] Entity recognition uses the BIOE format, which is a format for marking entity boundaries, where B represents the beginning of the entity, I represents the middle part of the entity, O represents the non-entity part, and E represents the end of the entity.

[0165] According to the extracted entities and relationships, relevant information is found in the knowledge graph and knowledge graph triples are generated:

[0166] Based on the entities and relations extracted from user questions, the knowledge graph is queried to obtain relevant triple information.

[0167] A triple is usually referred to as an entity-relationship-entity.

[0168] The search results may include direct entities related to the user's question and their relationships, and may even include entities with secondary relationships to ensure comprehensiveness of the information.

[0169] These knowledge graph triplets are organized into prompts and input into the big model:

[0170] The triple information extracted from the knowledge graph will be organized into a structured prompt, which is built according to the pattern of [knowledge graph triple - instruction - question].

[0171] This prompt will be input into the big model to guide the big model to generate answers based on the provided knowledge graph information.

[0172] The large model generates a preliminary answer based on the input prompt:

[0173] The large model receives a prompt containing knowledge graph triples and generates a preliminary answer based on it.

[0174] This process involves converting text information into a sequence of tokens, which are then vectorized and fed into a large model.

[0175] Extract entities and relations again from the preliminary answer:

[0176] After the large model generates a preliminary answer, this preliminary answer needs to be analyzed to extract entities and relationships from it again.

[0177] The extracted entities and relations will be reconstructed into triple form and become the triple information of the third knowledge graph.

[0178] In some embodiments of the present application, the second knowledge graph triple information is compared with the third knowledge graph triple information. If they are similar, the final answer is output. If they are not similar, the final answer is output after manual proofreading. Specifically, entities and relationships are extracted again from the preliminary answer generated by the large model to form new triples; the triples of the preliminary answer are compared with the triples of the user's question; if the two are significantly different, the preliminary answer is considered to be incorrect, and the knowledge graph retrieval enhancement generation technology is used for verification, and model optimization or manual correction is performed.

[0179] Specifically, entities and relations are extracted again from the preliminary answers generated by the large model to form new triples:

[0180] After the big model generates a preliminary answer based on the knowledge graph triples, it is necessary to further analyze the preliminary answer. This involves using a pre-trained model to identify entities (such as names of people, places, etc.) and the relationships between these entities (such as ownership relationships, location relationships, etc.) from the preliminary answer. The identified entities and relationships will be organized into triples, namely "entity-relationship-entity".

[0181] Compare the triples of the preliminary answer with the triples of the user's question:

[0182] After the preliminary answer is generated, the triples extracted from the answer are compared with the triples previously extracted from the user's question. This comparison is to ensure that the answer generated by the large model is consistent with the question originally asked by the user.

[0183] If the difference between the two is large, it is considered that the preliminary answer may be wrong, and the knowledge graph retrieval enhancement generation technology is used for verification, and the model is optimized or manually corrected:

[0184] If there is a large difference between the triples of the preliminary answer and the triples of the user's question, it means that the big model may not understand or answer the user's question correctly. At this time, the Knowledge Graph Retrieval Augmented Generation (KG-RAG) technology is used to verify the accuracy of the answer. KG-RAG technology can help confirm whether the entities and relationships in the preliminary answer are correct. If errors or inconsistencies are found, the answer can be corrected by optimizing the model or manual intervention to ensure that the final answer is both accurate and reliable.

[0185] In some embodiments of the present application, hyperparameters include, but are not limited to, learning rate, batch size, and number of training rounds.

[0186] Specifically, learning rate: This refers to the speed at which the model parameters are updated during training. A higher learning rate means that the parameters change more each time the gradient descends, which may cause the model to pass the lowest point when searching for the global minimum; on the contrary, a lower learning rate may allow the model to approach the minimum more accurately, but the training time will be longer. Choosing the right learning rate is crucial to the speed of model convergence and final performance.

[0187] Batch Size: When training a neural network, data is usually fed into the model in batches. Batch size refers to the number of samples in each batch of data. A larger batch size allows the model to iterate faster, but it cannot be increased indefinitely due to computing resource constraints. A smaller batch size can make the training process more stable because the gradient estimate is noisier, which helps to escape the local optimal solution, but it may also cause unstable training.

[0188] Number of Epochs: An epoch is a process of performing a forward propagation and a backward propagation on the entire training dataset. The number of epochs determines how many times the model will iterate on the entire dataset. Too many epochs may lead to overfitting, that is, the model performs well on the training data but has poor generalization ability on new data; too few epochs may lead to underfitting, that is, the model fails to fully learn the patterns in the data.

[0189] In some embodiments of the present application, the objective function includes but is not limited to a cross entropy loss function.

[0190] Specifically, the cross-entropy loss function is a loss function used to evaluate the difference between the model's prediction results and the actual labels. It is widely used in classification tasks, especially multi-class classification problems. In binary or multi-classification problems, the cross-entropy loss function can effectively measure the difference between the probability distribution predicted by the model and the actual labels.

[0191] Example 2

[0192] The first step is to perform the preparatory steps for knowledge question answering, namely, the construction of the knowledge graph, pre-training of the large model, and fine-tuning of the large model. The implementation roadmap of this step is as follows: Figure 1 shown.

[0193] First, a domain knowledge graph constructed by extracting expert prior knowledge is selected.

[0194] Then, select a suitable pre-trained large model (such as the BERT model), build a training set of question-answer pairs for the content involved in the knowledge graph, and fine-tune the pre-trained large model so that it can complete the named entity recognition and entity relationship extraction tasks, preparing for the extraction of triples in the original knowledge graph.

[0195] Then, based on the knowledge graph triple information extracted by the pre-trained large model, the large model prompt is constructed according to the thinking chain of [knowledge graph triple - instruction - question].

[0196] The original training set is converted into a prompt training set containing knowledge graph triple information, and the prompt training set is used to fine-tune the base large model, which is based on the Transformer architecture.

[0197] The second step is for users to use the model proposed by the present invention to answer domain knowledge questions. The implementation roadmap of this step is as follows: Figure 2 As shown in the figure, the specific method of implementing the technical route of the model is as follows:

[0198] Pre-trained large model extracts entities and relationships: Use the fine-tuned pre-trained large model to extract entities and relationships in user questions, and then input them into the knowledge graph to generate the corresponding "entity-relationship-entity" knowledge graph triples.

[0199] Among them, the present invention has made great improvements in the processing of the base model: the entity annotation method adopts the BIOE format (B represents the beginning of the entity, I represents inside the entity, O represents non-entity, and E represents the end of the entity). Compared with the conventional BIO format, BIOE marks the complete entity boundary and has richer information, which is convenient for the model to recognize and execute downstream tasks; considering that the content of the knowledge graph is mainly in Chinese and supplemented by English, the underlying model weight no longer adopts the commonly used BERT-base-Chinese model, but adopts the BERT-base-multilingual model to enhance the model's recognition ability for non-Chinese characters; for domain-specific texts, the content of the vocabulary vocab is modified to enhance the model's recognition ability for special texts.

[0200] When training and calling the model, the text is segmented by characters and the input token sequence is defined.

[0201]

[0202] in, and They are the tokens of [CLS] and [SEP] respectively. is the text sequence token. The corresponding output sequence is

[0203]

[0204] Then the objective function of named entity recognition is

[0205]

[0206] and

[0207]

[0208] Among them, softmax is a function that converts a set of real values ​​into a probability distribution; represents the sequence of the kth entity predicted by the model; It is the hidden state of the special tag [CLS] in the pre-trained large model, which is used as the feature representation of entity classification; is a weight matrix used to map the hidden state of the model to the score of the entity label; is a bias term that adds a constant offset to the score of each entity label.

[0209] Therefore, the overall goal of the model is: based on statistical analysis, for a given input sequence , select the entity label sequence that leads to the largest conditional probability value as the optimal answer.

[0210] Generate knowledge graph triples prompt: To ensure the comprehensiveness of information, entities and their second-degree relationship entities are selected here to form a subgraph, and the entities and relationships contained in the subgraph are generated into several triples.

[0211] Since the large model cannot well generate effective answers only through structured knowledge graph triples, a prompt is constructed according to the thinking chain of [knowledge graph triple - instruction - question]. The prompt can organize the triple information according to the given instructions and input it into the large model.

[0212] The big model generates a preliminary answer: After inputting the sorted prompt information, the text information is converted into a string through the word segmentation tool, and then vectorized and input into the big model. The model can perform tasks such as text generation and decision support. Define the input token sequence as

[0213]

[0214] in, is the token of prompt containing the knowledge graph triple information. The objective function of the model prediction result is

[0215]

[0216] Therefore, similar to the pre-trained large model, the overall goal of the large model is: for the input sequence , calculate and get the conditional probability of the maximum value The corresponding , and then generate the corresponding answer.

[0217] Knowledge graph traceability correction: Considering the unexplainable nature of large models in the prediction process, it is impossible to accurately measure the accuracy of the results only through statistical parameters.

[0218] Therefore, after the large model generates a preliminary answer to the user's question, the knowledge graph retrieval-augmented generation (KG-RAG) technology is used to verify the answer content, that is, to extract entities and relationships in the answer text and form triples, and compare them with the knowledge graph triples extracted from the initial question. If the two sets of information are far apart, the model is optimized or manually corrected. Through knowledge graph traceability correction, the hallucination problem of the large model can be effectively alleviated and the accuracy of the answer can be improved.

[0219] Example 3

[0220] A computer-readable storage medium stores a program which, when executed by a processor, implements the steps of applying a triple proofreading mechanism to improve a large-model knowledge question-answering method.

[0221] Example 4

[0222] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of applying a triplet proofreading mechanism to improve a large model knowledge question-answering method are implemented.

[0223] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0224] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0225] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for improving large-model knowledge question answering using a triplet proofreading mechanism, characterized in that: include: A knowledge graph is constructed based on the results of the expert prior knowledge extraction, a pre-trained large model is selected to construct a training set question-answer pair for the knowledge graph, and the pre-trained large model is fine-tuned to form a joint knowledge extraction model, and a thought chain is formed based on the first knowledge graph triple information extracted by the pre-trained large model to construct the large model; Converting the original training set into a training set containing knowledge graph triple information to fine-tune the large model; Using the fine-tuned pre-trained large model, entities and relationships in the user's question are extracted, and the knowledge graph is input to generate a second knowledge graph triple information, and the second knowledge graph triple information is input into the large model to obtain an initial answer to form a third knowledge graph triple information, and the second knowledge graph triple information is compared with the third knowledge graph triple information. If they are similar, the final answer is output; if they are not similar, the final answer is output after manual proofreading.

2. The method according to claim 1, characterized in that: The method of constructing a knowledge graph based on the extraction results of expert prior knowledge, selecting a pre-trained large model to construct a training set question-answer pair for the knowledge graph, and fine-tuning the pre-trained large model to form a joint knowledge extraction model is as follows: Extracting and determining entities, relationships between entities, and related attributes in the knowledge graph as original triples of the knowledge graph based on prior knowledge of experts in the field; Select a pre-trained large model to enhance the recognition of non-Chinese characters and modify the vocabulary to enhance the recognition of special texts; Construct a series of question-answer pairs based on the content of the knowledge graph, and the question-answer pairs are used for the subsequent pre-trained large model; The pre-trained large model is fine-tuned using the constructed question-answer pairs, and the fine-tuning is performed by supervised learning, using labeled training data to optimize model parameters.

3. The method according to claim 2, characterized in that The first knowledge graph triple information extracted according to the pre-trained large model forms a thought chain to construct a large model, specifically: Use the fine-tuned pre-trained large model to extract the first knowledge graph triples from the knowledge graph to form a thought chain, convert the training set containing the knowledge graph triple information into a prompt training set, and use the prompt training set to fine-tune the large model.

4. The method according to claim 3, characterized in that The original training set is converted into a training set containing knowledge graph triple information to fine-tune the large model, specifically: Extract knowledge triples from the knowledge graph using a pre-trained large model; Based on the extracted triple information, a prompt for the large model is constructed; the prompt is used to guide the large model to generate answers related to the knowledge graph; Convert the original training set into a prompt training set containing knowledge graph triple information, which will be used to fine-tune the large model; Select a large model based on the Transformer architecture as the base model; load the weights of the pre-trained large model as the starting point for fine-tuning; Define the token sequence format for input into the large model; define the objective function for named entity recognition, set hyperparameters, and use gradient accumulation to cope with video memory limitations.

5. The method according to claim 4, characterized in that The pre-trained large model after fine-tuning is used to extract entities and relationships in the user's question, and the knowledge graph is input to generate second knowledge graph triple information, and the second knowledge graph triple information is input to the large model to obtain an initial answer to form third knowledge graph triple information, specifically: The fine-tuned pre-trained large model is used to extract entities and relationships from the user's questions; based on the extracted entities and relationships, relevant information is searched in the knowledge graph and knowledge graph triples are generated; the knowledge graph triples are organized into prompts and input into the large model; the large model generates a preliminary answer based on the input prompt; entities and relationships are extracted again from the preliminary answer.

6. The method according to claim 5, characterized in that The third step of comparing the triple information of the second knowledge graph with the triple information of the third knowledge graph, and outputting the final answer if they are similar, and outputting the final answer after manual proofreading if they are not similar, specifically, extracting entities and relationships again from the preliminary answer generated by the large model, and forming new triples; comparing the triples of the preliminary answer with the triples of the user's question; If the difference between the two is large, the preliminary answer is considered to be wrong, and the knowledge graph retrieval enhanced generation technology is used for verification, and the model is optimized or manually corrected.

7. The method according to claim 4, characterized in that The hyperparameters include learning rate, batch size, and number of training rounds.

8. The method according to claim 4, characterized in that The objective function includes a cross entropy loss function.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it implements the steps in the method for improving large-model knowledge question-answering by applying a triple proofreading mechanism as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the steps in the method for improving large-model knowledge question-answering by applying a triple proofreading mechanism as described in any one of claims 1 to 8.

Images