Method and System for Detecting and Repairing Hallucinations in Outputs of Large Language Models

Abstract

The present invention relates to the technical field of knowledge Q&A, and relates to a method and system for detecting and repairing hallucinations in the output of large language models, including: dividing the answer content into several sentences, calculating the confidence of each sentence, determining whether the confidence of each sentence exceeds a set threshold, extracting facts from the sentences with confidence lower than the set threshold to obtain the triple data of each sentence; generating subgraph A of each sentence based on the triple data of each sentence; extracting subgraph B from the local knowledge graph based on the subject in the triple data of each sentence; calculating the similarity between subgraph A and subgraph B, if the similarity is less than the set threshold, then repairing subgraph A according to subgraph B to obtain the repaired subgraph A; obtaining the repaired sentence according to the triple data corresponding to the repaired subgraph A; after all the sentences with confidence lower than the set threshold are repaired, obtaining the final answer content. Improve the accuracy and reliability of the model output.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge Q&A, and particularly to a method and system for detecting and repairing hallucinations in the output of large language models. Background Art

[0002] Large Language Models (LLMs), as an important technology in the field of artificial intelligence, have demonstrated powerful capabilities in natural language processing tasks in recent years. LLMs represented by ChatGPT are widely used in multiple scenarios such as dialogue systems, knowledge Q&A, and automatic writing. However, in practical applications, LLMs have also exposed some problems that need to be solved urgently. Among them, the "Hallucination" problem is one of the most challenging difficulties. The so-called "Hallucination" refers to the content generated by the large language model that does not conform to objective facts, or the content is logically contradictory to the context. This phenomenon not only affects the reliability of the model output but may also cause serious consequences in fields such as medical care and education that require high accuracy.

[0003] The existing technology mainly detects hallucinations through the method of comparing with an external knowledge base. This technology compares the content generated by the model with the factual data in the external knowledge base to determine whether there are hallucinations. However, such methods rely on the coverage and update frequency of the knowledge base and are difficult to process each content generated by the large model in real time. If manual review is used, it is time-consuming and laborious, while using model review has limited support for fine-grained detection of documents.

[0004] In addition, even if the existing technology can detect the hallucination problem to a certain extent, it lacks an effective repair mechanism. When some content output by the model is judged to be a hallucination, it is often only marked as an error, and no repair or improvement solution can be provided. This results in the output quality of the large language model not being able to be further optimized, limiting its practical application in high-demand scenarios. Summary of the Invention

[0005] To solve the deficiencies of the existing technology, the present invention provides a method and system for detecting and repairing hallucinations in the output of large language models; through fine-grained confidence evaluation and knowledge graph verification technology, the problems in model generation are located and solved, thereby improving the accuracy and reliability of the model output.

[0006] On the one hand, a method for detecting and repairing hallucinations in the output of large language models is provided, including:

[0007] Obtain a question, input the question into the large language model, and obtain the answer content generated by the large language model;

[0008] Divide the answer content into several sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content. If it does not exceed, proceed to the next step;

[0009] Extract facts from sentences with confidence lower than the set threshold to obtain the triple data of each sentence. The triple data includes: subject, predicate, and object. Generate subgraph A for each sentence based on the triple data of each sentence;

[0010] Extract subgraph B from the local knowledge graph based on the subject in the triple data of each sentence. Calculate the similarity between subgraph A and subgraph B. If the similarity is greater than the set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output. If the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain the repaired subgraph A. Obtain the repaired sentence according to the triple data corresponding to the repaired subgraph A. After all sentences with confidence lower than the set threshold are repaired, obtain the final answer content.

[0011] On the other hand, a detection and repair system for hallucinations in the output of a large language model is provided, including:

[0012] A question-and-answer module configured to: obtain a question, input the question into the large language model, and obtain the answer content generated by the large language model;

[0013] A confidence calculation module configured to: divide the answer content into several sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content. If it does not exceed, proceed to the next step;

[0014] A fact extraction module configured to: extract facts from sentences with confidence lower than the set threshold to obtain the triple data of each sentence. The triple data includes: subject, predicate, and object. Generate subgraph A for each sentence based on the triple data of each sentence;

[0015] A detection and repair module configured to: extract subgraph B from the local knowledge graph based on the subject in the triple data of each sentence. Calculate the similarity between subgraph A and subgraph B. If the similarity is greater than the set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output. If the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain the repaired subgraph A. Obtain the repaired sentence according to the triple data corresponding to the repaired subgraph A. After all sentences with confidence lower than the set threshold are repaired, obtain the final answer content.

[0016] The above technical solution has the following advantages or beneficial effects:

[0017] The present invention can judge the reliability of the model output content based on the internal state of the model, filling the gap that the existing hallucination detection technology cannot be detected internally. When the model outputs, of the attribute value is used to directly judge whether the model has hallucinations. This method is different from other traditional methods that detect whether the model has hallucinations externally, reducing system overhead.

[0018] The confidence evaluation module calculates the confidence of each sentence generated by the model, identifies potential hallucination content, and makes annotations accordingly. This process is completed inside the large language model, so it can judge the credibility of the generated content in real time. By calculating the confidence score for each sentence, the system can quickly discover possible error information and correct it during the generation process. Compared with the method of "contrast detection" relying on an external knowledge base, the real-time performance and efficiency of internal evaluation significantly improve the adaptive ability of the model.

[0019] The fact extraction module simplifies the generated text into multiple basic viewpoints and converts them into triples, further ensuring the accuracy of the generated content. This module can extract viewpoints by combining specific guidelines, or by training a small-parameter model to extract viewpoints, improving the calculation speed and flexibility. By converting the text into structured triples, the system can facilitate subsequent verification and repair.

[0020] The knowledge graph verification module uses the data of the local knowledge graph and the external knowledge base for verification and repair. In this module, the local knowledge base is used to detect and repair the hallucinated parts. During this process, the system checks whether the generated triple content conforms to the existing information in the knowledge graph through graph similarity calculation and semantic analysis, and repairs it according to the data in the graph. This step enables the system to make full use of the reliability of the external knowledge base while maintaining an efficient internal repair ability to ensure the authenticity of the generated content.

[0021] Overall, through internal confidence evaluation, fact extraction, and local knowledge graph verification, the system can effectively detect and repair potential hallucination content during the generation process, avoiding the complexity brought by the external dependence of traditional methods. Compared with traditional external methods, the system of the present invention not only improves the accuracy and real-time performance of the generated content, but also greatly reduces the dependence on external resources, providing a more efficient and reliable hallucination detection and repair solution. This design is particularly suitable for fields with extremely high accuracy requirements, such as education, medical care, and law, showing great practical application potential and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The attached drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0023] Figure 1 It is a flowchart of the method for the first embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0025] The First Embodiment

[0026] This embodiment provides a method for detecting and repairing hallucinations in the output of a large language model;

[0027] As Figure 1 shown, the method for detecting and repairing hallucinations in the output of a large language model includes:

[0028] S101: Obtain a question, input the question into the large language model, and obtain the answer content generated by the large language model;

[0029] S102: Divide the answer content into several sentences, calculate the confidence level of each sentence, and determine whether the confidence level of each sentence exceeds a set threshold. If it exceeds, directly output the answer content; if it does not exceed, proceed to the next step;

[0030] S103: Extract facts from the sentences with confidence levels lower than the set threshold to obtain the triple data of each sentence. The triple data includes: subject, predicate, and object; generate sub-graph A for each sentence based on the triple data of each sentence;

[0031] S104: Extract sub-graph B from the local knowledge graph based on the subject in the triple data of each sentence; calculate the similarity between sub-graph A and sub-graph B. If the similarity is greater than the set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair sub-graph A according to sub-graph B to obtain the repaired sub-graph A; obtain the repaired sentence according to the triple data corresponding to the repaired sub-graph A; after all the sentences with confidence levels lower than the set threshold are repaired, obtain the final answer content.

[0032] Furthermore, the S101: obtains a question, inputs the question into a large language model, and obtains an answer content generated by the large language model; wherein the large language model refers to a large language model (LLM, Large Language Model).

[0033] Further, the S102: divide the answer content into a number of sentences, specifically divide the answer content into a number of sentences according to the punctuation marks indicating the end of the sentence, wherein the punctuation marks indicating the end of the sentence include: a period, an exclamation mark and a question mark; divide each sentence into a number of minimum units , for the smallest unit , delete the and the end of the question and answer , the remaining results are marked as valid , where the smallest unit , including: words, characters or symbols.

[0034] For example, in the model generation When the as well as Alternative of Attribute values, in the large model, Refers to the smallest unit in the input text, usually a word, character or symbol. At this time, the responses generated by the model are divided into sentences, and the punctuation marks that match the end of the sentence are matched in the paragraph, and the punctuation marks that indicate the beginning and end of the response are removed. The result is recorded as effective .

[0035] Furthermore, the step S102: calculating the confidence of each sentence includes:

[0036] Confidence score of the sentence , with each Confidence The specific calculation formula is:

[0037]

[0038] in, is the weighted average score of the current sentence, is the weighted variance score of the current sentence, and the specific formulas are:

[0039]

[0040]

[0041] in, for Confidence score; is the number of valid in the current sentence; represents the th valid part-of-speech weight, weight example: {{"Proper noun": 1.0}, {"Common noun": 0.8}, {"Verb": 0.7}, {"Adjective / Adverb": 0.3}, {"Function word": 0.1}};

[0042] Confidence score is determined by its importance ( ) and the weighted sum of three scoring components ( , , ), and the specific formula is:

[0043]

[0044] where, , , are three weight coefficients, which respectively control the importance of the probability score , distribution score , sequence score . represents importance, and , the recommended value is , , , because is the most intuitive confidence indicator and should account for the main weight; reflects the overall credibility of the distribution and has slightly lower importance; mainly targets the coherence of the generated text and has the lowest importance. Such settings can maintain an intuitive confidence judgment while taking into account the distribution characteristics and generation coherence, making the confidence evaluation more balanced and comprehensive.

[0045] Furthermore, , and its calculation formula is:

[0046] is determined by its part-of-speech and position weights:

[0047]

[0048] where, represents the part-of-speech weight, which is the in the weighted average score of the sentence;

[0049] represents the position weight, which is related to the position in the current sentence. The specific formula is:

[0050]

[0051] where, represents the position in the current sentence, represents all numbers in the current sentence, is the natural logarithm, with a value of 2.718281828459045; using and is based on an assumption: the words at the beginning and end of a sentence (such as topic words, summary words) are usually more important, and the closer to the middle, the lower their contribution to the overall semantics may be; such an operation is similar to position encoding, emphasizing the combination of the distribution of semantic features and position information. Adding a constant 1 to the formula is to ensure that the lowest weight value is not too small, thus avoiding the subsequent calculation failure caused by a weight of 0; the introduction of the coefficient 0.2 is to adjust the increment amplitude so that the position weight maintains an appropriate influence in the actual context.

[0052] Furthermore, represents the probability score, and its specific formula is:

[0053]

[0054] where, and represent the highest probability value and the second highest probability value in the candidate respectively, represents the normalized probability of the highest probability value, that is, the normalized probability. The normalized probability reflects the relative probability in all candidate ; , , are the decomposition weight coefficients of (probability score), controlling the probability, and the difference, the importance of the normalized probability respectively, and , the recommended value is , , , The probability is the core indicator directly reflecting the confidence level and should account for the main weight; and The difference value is used to measure the uncertainty among candidates, and appropriate weights enhance robustness; normalization has a relatively small impact, so the weight is the lowest. Such a setting can synthesize the probability characteristics of candidates and reduce the deviation caused by the imbalance of a single indicator.

[0055] Regarding the calculation formula is:

[0056]

[0057] wherein, represents there are candidates , represents the highest probability value among the candidates , represents the probability value of the th candidate .

[0058] Furthermore, represents the distribution score, and the specific calculation formula for the is:

[0059]

[0060] wherein, , , is the (distribution score) decomposition weight coefficient, which respectively controls the importance of (entropy), (variance), (Gini coefficient), and , the recommended value is , , .

[0061] Since entropy is the core indicator to measure the chaos degree of the probability distribution, it should account for the main weight; variance represents the probability volatility, and its importance is secondary; the Gini coefficient measures the uniformity of the probability distribution, and the impact is relatively small. The recommended weight values can effectively reflect the credibility of the distribution and ensure the comprehensiveness and balance of the distribution indicators. Specifically, the calculation formulas for the relevant contents are as follows:

[0062] The entropy is, and the calculation formula is:

[0063]

[0064] wherein, represents the number of candidates , represents the th candidate The probability value.

[0065] is the variance, and the calculation formula is:

[0066]

[0067] Among them, represents the number of candidate quantities, represents the th candidate 's probability value, represents the average probability of candidates , and the calculation formula can be expressed as .

[0068] is the Gini coefficient, and the calculation formula is:

[0069]

[0070] Among them, represents the number of candidate quantities, represents the th candidate 's probability value.

[0071] Furthermore, the represents the sequence score, which evaluates the coherence in the generated sequence, and the calculation formula is:

[0072]

[0073] Among them, , are weight coefficients, which respectively control the (coherence score) and (position weight) importance, and , the recommended value is , .

[0074] Because is a key indicator for evaluating the coherence of the generated sequence and should account for the main weight; is used to supplement position information and has a relatively low weight. Such a setting can further refine the influence of different positions in the generated sequence on the confidence while emphasizing coherence.

[0075] is the coherence score, indicating the probability continuity between the current and the previous , and the calculation formula is:

[0076]

[0077] Among them, represents the current maximum probability candidate probability value, represents the previous maximum probability candidate probability value.

[0078] is the position weight, which is consistent with that described in the formula calculated above. Same.

[0079] Calculate the confidence score of each , and then calculate the confidence of the sentence according to . And process the data into the following format:

[0080]

[0081]

[0082]

[0083]

[0084] Among them, is the sentence obtained by dividing the paragraph in order, represents the confidence score of each sentence, that is, the above , represents the confidence score of each , that is, the above .

[0085] Furthermore, determine whether the confidence of each sentence exceeds the set threshold, where the set threshold refers to:

[0086] (1-1) Construct a data set; divide the data set into a training set and a validation set according to the ratio of 8:2;

[0087] (1-2) Construct a network, train the network with the training set, and stop training when the loss function value no longer decreases to obtain the trained network;

[0088] (1-3) Validate the trained network with the validation set. On this basis, enumerate different thresholds within the prediction probability range [0,1], and calculate the F1 score of the validation set under different set thresholds. Take the threshold corresponding to the maximum F1 score as the final set threshold.

[0089] Exemplarily, (1-1) construct a dataset; divide the dataset into a training set and a validation set according to an 8:2 ratio, including: for the paragraphs generated by the model, split them according to the sentence-ending punctuation, and label the dataset in the form of ; then, according to manual judgment, determine whether each sentence has hallucinated, and label it in the form of , where 1 represents hallucination and 0 represents no hallucination; label the confidence score corresponding to each sentence in the form of .

[0090] For each confidence score labeling form: Token_Score: [[0.91, 0.75, 0.89...], [0.68, 0.72, 0.40...], [0.88, 0.95, 0.97...],...].

[0091] Exemplarily, (1-2) construct a network, train the network using the training set, and stop training when the value of the loss function no longer decreases to obtain the trained network, including:

[0092] Use a lightweight network to extract sentence features and capture the non-linear relationship of the confidence distribution. The loss function uses the cross-entropy function as:

[0093] ;

[0094] Among them, represents the number of samples in the training set, represents the hallucination label (0 for no hallucination, 1 for hallucination), represents the confidence score of the th sentence, that is, the in the dataset, is the logarithmic function, is (the loss value), indicating the difference between the prediction result of the model and the true label. Thus, through the gradient descent algorithm, reduce the value to enable the model to learn the relationship between the confidence distribution and the hallucination label and output the hallucination probability of each sentence.

[0095] Exemplarily, (1-3) select a threshold: enumerate different thresholds within the prediction probability range [0, 1] , here the fixed-step method is adopted, with a step size of 0.01, and gradually search within the range of [0, 1], and classify according to the following rules:

[0096] ;

[0097] Among them, the classification result being 0 represents no hallucination, and the classification result being 1 represents hallucination, which represents the confidence score of the sentence, is the enumerated threshold.

[0098] Calculate the scores under different thresholds on the validation set The score is a commonly used evaluation metric in binary classification problems and can more accurately reflect the classification result of the model. Its calculation formula is:

[0099] ;

[0100] Among them, is the precision, which represents the number of true positive samples among the samples predicted as positive by the model; is the recall rate, which represents the proportion of all true positive samples that the model can correctly predict as positive. Specifically, the calculation formulas for relevant content are as follows:

[0101] ;

[0102] ;

[0103] Among them, is the true positive (True Positive), that is, the number of samples correctly predicted as positive; is the false positive (False Positive), that is, the number of samples wrongly predicted as positive; is the false negative (False Negative), that is, the number of samples wrongly predicted as negative.

[0104] Separate the scores under different thresholds are obtained respectively, such that when the score is the largest is denoted as and is used as the final static threshold. Based on the static threshold, by specifying the confidence weight by the user, different confidence thresholds in different scenarios can be achieved.

[0105] It should be understood that the confidence evaluation evaluates the confidence of each output content generated by the large language model, and through each

[0106] ​​Calculate the confidence score based on the generation probability, and determine which sentences may have hallucinations based on the set threshold, mark them accordingly, and send them to the next step for processing.

[0107] Specifically, for confidence evaluation, first, the large language model generates an answer to the question posed by the user. The system obtains the answer generated by the large language model and each of the attribute values. Next, divide the model's answer into multiple sentences, calculate the confidence score for each sentence through the set confidence function, and then determine whether the sentence confidence passes according to the threshold. For the threshold, we first let the model perform multiple rounds of iteration to generate a certain number of answers, then calculate the confidence for these answers respectively, and then manually mark the sentences with hallucinations to obtain a simple dataset. Train through a simple lightweight network and cross-entropy loss function to finally obtain the optimal threshold. At this time, the threshold is recorded as the static threshold. The user can dynamically change the weight of the threshold according to different actual application scenarios to balance the creativity and authenticity of the model. At this time, the threshold is recorded as the dynamic threshold. Next, encapsulate the model's reply and the information on whether it passes the confidence threshold according to the set data format and send it to the corresponding processing in the next step. This step can quickly locate potential hallucination content, avoid comprehensive verification of all sentences, thereby improving the system detection efficiency, and at the same time support flexible switching between static and dynamic thresholds to meet different scenario requirements.

[0108] Furthermore, in S103: Extract facts from sentences with confidence lower than the set threshold to obtain triple data for each sentence. Among them, fact extraction can be implemented using a large language model.

[0109] Exemplarily, use a local model to extract facts, combined with a prompt which is the input text or instruction used to guide the model to generate output in natural language processing (NLP) tasks. It is a clear prompt given to the model to guide the model to perform specific tasks, such as generating text, answering questions, translating, or extracting information, etc. Generally, a prompt consists of three parts: "task background description", "specific requirements", and "context and keywords". A feasible prompt example in this case is as follows:

[0110] prompt : You are an assistant good at text analysis and summarization. Please summarize the following text into several short viewpoints: .

[0111] Requirement: Each point should be presented in an independent sentence form. There is no limit to the number of points, but do not repeat or omit any. Ensure that the points are concise and clear, as short as possible, preferably with only subject, predicate, and object, and cover the core information of the text. Unless specifically specified by the user, all points should be presented in Chinese.

[0112] The following is an example:

[0113] Example input: "Copernicus was an astronomer from Poland who proposed the heliocentric theory."

[0114] Example output: {" ": "Copernicus was an astronomer from Poland who proposed the heliocentric theory.",

[0115] " ": ["Copernicus was from Poland", "Copernicus was an astronomer", "Copernicus proposed the heliocentric theory"] "} Through this prompt the model can be called to perform fact extraction work.

[0116] Alternatively, fact extraction is performed on sentences with confidence lower than a set threshold to obtain triple data for each sentence, where fact extraction is implemented using a trained model.

[0117] Furthermore, fact extraction is performed on sentences with confidence lower than a set threshold to obtain triple data for each sentence, including:

[0118] (2-1): Input the sentence with confidence lower than the set threshold into the trained model, and the model outputs the key statements in the sentence. The key statements include: subject, predicate, and object;

[0119] (2-2): Perform triple data extraction on the key statements to obtain the first entity - relationship - second entity data; where the first entity is the subject in the key statement, the relationship is the predicate in the key statement, and the second entity is the object in the relationship statement.

[0120] Furthermore, the (2-1): Input the sentence with confidence lower than the set threshold into the trained model, and the model outputs the key statements in the sentence, including:

[0121] (2-1-1): Construct a data set;

[0122] (2-1-2): Use the data set to train the model. During the training process, the input value is the sentence with confidence lower than the set threshold, and the output value is the key statement. The key statement includes: subject, predicate, and object;

[0123] (2-1-3): After the model training is completed, call the trained model to output the key statements in the sentence.

[0124] Exemplarily, the (2-1-1): Construct a dataset, including:

[0125] Obtain through Wikipedia, Baidu Encyclopedia or other reliable encyclopedias, and manually annotate to generate training data. The format of the data is sentences, and the output is the corresponding short viewpoints. The following is an example of the dataset:

[0126] {" ": "Nicolaus Copernicus (February 19, 1473 - May 24, 1543) was a Polish astronomer, mathematician and economist, famous for proposing the 'heliocentric theory'. He published the book 'On the Revolutions of the Celestial Spheres' in 1543. The publication of this book represents a major turning point in the history of astronomy development and plays a very important role in the development of astronomy in later generations.",

[0127] " ": {" ": "Copernicus was Polish", " ": "Copernicus was an astronomer", " ": "Copernicus proposed the 'heliocentric theory'", " ": "Copernicus wrote 'On the Revolutions of the Celestial Spheres'", " ": "On the Revolutions of the Celestial Spheres was published in 1543", "}}.

[0128] Exemplarily, (2-1-2): Use the dataset to train the BERT model. During the training process, the input value is the sentence with a confidence level lower than the set threshold, and the output value is the key statement, and the key statement includes: subject, predicate and object, including:

[0129] The training model selects a -architecture model, and the training objective is to automatically predict the independent viewpoints in the current sentence according to the input sentence. We regard the task of the model as a sequence labeling problem, that is, label each word as a component of a viewpoint (topic, predicate, object).

[0130] The training loss function of the model is:

[0131]

[0132] Among them, is the number of samples, is the The total number of opinions in a sentence, is the conditional probability, indicating the probability of generating the th opinion. After that, training starts, and the score is calculated to verify the closeness between the generated opinion and the target opinion.

[0133] Based on this, the model performance is optimized through the following steps: Use the validation set to evaluate the model performance, and judge the accuracy of opinion extraction of the model according to the score; After multiple rounds of training-validation cycles, select the model with the best performance on the validation set; Use the test set to conduct a final evaluation of the optimal model to ensure the performance of the model on unseen data. After training is completed, save the model with the best effect, and then apply it to the actual data.

[0134] Exemplarily, for the (2-1-3): After the model training is completed, call the trained model to output the key sentences in the sentence, including: Call the trained model. After the model training is completed, call the trained model to extract short opinions in the given sentence. The following is an example:

[0135] Input: "Copernicus is an astronomer from Poland, born in 1473, and proposed the heliocentric theory.",

[0136] Output: " {" ": "Copernicus is an astronomer from Poland who proposed the heliocentric theory.", " ": ["Copernicus is Polish", "Copernicus is an astronomer", "Copernicus proposed the heliocentric theory" ] "}。

[0137] Furthermore, for the (2-2): Extract triple data from the key sentences to obtain entity-relationship-entity data, including: Use a large language model to extract the triple data.

[0138] Exemplarily, continue to use the strategy of prompt + local model, extract and then perform entity and relationship extraction.

[0139] (2-2-1): Design the prompt , and a feasible prompt example is as follows:

[0140] Prompt = You are an assistant good at information extraction. Please extract all entities and their relationships from the following facts and display them in the form of triples. The output format is:

[0141] [{"Subject": "Entity 1", "Predicate": "Relationship", "Object": "Entity 2"}, the following are several examples:

[0142] Example input 1: "Copernicus proposed the heliocentric theory."

[0143] Example output 1: [{"Subject": "Copernicus", "Predicate": "Proposed", "Object": "the heliocentric theory"}]

[0144] Example input 2: "Copernicus is Polish."

[0145] Example output 2: [{"Subject": "Copernicus", "Predicate": "Nationality", "Object": "Poland"}]

[0146] Example input 3: "Copernicus wrote 'On the Revolutions of the Celestial Spheres', which was published in 1543."

[0147] Example output 3: [{"Subject": "Copernicus", "Predicate": "Wrote", "Object": "'On the Revolutions of the Celestial Spheres'"}, {"Subject": "'On the Revolutions of the Celestial Spheres'", "Predicate": "Publication Time", "Object": "1543"}]

[0148] Now start processing the following facts ; Requirements: Please process each fact one by one, extracting clear entities and relationships. Ensure accurate entity classification and concise relationship description, such as "Proposed", "Born in", "Nationality", etc. The output content is limited to the above specified format. If some facts lack a subject or an object, it can be analyzed in combination with the content in , and if no result can be obtained, it is marked as "unknown".

[0149] (2-2-2) Call the prompt Combine with the local model for relationship extraction. For the obtained results, it can be combined with the prompt for secondary verification. A feasible prompt is as follows:

[0150] Prompt =You are an assistant good at information extraction. Please check whether there are any omissions or errors according to the given factual views and the corresponding extracted entity-relationship list. Factual views: ; Entity-relationship list: ; Requirements: If modification is needed, please keep the original format of the entity-relationship list and generate the revised result; if no modification is needed, please only output "No content needs to be modified".

[0151] This prompt Implement a multi-round questioning mechanism to ensure the accuracy of the content. After completion, organize the data and send the generated data to the next step.

[0152] Further, the S103: generating a sub-graph A for each sentence based on the triple data of each sentence includes:

[0153] (3-1): Map the subject and object in the triple to the subject node and object node of the sub-graph A respectively. The attributes of the subject node and object node both include the entity name;

[0154] (3-2): Use the predicate in the triple as the connection edge between the subject node and the object node.

[0155] Exemplarily, record the entity-relationship list as a triple , import it into the knowledge graph database to form an operable graph data structure, and adopt a graph database.

[0156] Map "subject" and "object" to the nodes of the graph. The attributes of the nodes include (entity name). Example: {" ": "Copernicus"}; Map "predicate" to the edge between the nodes. The attributes of the edge include (source node), (target node) and (relationship type).

[0157] Example: {" ": "Copernicus", " ": "heliocentrism", " ": "proposed"}

[0158] Considering robustness, for incomplete or unknown entities, temporarily mark them with placeholders, such as "unknown_node_X".

[0159] After the mapping is completed, perform the import in , and record the graph as sub-graph A.

[0160] Preprocessing, the processing content includes: detecting and cleaning duplicate data: checking whether there are duplicate nodes or edges. Deleting redundant content to ensure the uniqueness of the graph. Attribute verification and completion: ensuring that all required attributes of the nodes and edges are complete. After completion, pass the triple to the next step for subsequent operations.

[0161] The fact extraction simplifies sentences with low confidence into multiple basic viewpoints (such as entities and relationships), and forms a structured knowledge graph (such as triples) by extracting these viewpoints, which is convenient for subsequent verification and repair operations.

[0162] Specifically, fact extraction first processes the data for confidence evaluation, extracts the sentences that do not pass the confidence evaluation among them, and then simplifies the extracted sentences into multiple basic viewpoints, which are called " ". In this step, there are two implementation methods: one is to extract viewpoints from sentences through the method of "local large model + "; the other is to train a simple model. The base model selects a model that has good natural language understanding . Select data from reliable data sources such as Wikipedia by itself, make a dataset, and divide the training set and validation set according to a certain proportion to train the model, ensuring that the trained model can extract viewpoints from sentences as required. Compared with the first method, the second method can reduce the consumption of system resources because it uses a model with a small number of parameters, and is easy to migrate and maintain. Next, the good " " will be used for entity and relationship extraction. Here, the method of "local large model + " is adopted to abstract the " " into multiple triple forms, denoted as " ". The system encapsulates the " " in the set data format and sends it to the next module for processing. This module significantly reduces the complexity of verification and repair by decomposing complex sentences into structured viewpoints, and supports lightweight implementation with less resource consumption, which is suitable for high-concurrency scenarios.

[0163] Furthermore, the S104: Extract subgraph B from the local knowledge graph based on the subject in each sentence triple data, including:

[0164] Extract subgraph B from the local knowledge graph according to the subject in each sentence triple data. The depth of subgraph B is 2. Subgraph B includes a first entity node, a first connection edge, and a second entity node; the first entity node is the subject in the triple data.

[0165] Exemplarily, in the triple , first extract the subject in each sentence triple data, denoted as the query entity list. Example: Input Triple:

[0166] [{"subject": "Copernicus", "predicate": "wrote", "object": "On the Revolutions of the Celestial Spheres"}],

[0167] Output subject: ["Copernicus"].

[0168] Extract the subgraph related to the target entity and relationship from the local knowledge graph for comparison. According to the subject set, set the depth to 2, extract the relevant subgraph from the knowledge graph, and record it as subgraph B in the style of subgraph A. Subgraph B corresponds to one subject. An example is as follows:

[0169] Subject: ["Copernicus"]

[0170] Subgraph B: [{" ": "Copernicus", " ": "On the Revolutions of the Celestial Spheres", " ": "authored"},

[0171] {" ": "Copernicus", " ": "heliocentric theory", " ": "proposed"}, ]

[0172] Furthermore, calculating the similarity between subgraph A and subgraph B includes:

[0173] To calculate the similarity between subgraph A and subgraph B, first define the nodes in subgraph A and subgraph B as 、 , and the edges as 、 . The formula for the total similarity score is:

[0174]

[0175] Among them, represents the edge similarity law, and represents the node similarity law. Considering that the expressions in subgraph A and subgraph B may not be fixed, we calculate and using the Embedding model bge-large to perform Embedding on the names. Embedding is a method of converting data (such as words, sentences, images, etc.) into a fixed-length vector representation. This vector contains the semantic information of the data and can capture the similarity between different data in the vector space. Through Embedding, we can map the original high-dimensional data to a low-dimensional vector space, making similar data points closer in the space and dissimilar data points farther apart. After that, we determine the semantic similarity of the vectors obtained by Embedding through cosine similarity.

[0176] The calculation formula is:

[0177] The calculation formula of

[0178]

[0179] Among them, and are the results after embedding the attribute values of the nodes, and are the results after embedding the attribute values of the edges; represents the cosine similarity, and the calculation formula is:

[0180]

[0181] Among them, represents the modulus length of, represents the modulus length of, that is, the length of the vector; the cosine similarity measures the similarity between two vectors by calculating the angle between them. If the directions of the two vectors are the same or similar, the cosine similarity is close to 1. If the directions are opposite, it is close to -1. If they are irrelevant, it is close to 0. Next, is calculated. When , it means that the content in subgraph A is basically the same as that in subgraph B, that is, it means that the similarity between the generated content and the knowledge graph is relatively high, and the generated content is regarded as correct. The threshold is initially set to 0.9, which is based on the common threshold settings (0.8 - 0.9) in semantic matching tasks. At the same time, considering the high requirements of the task for the authenticity of the content, it is set to a relatively strict matching standard of 0.9. It can be further adjusted through actual tests later to balance the correctness and coverage of the content. If it is less than the threshold, it means that the content generated by the model may have hallucinations, and the content generated by the model needs to be modified.

[0182] Furthermore, if the similarity is less than the set threshold, then subgraph A is repaired according to subgraph B to obtain the repaired subgraph A, including:

[0183] Decompose the subject, predicate, and object of subgraph A from the triple data of subgraph A;

[0184] According to the subject of subgraph A, filter out the triple data in subgraph B whose second entity is the object of subgraph A;

[0185] Perform Embedding processing on the relationship of the filtered triple data and the predicate of subgraph A, and then calculate the cosine similarity between the relationship of the filtered triple data and the predicate of subgraph A;

[0186] If the cosine similarity is greater than 0.9, then based on the entity-relationship-entity data of the triple data obtained by screening, the predicate of subgraph A is modified and replaced to obtain the repaired subgraph A.

[0187] Alternatively, if the similarity is less than the set threshold, then subgraph A is repaired according to subgraph B to obtain the repaired subgraph A, and it further includes:

[0188] From the triple data of subgraph A, disassemble the subject, predicate, and object of subgraph A;

[0189] According to the subject of subgraph A, screen out the triple data in subgraph B whose relationship is the predicate of subgraph A;

[0190] Based on the entity-relationship-entity data of the screened triple data, modify and replace the object of subgraph A to obtain the repaired subgraph A.

[0191] Exemplarily, first, with the help of tools such as python, traverse the nodes and edges in subgraph B, convert the content related to subgraph A in subgraph B into "entity-relationship-entity" triples, and extract the original facts as the basis for repair. Next, locate the sentence whose confidence fails, denoted as Sentence_old, and continue to use the prompt+local model strategy for repair. A feasible prompt is as follows:

[0192] Prompt = You are an assistant good at information extraction. The following is the content generated by the model, and some of the information is incorrect and needs to be corrected according to the known knowledge. Please repair the generated content based on the following facts:

[0193] Generated content: {Sentence_old}

[0194] Correct knowledge: {The relevant entity-relationship-entity triples extracted from Graph B}

[0195] The following is an output example:

[0196] {"Original generated content": "Copernicus is an Italian astronomer who proposed the geocentric theory",

[0197] "New generated content": "Copernicus is a Polish astronomer who proposed the heliocentric theory"}

[0198] The repaired content should be grammatically concise, ensure the facts are correct, and when output, it should be as consistent as possible with the original content, only modifying the incorrect parts, and the format refers to the given output example.

[0199] Finally, replace the newly generated content part in the output with Sentence_old. After all sentences with unqualified confidence levels are replaced, the final output can be obtained.

[0200] In the above-mentioned fact extraction, the extracted basic viewpoints will be compared with the existing knowledge graph. By calculating the graph similarity, it is judged whether the generated content is consistent with the facts. For the content that fails the verification, the system will use the relevant information in the knowledge graph to repair the generated text, so as to ensure that the final output conforms to the actual facts.

[0201] Specifically, the key focus of knowledge graph verification is on "repair", which corrects the parts where the model has hallucinations. First, data extraction is performed. The triples therein are transformed into a knowledge graph with the help of a knowledge graph and a graph database such as Neo4j, and certain processing is carried out on the graph, such as deleting redundant nodes and duplicate edges. Next, for the triple "triple", the system first extracts the first piece of information therein, denoted as "subject". After collecting the "subjects" in the "triple", repair is carried out with the help of the local knowledge base. The local knowledge base is a relatively complete knowledge graph, which has common knowledge and is regularly maintained and updated to ensure its reliability. The system extracts a sub-graph related to the "subject" list and with a depth of 2 from the local knowledge graph. Next, a graph similarity calculation formula is set. The formula includes vertex similarity and edge similarity. At the same time, considering the problem of multiple expression ways of the same semantics, an Embedding model is also used in the formula to calculate the semantic similarity of vertex and edge attribute values, and the graph similarity is comprehensively calculated. After calculating the graph similarity, a judgment is made. In the field of natural language processing, the common threshold for semantic matching is [0.8, 0.9]. The system adopts a relatively strict threshold of 0.9 here. For the case where the graph similarity ≥ 0.9, it is determined that the model output is true; for the case where it is less than 0.9, it is determined that hallucinations have occurred. At this time, first, the nodes and edges in sub-graph B related to the content of sub-graph A are converted into triples, denoted as "triple_fact". Then, in the way of "local large model + prompt", the problematic sentence and "triple_fact" are sent into the model for repair, and the result is output after the repair is completed. Combining the knowledge graph verification and repair functions can not only effectively detect hallucination problems, but also ensure the factual accuracy of the output content and provide an extensible verification method to meet the needs of different fields.

[0202] Through a seamless connection in a serial workflow, an efficient closed-loop method is formed. In terms of data circulation, first, the confidence evaluation makes a preliminary confidence judgment on the output content generated by the large language model, marks the sentences with low confidence, and transfers them to the fact extraction module for further processing; the fact extraction simplifies the low-confidence sentences, extracts the basic viewpoints and converts them into structured triples, and transfers them to the knowledge graph verification through a standardized data format; the knowledge graph verification verifies according to the existing knowledge graph, compares the relevant knowledge in the graph with the generated content, and repairs the content that fails the verification, and finally returns the corrected content that conforms to the actual facts.

[0203] Overall, through precise division of labor and module collaboration, this method effectively improves the credibility of the content generated by the large language model, avoids the wrong information caused by hallucinations, and is particularly suitable for fields that require high accuracy, such as education, medical care, and law. By automatically detecting and repairing the hallucination phenomenon in the content generated by the large language model, this method not only improves the accuracy and reliability of the large language model in practical applications, but also enhances its interpretability and applicability in actual scenarios, showing great application potential and broad market prospects.

[0204] Embodiment 2

[0205] This embodiment provides a detection and repair system for hallucinations in the output of a large language model, including:

[0206] A question-and-answer module, which is configured to: obtain a question, input the question into the large language model, and obtain the answer content generated by the large language model;

[0207] A confidence calculation module, which is configured to: divide the answer content into several sentences, calculate the confidence of each sentence, judge whether the confidence of each sentence exceeds the set threshold, if it exceeds, directly output the answer content, and if it does not exceed, proceed to the next step;

[0208] A fact extraction module, which is configured to: extract facts from the sentences with confidence lower than the set threshold to obtain the triple data of each sentence, where the triple data includes: subject, predicate, and object; generate a sub-graph A for each sentence based on the triple data of each sentence;

[0209] The detection and repair module is configured to: extract sub-graph B from the local knowledge graph based on the subject in each sentence triple data; calculate the similarity between sub-graph A and sub-graph B. If the similarity is greater than the set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, sub-graph A is repaired according to sub-graph B to obtain the repaired sub-graph A; according to the triple data corresponding to the repaired sub-graph A, obtain the repaired sentence; after all sentences with a confidence level lower than the set threshold are repaired, obtain the final answer content.

[0210] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A detection and repair method for output hallucination of a large language model, characterized by: include: Get the question, input the question into the large language model, and get the answer content generated by the large language model; Divide the answer into several sentences and calculate the confidence of each sentence, including: Confidence score of the sentence , with each Confidence The specific calculation formula is: ； in, is the weighted average score of the current sentence, is the weighted variance score of the current sentence, and the specific formulas are: ； ； in, for Confidence score of Valid for the current sentence The number of Representative Valid The part-of-speech weight; The confidence score , the specific formula is: ； in, , , are three weight coefficients, which control the probability scores respectively , distribution score , sequence score The importance of express the importance of ; Determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content. If it does not exceed, proceed to the next step; Fact extraction is performed on sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: subject, predicate and object; subgraph A of each sentence is generated based on the triple data of each sentence; Based on the subject in the triple data of each sentence, subgraph B is extracted from the local knowledge graph; the similarity between subgraph A and subgraph B is calculated. If the similarity is greater than the set threshold, the answer content generated by the large language model is considered correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, subgraph A is repaired according to subgraph B to obtain a repaired subgraph A; based on the triple data corresponding to the repaired subgraph A, a repaired sentence is obtained; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

2. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: , and its calculation formula is: Determined by its part of speech and position weight: ； in, Represents the part-of-speech weight, which is the weighted average score of the sentence ; represents the position weight, and It is related to the position in the current sentence. The specific formula is: ； in, represent The position in the current sentence, Represents all the number, is the natural logarithm; Represents the probability score, and its specific formula is: ； in, and Representatives of candidates The highest probability value and the second highest probability value, The normalized probability represents the highest probability value; , , is the probability score The decomposition weight coefficients control , , The importance of probability, and ; about The calculation formula is: ； in, Representatives Candidates , Representative Candidate The highest probability value among Representative Candidates The probability value of .

3. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: represents the distribution score, The specific calculation formula is: ； in, , , The distribution score The decomposition weight coefficients control , , Gini coefficient the importance of ; is entropy, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of is the variance, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of Representative Candidate The probability mean of is expressed as ; is the Gini coefficient, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of .

4. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: Said Represents the sequence score, evaluation The coherence in the generated sequence is calculated as: ； in, , are weight coefficients, respectively controlling the coherence score and position weight the importance of ; is the coherence score, indicating the current With the previous The probability continuity of is calculated as: ； in, Represents the current The maximum probability candidate The probability value of Represents the previous The maximum probability candidate The probability value of .

5. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: Determine whether the confidence of each sentence exceeds the set threshold, where the set threshold refers to: Construct a data set; divide the data set into a training set and a validation set according to the set ratio; Construct a Transformer Encoder network and use the training set to train the network. When the loss function value no longer decreases, stop training and obtain the trained network. The trained network is verified using the validation set. On this basis, different thresholds in the prediction probability range [0, 1] are enumerated, and the F1 scores of the validation set under different set thresholds are calculated. The threshold corresponding to the maximum F1 score is taken as the final set threshold. Select threshold: Enumerate different thresholds within the prediction probability range [0, 1] , here we use the fixed step method with a step size of 0.01, and gradually search in the range of [0, 1], and classify according to the following rules: ； Among them, the classification results A value of 0 means no hallucinations occur, and a value of 1 means hallucinations occur. represents the confidence score of the sentence, is the threshold of the enumeration; Calculate different thresholds on the validation set Next Fraction, The score calculation formula is: ； in, For accuracy, Indicates the number of true positive examples among the samples predicted by the model as positive examples; is the recall rate, It represents the proportion of all true positive examples that the model can correctly predict as positive examples. The calculation formula is as follows: ； ； in, is the number of true positive examples, which are correctly predicted as positive examples; is the number of false positive examples, FalsePositive, which is the number of false positive examples predicted as positive examples; is the number of false negative examples, which is the number of false negative examples predicted as negative examples; Next Score, so When the score is maximum Recorded as ,Will as the final static threshold.

6. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: The fact extraction is performed on sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, including: Input sentences with confidence levels lower than a set threshold into the trained BERT model, and the model outputs key sentences in the sentences, including subject, predicate, and object; For the key sentence, triple data is extracted to obtain the first entity-relationship-second entity data; wherein the first entity is the subject in the key sentence; the relationship is the predicate in the key sentence; the second entity is the object in the relationship sentence; The sentences with confidence levels lower than the set threshold are input into the trained BERT model, and the model outputs the key sentences in the sentences, including: Construct a data set; use the data set to train the BERT model. During the training process, the input value is a sentence whose confidence is lower than a set threshold, and the output value is a key sentence, which includes a subject, a predicate, and an object. After the model training is completed, call the trained model to output the key sentences in the sentence. Generate a subgraph A of each sentence based on the triple data of each sentence, including: mapping the subject and object in the triple to the subject node and object node of the subgraph A respectively, and the attributes of the subject node and the object node both include entity names; and use the predicate in the triple as a connecting edge between the subject node and the object node; Based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph, including: according to the subject in each sentence triple data, extract subgraph B from the local knowledge graph, the depth of subgraph B is 2, and subgraph B includes a first entity node, a first connecting edge, and a second entity node; the first entity node is the subject in the triple data.

7. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: The calculation of the similarity between subgraph A and subgraph B includes: calculating the similarity between subgraph A and subgraph B, first defining the nodes in subgraph A and subgraph B as , , , , the total similarity score formula is: ； in, represents the edge similarity law, Representative point similarity law; The calculation formula is: ； The calculation formula is: ； in, , This is the result of embedding the attribute value of the node. , It is the result after embedding the attribute value of the edge; Represents cosine similarity, and the calculation formula is: ； in, represent The module length, represent The mold length.

8. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: If the similarity is less than the set threshold, the sub-image A is repaired according to the sub-image B to obtain the repaired sub-image A, including: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, triple data whose second entity is the object of subgraph A is filtered out from subgraph B; Embed the relationship between the filtered triple data and the predicate of subgraph A, and then calculate the cosine similarity between the relationship between the filtered triple data and the predicate of subgraph A; If the cosine similarity is greater than 0.9, the predicate of subgraph A is modified and replaced based on the entity-relationship-entity data of the screened triple data to obtain the repaired subgraph A; If the similarity is less than the set threshold, the sub-image A is repaired according to the sub-image B to obtain the repaired sub-image A, and further comprising: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, select triples from subgraph B whose relations are the predicate of subgraph A; Based on the entity-relationship-entity data of the screened triple data, the object of subgraph A is modified and replaced to obtain the repaired subgraph A.

9. A detection and repair system for output hallucinations in large language models, characterized by: include: A question-answering module is configured to: obtain a question, input the question into a large language model, and obtain an answer content generated by the large language model; The confidence calculation module is configured to: divide the answer content into a number of sentences, and calculate the confidence of each sentence, including: Confidence score of the sentence , with each Confidence The specific calculation formula is: ； in, is the weighted average score of the current sentence, is the weighted variance score of the current sentence, and the specific formulas are: ； ； in, for Confidence score of Valid for the current sentence The number of Representative Valid The part-of-speech weight; The confidence score , the specific formula is: ； in, , , are three weight coefficients, which control the probability scores respectively , distribution score , sequence score The importance of express the importance of ; Determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content. If it does not exceed, proceed to the next step; A fact extraction module is configured to: extract facts from sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: a subject, a predicate and an object; and generate a subgraph A of each sentence based on the triple data of each sentence; The detection and repair module is configured as follows: based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph; calculate the similarity between subgraph A and subgraph B, if the similarity is greater than a set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain a repaired subgraph A; obtain a repaired sentence according to the triple data corresponding to the repaired subgraph A; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

Images