Method and system for detecting and repairing output illusion of large language model

Abstract

The invention relates to the technical field of knowledge questions and answers, in particular to a method and system for detecting and repairing large language model output illusion, which comprises the following steps of: dividing answer content into a plurality of sentences, calculating the confidence coefficient of each sentence, judging whether the confidence coefficient of each sentence exceeds a set threshold value or not, performing fact extraction on the sentences with the confidence coefficient lower than the set threshold value, obtaining triple data of each sentence; generating a subgraph A of each sentence based on the triple data of each sentence; extracting a sub-graph B from the local knowledge graph based on the subject in the triple data of each sentence; calculating the similarity between the sub-graph A and the sub-graph B, and if the similarity is smaller than a set threshold value, repairing the sub-graph A according to the sub-graph B to obtain a repaired sub-graph A; according to the triad data corresponding to the repaired subgraph A, obtaining a repaired sentence; and after all the sentences with the confidence lower than a set threshold value are repaired, final answer content is obtained. And the accuracy and reliability of model output are improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge question answering technology, and in particular to a method and system for detecting and repairing output hallucinations of a large language model. Background Art

[0002] As an important technology in the field of artificial intelligence, large language models (LLM) have demonstrated powerful capabilities in natural language processing tasks in recent years. Large language models represented by ChatGPT are widely used in multiple scenarios such as dialogue systems, knowledge question-answering, and automatic writing. However, in practical applications, large language models have also exposed some problems that need to be solved urgently, among which the "Hallucination" problem is one of the most challenging problems. The so-called "hallucination" refers to the fact that the content generated by the large language model does not match the objective facts, or the content is inconsistent with the context logic. This phenomenon not only affects the reliability of the model output, but may also cause serious consequences in fields such as medical care and education that require high accuracy.

[0003] Existing technologies mainly use external knowledge base comparison methods to detect hallucinations. This technology compares the content generated by the model with the factual data in the external knowledge base to determine whether there is a hallucination. However, this type of method relies on the coverage and update frequency of the knowledge base, and it is difficult to process every content generated by a large model in real time. If manual review is used, it is time-consuming and laborious, while the model review method has limited support for fine-grained document detection.

[0004] In addition, even if existing technologies can detect hallucination problems to a certain extent, they lack effective repair mechanisms. When certain content in the model output is judged to be hallucination, it is often just marked as an error without providing a solution to repair or improve it. This results in the inability to further optimize the output quality of large language models, limiting their practical application in high-demand scenarios. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a method and system for detecting and repairing hallucinations in large language model outputs; through fine-grained confidence evaluation and knowledge graph verification technology, problems in model generation are located and solved, thereby improving the accuracy and reliability of model output.

[0006] On the one hand, a detection and repair method for large language model output hallucination is provided, including: Get the question, input the question into the large language model, and get the answer content generated by the large language model; Divide the answer content into several sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content, if not, proceed to the next step; Fact extraction is performed on sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: subject, predicate and object; subgraph A of each sentence is generated based on the triple data of each sentence; Based on the subject in the triple data of each sentence, subgraph B is extracted from the local knowledge graph; the similarity between subgraph A and subgraph B is calculated. If the similarity is greater than the set threshold, the answer content generated by the large language model is considered correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, subgraph A is repaired according to subgraph B to obtain a repaired subgraph A; based on the triple data corresponding to the repaired subgraph A, a repaired sentence is obtained; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

[0007] On the other hand, a detection and repair system for large language model output hallucinations is provided, including: A question-answering module is configured to: obtain a question, input the question into a large language model, and obtain an answer content generated by the large language model; The confidence calculation module is configured to: divide the answer content into a number of sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds a set threshold. If so, directly output the answer content; if not, proceed to the next step; A fact extraction module is configured to: extract facts from sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: a subject, a predicate and an object; and generate a subgraph A of each sentence based on the triple data of each sentence; The detection and repair module is configured as follows: based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph; calculate the similarity between subgraph A and subgraph B, if the similarity is greater than a set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain a repaired subgraph A; obtain a repaired sentence according to the triple data corresponding to the repaired subgraph A; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

[0008] The above technical solution has the following advantages or beneficial effects: The present invention can judge the reliability of the model output content based on the internal state of the model, filling the gap that the existing hallucination detection technology cannot detect internally. of Attribute values ​​are used to directly determine whether the model has produced hallucinations. This method is different from other traditional methods that detect whether the model has produced hallucinations externally, reducing system overhead.

[0009] The confidence assessment module calculates the confidence of each sentence generated by the model, identifies potential hallucination content, and labels it accordingly. This process is completed inside the large language model, so the credibility of the generated content can be judged in real time. By calculating the confidence score for each sentence, the system can quickly detect possible erroneous information and correct it during the generation process. Compared with the method of relying on an external knowledge base for "comparative testing", the real-time and efficient nature of internal evaluation significantly improves the model's adaptability.

[0010] The fact extraction module further ensures the accuracy of the generated content by simplifying the generated text into multiple basic opinions and converting them into triples. It can extract opinions by guiding, or by training a small parameter model, which improves the calculation speed and flexibility. By converting text into structured triples, the system can easily perform subsequent verification and repair.

[0011] The knowledge graph verification module uses data from the local knowledge graph and external knowledge base for verification and repair. In this module, the local knowledge base is used to detect and repair the hallucinated parts. In this process, the system verifies whether the generated triple content is consistent with the existing information in the knowledge graph through graph similarity calculation and semantic analysis, and repairs it based on the data in the graph. This step enables the system to fully utilize the reliability of the external knowledge base while maintaining efficient internal repair capabilities to ensure the authenticity of the generated content.

[0012] Overall, the system can effectively detect and repair potential hallucination content during the generation process through internal confidence assessment, fact extraction, and local knowledge graph verification, avoiding the complexity brought by external dependence of traditional methods. Compared with traditional external methods, the system of the present invention not only improves the accuracy and real-time performance of generated content, but also greatly reduces dependence on external resources, providing a more efficient and reliable hallucination detection and repair solution. This design is particularly suitable for fields with extremely high accuracy requirements, such as education, medical care, and law, and shows great practical application potential and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0014] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION

[0015] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0016] Embodiment 1 This embodiment provides a method for detecting and repairing output hallucinations of a large language model; like Figure 1 As shown, the detection and repair method of the large language model output hallucination includes: S101: Obtain a question, input the question into a large language model, and obtain an answer generated by the large language model; S102: Divide the answer content into several sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds a set threshold. If it exceeds, directly output the answer content; if it does not exceed, proceed to the next step; S103: extracting facts from sentences whose confidence is lower than a set threshold, and obtaining triple data of each sentence, wherein the triple data includes: a subject, a predicate, and an object; and generating a subgraph A of each sentence based on the triple data of each sentence; S104: Based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph; calculate the similarity between subgraph A and subgraph B, if the similarity is greater than the set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain a repaired subgraph A; obtain a repaired sentence according to the triple data corresponding to the repaired subgraph A; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

[0017] Furthermore, the S101: obtains a question, inputs the question into a large language model, and obtains an answer content generated by the large language model; wherein the large language model refers to a large language model (LLM, Large Language Model).

[0018] Further, the S102: divide the answer content into a number of sentences, specifically divide the answer content into a number of sentences according to the punctuation marks indicating the end of the sentence, wherein the punctuation marks indicating the end of the sentence include: a period, an exclamation mark and a question mark; divide each sentence into a number of minimum units , for the smallest unit , delete the and the end of the question and answer , the remaining results are marked as valid , where the smallest unit , including: words, characters or symbols.

[0019] For example, in the model generation When the as well as Alternative of Attribute values, in the large model, Refers to the smallest unit in the input text, usually a word, character or symbol. At this time, the responses generated by the model are divided into sentences, and the punctuation marks that match the end of the sentence are matched in the paragraph, and the punctuation marks that indicate the beginning and end of the response are removed. The result is recorded as effective .

[0020] Furthermore, the step S102: calculating the confidence of each sentence includes: Confidence score of the sentence , with each Confidence The specific calculation formula is:

[0021] in, is the weighted average score of the current sentence, is the weighted variance score of the current sentence, and the specific formulas are:

[0022]

[0023] in, for Confidence score of Valid for the current sentence The number of Representative Valid Part-of-speech weights, weight examples: {{“proper noun”: 1.0}, {“common noun”: 0.8}, {“verb”: 0.7}, {“adjective / adverb”: 0.3}, {“function word”: 0.1}}; The confidence score Due to its importance ( ) and three scoring components ( , , ), the specific formula is:

[0024] in, , , are three weight coefficients, which control the probability scores respectively , distribution score , sequence score The importance of express the importance of The recommended value is , , ,because, It is the most intuitive confidence indicator and should have the main weight; Reflects the overall credibility of the distribution, with slightly lower importance; It mainly focuses on the coherence of the generated text, which is of the lowest importance. This setting can take into account both the distribution characteristics and the generation coherence while maintaining intuitive confidence judgment, making the confidence evaluation more balanced and comprehensive.

[0025] Further, , and its calculation formula is: Determined by its part of speech and position weight:

[0026] in, Represents the part-of-speech weight, which is the weighted average score of the sentence ; represents the position weight, and It is related to the position in the current sentence. The specific formula is:

[0027] in, represent The position in the current sentence, Represents all the number, is the natural logarithm, the value is 2.718281828459045; use and It is based on an assumption: words at the beginning and end of a sentence (such as subject words and summary words) are usually more important, and the closer they are to the middle, the lower their contribution to the overall semantics; this operation is similar to positional encoding, emphasizing the combination of the distribution of semantic features and positional information. The constant 1 is added to the formula to ensure that the minimum weight value is not too small, so as to avoid the subsequent calculation failure caused by the weight being 0; the introduction of the coefficient 0.2 is to adjust the increment amplitude so that the position weight maintains a moderate influence in the actual context.

[0028] Further, Represents the probability score, and its specific formula is:

[0029] in, and Representatives of candidates The highest probability value and the second highest probability value, The normalized probability representing the highest probability value, that is, the normalized Probability, Normalized The probability reflects Among all candidates The relative probability of , , for The decomposition weight coefficients of (probability scores) control Probability, and Difference, Standardization The importance of probability, and The recommended value is , , , Probability is the core indicator that directly reflects confidence and should account for the main weight; and The difference is used to measure the uncertainty between candidates, and appropriate weights are used to enhance robustness; standardization The influence of is small, so the weight is the lowest. This setting can integrate the probability characteristics of the candidate items and reduce the deviation caused by the imbalance of a single indicator.

[0030] about The calculation formula is:

[0031] in, Representatives Candidates , Representative Candidate The highest probability value among Representative Candidates The probability value of .

[0032] Further, represents the distribution score, The specific calculation formula is:

[0033] in, , , for The decomposition weight coefficients of (distribution scores) control (entropy), (variance), (Gini coefficient), and The recommended value is , , .

[0034] Because entropy is the core indicator for measuring the degree of disorder in probability distribution, it should have the main weight; variance represents the volatility of probability, and its importance is second; the Gini coefficient measures the uniformity of probability distribution, and its influence is relatively small. It is recommended that the weight value can effectively reflect the credibility of the distribution and ensure the comprehensiveness and balance of the distribution indicators. Specifically, the calculation formula for the relevant content is as follows: is entropy, and the calculation formula is:

[0035] in, Representative candidate quantity, Representative Candidates The probability value of .

[0036] is the variance, and the calculation formula is:

[0037] in, Representative candidate quantity, Representative Candidates The probability value of Representative Candidate The probability mean of .

[0038] is the Gini coefficient, and the calculation formula is:

[0039] in, Representative candidate quantity, Representative Candidates The probability value of .

[0040] Furthermore, the Represents the sequence score, evaluation The coherence in the generated sequence is calculated as:

[0041] in, , are weight coefficients, respectively controlling (coherence score) and (position weight), and The recommended value is , .

[0042] because, It is the key indicator for evaluating the coherence of the generated sequence and should have the main weight; It is used to supplement the position information, and its weight is relatively low. This setting can further refine the influence of different positions on the confidence in the generated sequence on the basis of emphasizing coherence.

[0043] is the coherence score, indicating the current With the previous The probability continuity of is calculated as:

[0044] in, Represents the current The maximum probability candidate The probability value of Represents the previous The maximum probability candidate The probability value of .

[0045] is the position weight, which is the same as the calculation above As stated in the formula Consistent.

[0046] Calculate each The confidence score of , then according to Calculate the confidence of the sentence. And process the data into the following format:

[0047]

[0048]

[0049]

[0050] in, To divide the paragraph into sentences in order, Represents the confidence score of each sentence, which is the above , Represent each The confidence score of .

[0051] Furthermore, the determination of whether the confidence of each sentence exceeds a set threshold value, wherein the set threshold value refers to: (1-1) Construct a data set; divide the data set into a training set and a validation set in a ratio of 8:2; (1-2) Construction Network, use the training set to train the network, when the loss function value no longer decreases, stop training and get the trained network; (1-3) The trained network is verified using the validation set. On this basis, different thresholds within the prediction probability range [0,1] are enumerated, and the F1 scores of the validation set under different set thresholds are calculated. The threshold corresponding to the maximum F1 score is used as the final set threshold.

[0052] For example, (1-1) construct a data set; divide the data set into a training set and a validation set in a ratio of 8:2, including: for the paragraphs generated by the model, split them according to the sentence ending punctuation, and annotate the data set in the form of ; Then, we manually judge whether each sentence produces hallucinations, and the annotation format is , where 1 represents hallucination and 0 represents no hallucination; the confidence score corresponding to each sentence is marked as .

[0053] For each The confidence score annotation format is: Token_Score: [[0.91, 0.75, 0.89...], [0.68, 0.72, 0.40 ...], [0.88, 0.95, 0.97 ...], ...].

[0054] For example, (1-2) constructs The network is trained using the training set. When the loss function value no longer decreases, the training is stopped to obtain the trained network, including: use Lightweight network, extracts sentence features, and captures the nonlinear relationship of confidence distribution. The loss function uses the cross entropy function: ; in, represents the number of samples in the training set, represents the hallucination label (0 for no hallucination, 1 for hallucination), Representative The confidence score of the sentence is , is a logarithmic function, for (loss value), which indicates the difference between the model's prediction result and the true label. Thus, the gradient descent algorithm is used to reduce The value of enables the model to learn the relationship between the confidence distribution and the hallucination label and output the hallucination probability of each sentence.

[0055] For example, (1-3) selects the threshold: enumerate different thresholds in the prediction probability range [0, 1] , here we use the fixed step method with a step size of 0.01, and gradually search in the range of [0, 1], and classify according to the following rules: ; Among them, the classification results A value of 0 means no hallucinations occur, and a value of 1 means hallucinations occur. represents the confidence score of the sentence, is the threshold for enumeration.

[0056] Calculate different thresholds on the validation set Next Fraction, The score is a commonly used evaluation indicator in binary classification problems. It can more accurately reflect the results of model classification. Its calculation formula is: ; in, For accuracy, Indicates the number of true positive examples among the samples predicted by the model as positive examples; is the recall rate, It represents the proportion of all true positive examples that the model can correctly predict as positive examples. Specifically, the calculation formula for the relevant content is as follows: ; ; in, is the true positive, that is, the number of correctly predicted positive examples; is the number of false positives, i.e., the number of incorrectly predicted positive examples; is the number of false negative examples, that is, the number of incorrectly predicted negative examples.

[0057] At different thresholds Next Score, so When the score is maximum Recorded as ,Will as the final static threshold.

[0058] Based on the static threshold, different confidence thresholds in different scenarios can be achieved by users specifying confidence weights.

[0059] It should be understood that the confidence assessment is to perform a confidence assessment on each output content generated by the large language model. The confidence score is calculated based on the generation probability, and the set threshold is used to determine which sentences may have hallucinations, mark them accordingly, and send them to the next step for processing.

[0060] Specifically, for confidence evaluation, the large language model first generates an answer to the question raised by the user, and the system obtains the answer generated by the large language model and each of The model’s answer is divided into multiple sentences. The confidence score of each sentence is calculated by the set confidence function, and then the confidence score of the sentence is judged according to the threshold. For the threshold, we first let the model perform multiple rounds of iterations to generate a certain number of answers, and then calculate the confidence of these answers respectively. Then, we manually mark the sentences with hallucinations to obtain a simple data set. The lightweight network and cross entropy loss function are used for training, and the optimal threshold is finally obtained. At this time, the threshold is recorded as a static threshold. Users can dynamically change the weight of the threshold according to different actual application scenarios to balance the creativity and authenticity of the model. At this time, the threshold is recorded as a dynamic threshold. Next, the model's response and the information on whether it passes the confidence threshold are packaged according to the set data format and sent to the next step for corresponding processing. This step can quickly locate potential hallucination content and avoid comprehensive verification of all sentences, thereby improving the system detection efficiency. At the same time, it supports flexible switching between static and dynamic thresholds to meet the needs of different scenarios.

[0061] Furthermore, the S103: extracting facts from sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the fact extraction can be implemented by using a large language model.

[0062] For example, a local model is used to extract facts, combined with prompt words Go ahead, prompt word It is the input text or instruction used to guide the model to generate output in natural language processing (NLP) tasks. It is a clear prompt given to the model to guide the model to perform specific tasks, such as generating text, answering questions, translating or extracting information. The general prompt word is It consists of three parts: "Task Background Description", "Specific Requirements", and "Context and Keywords". A feasible prompt word in this example Here is an example: Prompt word : You are an assistant who is good at text analysis and summarization. Please summarize the following text into a few brief points: .

[0063] Requirements: Each point should be presented in the form of an independent sentence. There is no limit to the number of points, but do not repeat or omit them. Make sure the points are concise and clear, as short as possible, preferably with only the subject, predicate and object, and cover the core information of the text. Unless otherwise specified by the user, all points should be presented in Chinese.

[0064] Here is an example: Sample input: "Copernicus was a Polish astronomer who proposed the heliocentric theory." Sample output: {" ": "Copernicus was a Polish astronomer who proposed the heliocentric theory. ", " ": ["Copernicus is a Pole", "Copernicus is an astronomer", "Copernicus proposed the heliocentric theory" ] "}, through this prompt word The model is called to perform fact extraction.

[0065] Alternatively, the fact extraction is performed on the sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the fact extraction is implemented using a trained model.

[0066] Furthermore, the sentence whose confidence is lower than the set threshold is subjected to fact extraction to obtain triple data of each sentence, including: (2-1): Input sentences with confidence levels lower than the set threshold into the trained In the model, the model outputs the key sentences in the sentence, and the key sentences include: subject, predicate and object; (2-2): For the key sentence, triple data is extracted to obtain the first entity-relationship-second entity data; wherein the first entity is the subject in the key sentence; the relationship is the predicate in the key sentence; and the second entity is the object in the relationship sentence.

[0067] Furthermore, (2-1): sentences with confidence levels lower than a set threshold are input into the trained In the model, the model outputs the key sentences in the sentence, including: (2-1-1): Build a data set; (2-1-2): Use the dataset to The model is trained. During the training process, the input value is a sentence whose confidence is lower than a set threshold, and the output value is a key sentence. The key sentence includes: a subject, a predicate, and an object; (2-1-3): After the model training is completed, call the trained model to output the key sentences in the sentence.

[0068] Exemplarily, the (2-1-1): constructing a data set includes: The training data is obtained through Wikipedia, Baidu Encyclopedia or other credible encyclopedias, and manually annotated. The format of the data is sentences, and the output is a brief viewpoint corresponding to it. The following is an example of a data set: {" ": "Nicolaus Copernicus (February 19, 1473 - May 24, 1543) was a Polish astronomer, mathematician and economist, famous for proposing the "heliocentric theory". In 1543, he published the book "On the Revolutions of the Celestial Spheres", which represented a major turning point in the history of astronomy and played a very important role in the development of astronomy in later generations. ", " ": {" ": "Copernicus was Polish"," ": "Copernicus was an astronomer"," ": "Copernicus proposed the heliocentric theory"," ": "Copernicus wrote On the Revolutions of the Celestial Spheres"," ": "On the Revolutions of the Celestial Spheres" was published in 1543",}}.

[0069] Exemplarily, (2-1-2): The BERT model is trained using a data set. During the training process, the input value is a sentence whose confidence is lower than a set threshold, and the output value is a key sentence. The key sentence includes: a subject, a predicate, and an object, including: The training model is selected based on Architectural The training goal of the model is to automatically predict the independent opinions in the current sentence based on the input sentence. We regard the task of the model as a sequence labeling problem, that is, labeling each word as a component of an opinion (subject, predicate, object).

[0070] train The loss function of the model is:

[0071] in, is the number of samples, It is The total number of opinions in the sentences, is the conditional probability, indicating the generation of Then we start training and calculate Score, verifying how close the generated view is to the target view.

[0072] On this basis, the model performance is optimized through the following steps: Use the validation set to evaluate the performance of the model, according to The score determines the accuracy of the model's opinion extraction; after multiple rounds of training-validation cycles, the model with the best performance on the validation set is selected; the test set is used to conduct a final evaluation of the best model to ensure the model's performance on unseen data. After training is completed, the best model is saved and then applied to actual data.

[0073] Exemplarily, (2-1-3): after the model training is completed, calling the trained model to output the key sentences in the sentence includes: calling the trained model. After the model training is completed, calling the trained model to extract a brief idea from a given sentence. The following is an example: Input: "Copernicus was a Polish astronomer born in 1473 who proposed the heliocentric theory.", Output: "{" ": "Copernicus was a Polish astronomer who proposed the heliocentric theory. ", " ": ["Copernicus was a Pole", "Copernicus was an astronomer", "Copernicus proposed the heliocentric theory" ]"}.

[0074] Furthermore, the (2-2): extracting triple data from key sentences to obtain entity-relationship-entity data includes: extracting triple data using a large language model.

[0075] For example, the prompt word is continued + Local model strategy, will Extract it and then perform entity and relationship extraction.

[0076] (2-2-1): Design prompt words , a feasible prompt word Here is an example: Prompt word =You are an assistant who is good at information extraction. Please extract all entities and their relationships from the following facts and present them in the form of triples. The output format is: [{"Subject": "Entity 1", "Predicate": "Relationship", "Object": "Entity 2"}], here are a few examples: Sample input 1: "Copernicus proposed the heliocentric theory." Sample output 1: [{"Subject": "Copernicus", "Predicate": "Proposed", "Object": "Heliocentrism"}] Sample input 2: "Copernicus was Polish." Sample output 2: [{"Subject": "Copernicus", "Predicate": "Nationality", "Object": "Poland"}] Sample input 3: "Copernicus wrote On the Revolutions of the Celestial Spheres, published in 1543." Sample output 3: [{"Subject": "Copernicus", "Predicate": "Author", "Object": "On the Revolutions of the Celestial Spheres"}, {"Subject": "On the Revolutions of the Celestial Spheres", "Predicate": "Published", "Object": "1543"}] Now let's deal with the following facts ; Requirements: Please process each fact one by one and extract clear entities and relationships. Ensure that the entity classification is accurate and the relationship description is concise, such as "proposed", "born in", "nationality", etc. The output content is limited to the above specified format. If some facts lack a subject or object, you can combine If no result is obtained in the content analysis, it is marked as "unknown".

[0077] (2-2-2) Call prompt word Combine the local model to extract relations. For the results obtained, you can combine the prompt words Perform secondary verification. A feasible prompt word Here is an example: Prompt word =You are an assistant who is good at information extraction. Please check whether there are omissions or errors based on the given factual viewpoints and the corresponding entity-relationship list that has been extracted. Factual viewpoints: ;Entity-relationship list: ; Requirements: If modification is required, please keep the original format of the entity-relationship list and generate the revised results; If no modification is required, please only output "no need to modify the content".

[0078] The prompt word Implement a multi-round questioning mechanism to ensure the accuracy of the content. After the completion, organize the data and send the generated data to the next step.

[0079] Furthermore, the S103: generating a subgraph A of each sentence based on the triple data of each sentence includes: (3-1): Map the subject and object in the triple to the subject node and object node of subgraph A respectively. The attributes of the subject node and object node both include entity names. (3-2): Use the predicate in the triple as the connecting edge between the subject node and the object node.

[0080] Exemplarily, the entity-relationship list is recorded as a triple , imported into the knowledge graph database to form an operational graph data structure, using Graph database.

[0081] Map the "subject" and "object" to nodes of the graph. The attributes of the nodes include (entity name). Example: {" ": "Copernicus"}; Map the "predicate" to the edge between nodes, and the attributes of the edge include (source node), (target node) and (Relationship type).

[0082] Example: {" ": "Copernicus", " ": "Heliocentrism", " ": "propose"} For robustness reasons, incomplete or unknown entities are temporarily marked with placeholders, such as "unknown_node_X".

[0083] After the mapping is completed The import is performed in , and the graph is denoted as subgraph A.

[0084] Preprocessing, including: Detect and clean up duplicate data: Check whether there are duplicate nodes or edges. Delete redundant content to ensure the uniqueness of the graph. Attribute verification and completion: Ensure that all required attributes of nodes and edges are complete. After completion, the triples are Pass in the next step for subsequent operations.

[0085] The fact extraction simplifies sentences with low confidence into multiple basic ideas (e.g., entities and relationships), and forms a structured knowledge graph (e.g., triples) by extracting these ideas to facilitate subsequent verification and repair operations.

[0086] Specifically, fact extraction first processes the confidence evaluation data, extracts the sentences that are marked as not passing the confidence evaluation, and then simplifies the proposed sentences into multiple basic viewpoints, which are called " ", at this step, there are two ways to achieve: one is to use "local large model + " method to extract opinions from sentences; the other is to train a simple model, the base model uses a model that has a good understanding of natural language Model, select data from reliable data sources such as Wikipedia, make a good data set, and divide the training set and validation set into proportions to train the model, ensuring that the trained model can extract opinions from sentences as required. Compared with the first method, method 2 uses a model with a small number of parameters, which can reduce the consumption of system resources and is easy to migrate and maintain. Next, we will propose a good " "To extract entities and relationships, we use "local large model + " " is abstracted into multiple triples, denoted as " ”, the system will "It is packaged according to the set data format and sent to the next module for processing. This module significantly reduces the complexity of verification and repair by decomposing complex sentences into structured viewpoints, and supports lightweight implementation, consumes less resources, and is suitable for high-concurrency scenarios.

[0087] Furthermore, the S104: extracting a subgraph B from the local knowledge graph based on the subject in each sentence triple data includes: According to the subject in each sentence triple data, subgraph B is extracted from the local knowledge graph. The depth of subgraph B is 2. Subgraph B includes a first entity node, a first connecting edge, and a second entity node. The first entity node is the subject in the triple data.

[0088] For example, in the triple In the example, we first extract the subject in each sentence triple data and record it as the query entity list. Triplet: [{"Subject": "Copernicus", "Predicate": "Wrote", "Object": "On the Revolutions of the Celestial Spheres"}], Output subject: ["Copernicus"].

[0089] Extract subgraphs related to the target entity and relationship from the local knowledge graph for comparison. According to the subject set, set the depth to 2, extract the relevant subgraphs in the knowledge graph, and follow the style of subgraph A to record it as subgraph B. Subgraph B corresponds to a subject. An example is shown below: Subject: ["Copernicus"] Sub-graph B: [{" ": "Copernicus", " ": "On the Movement of Celestial Bodies", " ": "authored by"}, {" ": "Copernicus", " ": "Heliocentrism", " ": "propose"}, ] Further, the calculating the similarity between subgraph A and subgraph B includes: To calculate the similarity between subgraph A and subgraph B, we first define the nodes in subgraph A and subgraph B as , , , , the total similarity score formula is:

[0090] in, represents the edge similarity law, Considering that the expressions in subgraph A and subgraph B may not be fixed, we calculate and When embedding the name, we use the embedding model bge-large. Embedding is a method of converting data (such as words, sentences, images, etc.) into a fixed-length vector representation. This vector contains the semantic information of the data and can capture the similarity between different data in the vector space. Through Embedding, we can map high-dimensional raw data to low-dimensional vector space, so that similar data points are closer in space, while dissimilar data points are farther away. Then we determine the semantic similarity of the vector obtained by Embedding through cosine similarity.

[0091] The calculation formula is: The calculation formula is:

[0092] in, , This is the result of embedding the attribute value of the node. , It is the result after embedding the attribute value of the edge; Represents cosine similarity, and the calculation formula is:

[0093] in, represent The module length, represent The modulus is the length of the vector; cosine similarity measures the similarity between two vectors by calculating the angle between them. If the two vectors are in the same or similar direction, the cosine similarity is close to 1, if they are in opposite directions, it is close to -1, and if they are irrelevant, it is close to 0. Next, we calculate ,when , it means that the content in subgraph A is basically consistent with subgraph B, which means that the generated content has a high degree of similarity with the knowledge graph and is considered correct. The threshold is initially set to 0.9, which is based on the common threshold setting (0.8-0.9) in semantic matching tasks. Considering the high requirements of the task on the authenticity of the content, it is set to a stricter matching standard of 0.9. Further adjustments can be made through actual tests to balance the correctness and coverage of the content. If it is less than the threshold, it means that the content generated by the model may be hallucinating and the model-generated content needs to be modified.

[0094] Furthermore, if the similarity is less than a set threshold, the sub-image A is repaired according to the sub-image B to obtain the repaired sub-image A, including: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, triple data whose second entity is the object of subgraph A is filtered out from subgraph B; Embed the relationship between the filtered triple data and the predicate of subgraph A, and then calculate the cosine similarity between the relationship between the filtered triple data and the predicate of subgraph A; If the cosine similarity is greater than 0.9, the predicate of subgraph A is modified and replaced based on the entity-relationship-entity data of the screened triple data to obtain a repaired subgraph A.

[0095] Alternatively, if the similarity is less than a set threshold, repairing sub-image A according to sub-image B to obtain a repaired sub-image A, further comprising: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, select triples from subgraph B whose relations are the predicate of subgraph A; Based on the entity-relationship-entity data of the screened triple data, the object of subgraph A is modified and replaced to obtain the repaired subgraph A.

[0096] For example, we first traverse the nodes and edges in subgraph B with the help of tools such as Python, convert the content in subgraph B related to subgraph A into "entity-relationship-entity" triples, and extract the original facts as the basis for repair. Next, we locate the sentence whose confidence level does not pass, record it as Sentence_old, and continue to use the prompt+local model strategy for repair. A feasible prompt is as follows: Prompt=You are an assistant who is good at extracting information. The following is the content generated by the model. Some of the information has problems and needs to be corrected based on known knowledge. Please fix the generated content based on the following facts: Generated content: {Sentence_old} Correct knowledge: {Related entity-relationship-entity triples extracted from graph B} The following is an example of the output: {"Original generated content": "Copernicus was an Italian astronomer who proposed the geocentric theory", "Newly generated content": "Copernicus was a Polish astronomer who proposed the heliocentric theory"} The repaired content should keep the grammar concise and ensure the facts are correct. When outputting, please keep it as consistent as possible with the original content and only modify the incorrect parts. The format should refer to the output example given.

[0097] Finally, the newly generated content in the output will replace Sentence_old. After all sentences that fail to pass the confidence level are replaced, the final output can be obtained.

[0098] In the fact extraction, the extracted basic viewpoints will be compared with the existing knowledge graph. Through the graph similarity calculation, it is judged whether the generated content is consistent with the facts. For the content that fails the verification, the system will use the relevant information of the knowledge graph to repair the generated text, so as to ensure that the final output is consistent with the actual facts.

[0099] Specifically, the knowledge graph verification focuses on "repair" to correct the part of the model that produces hallucinations. First, data extraction is performed, and the triples in it are converted into knowledge graphs using graph databases such as Neo4j with the help of knowledge graphs, and the graphs are processed to a certain extent, deleting redundant nodes and repeated edges. Next, for the triple "triple", the system first extracts the first information in it, recorded as the "subject". After collecting the "subject" in the "triple", it is repaired with the help of the local knowledge base. The local knowledge base is a relatively complete knowledge graph with common knowledge and regular maintenance and updates to ensure its reliability. The system extracts a subgraph related to the "subject" list with a depth of 2 in the local knowledge graph, and then sets the graph similarity calculation formula. The formula includes point similarity and edge similarity. At the same time, considering the problem of multiple expressions of the same semantics, the formula also uses the Embedding model to calculate the semantic similarity of point and edge attribute values, and comprehensively calculates the graph similarity. After calculating the graph similarity, a judgment is made. In the field of natural language processing, the common threshold for semantic matching is [0.8, 0.9]. The system uses a more stringent threshold of 0.9 here. For cases where the graph similarity is ≥0.9, the model output is considered to be true; for cases where it is less than 0.9, it is considered that hallucinations have occurred. At this time, the nodes and edges related to the content of subgraph A in subgraph B are first converted into triples, recorded as "triple_fact", and then the "local large model + prompt" method is used to send the problematic sentences and "triple_fact" into the model for repair. After the repair is completed, the results are output. Combined with the knowledge graph verification and repair functions, it can not only effectively detect hallucination problems, but also ensure the factual accuracy of the output content, and provide an extensible verification method to meet the needs of different fields.

[0100] Through the seamless connection of serial workflows, an efficient closed-loop method is formed. In terms of data circulation, first, the confidence assessment makes a preliminary confidence judgment on the output content generated by the large language model, marks the sentences with low confidence, and passes them to the fact extraction module for further processing; fact extraction simplifies the low-confidence sentences, extracts the basic ideas and converts them into structured triples, which are passed to the knowledge graph verification in a standardized data format; the knowledge graph verification verifies according to the existing knowledge graph, compares the relevant knowledge in the graph with the generated content, and repairs the content that fails the verification, and finally returns the corrected content that conforms to the actual facts.

[0101] Overall, this method effectively improves the credibility of the content generated by the large language model through precise division of labor and collaborative work of modules, and avoids erroneous information caused by hallucinations. It is particularly suitable for fields that require high accuracy, such as education, medical care, and law. By automatically detecting and repairing hallucinations in the content generated by the large language model, this method not only improves the accuracy and reliability of the large language model in practical applications, but also enhances its interpretability and applicability in practical scenarios, showing great application potential and broad market prospects.

[0102] Embodiment 2 This embodiment provides a detection and repair system for large language model output hallucinations, including: A question-answering module is configured to: obtain a question, input the question into a large language model, and obtain an answer content generated by the large language model; The confidence calculation module is configured to: divide the answer content into a number of sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds a set threshold. If so, directly output the answer content; if not, proceed to the next step; A fact extraction module is configured to: extract facts from sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: a subject, a predicate and an object; and generate a subgraph A of each sentence based on the triple data of each sentence; The detection and repair module is configured as follows: based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph; calculate the similarity between subgraph A and subgraph B, if the similarity is greater than a set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain a repaired subgraph A; obtain a repaired sentence according to the triple data corresponding to the repaired subgraph A; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A detection and repair method for output hallucination of a large language model, characterized by: include: Get the question, input the question into the large language model, and get the answer content generated by the large language model; Divide the answer content into several sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds the set threshold. If it exceeds, directly output the answer content, if not, proceed to the next step; Fact extraction is performed on sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: subject, predicate and object; subgraph A of each sentence is generated based on the triple data of each sentence; Based on the subject in the triple data of each sentence, subgraph B is extracted from the local knowledge graph; the similarity between subgraph A and subgraph B is calculated. If the similarity is greater than the set threshold, the answer content generated by the large language model is considered correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, subgraph A is repaired according to subgraph B to obtain a repaired subgraph A; based on the triple data corresponding to the repaired subgraph A, a repaired sentence is obtained; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

2. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: Calculate the confidence of each sentence, including: Confidence score of the sentence , with each Confidence The specific calculation formula is: ； in, is the weighted average score of the current sentence, is the weighted variance score of the current sentence, and the specific formulas are: ； ； in, for Confidence score of Valid for the current sentence The number of Representative Valid The part-of-speech weight; The confidence score , the specific formula is: ； in, , , are three weight coefficients, which control the probability scores respectively , distribution score , sequence score The importance of express the importance of .

3. The method for detecting and repairing output hallucination of a large language model as claimed in claim 2, characterized in that: , and its calculation formula is: Determined by its part of speech and position weight: ； in, Represents the part-of-speech weight, which is the weighted average score of the sentence ; represents the position weight, and It is related to the position in the current sentence. The specific formula is: ； in, represent The position in the current sentence, Represents all the number, is the natural logarithm; Represents the probability score, and its specific formula is: ； in, and Representatives of candidates The highest probability value and the second highest probability value, The normalized probability represents the highest probability value; , , is the probability score The decomposition weight coefficients control Probability, and Difference, Standardization The importance of probability, and ; about The calculation formula is: ； in, Representatives Candidates , Representative Candidate The highest probability value among Representative Candidates The probability value of .

4. The method for detecting and repairing output hallucination of a large language model as claimed in claim 2, characterized in that: represents the distribution score, The specific calculation formula is: ； in, , , The distribution score The decomposition weight coefficients control the entropy ,variance , Gini coefficient the importance of ; is entropy, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of is the variance, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of Representative Candidate The probability mean of is expressed as ; is the Gini coefficient, and the calculation formula is: ； in, Representative candidate quantity, Representative Candidates The probability value of .

5. The method for detecting and repairing output hallucination of a large language model as claimed in claim 2, characterized in that: Said Represents the sequence score, evaluation The coherence in the generated sequence is calculated as: ； in, , are weight coefficients, controlling the coherence scores respectively and position weight the importance of ; is the coherence score, indicating the current With the previous The probability continuity of is calculated as: ； in, Represents the current The maximum probability candidate The probability value of Represents the previous The maximum probability candidate The probability value of .

6. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: Determine whether the confidence of each sentence exceeds the set threshold, where the set threshold refers to: Construct a data set; divide the data set into a training set and a validation set according to the set ratio; Construct a Transformer Encoder network and use the training set to train the network. When the loss function value no longer decreases, stop training and obtain the trained network. The trained network is verified using the validation set. On this basis, different thresholds in the prediction probability range [0, 1] are enumerated, and the F1 scores of the validation set under different set thresholds are calculated. The threshold corresponding to the maximum F1 score is taken as the final set threshold. Select threshold: Enumerate different thresholds within the prediction probability range [0, 1] , here we use the fixed step method with a step size of 0.01, and gradually search in the range of [0, 1], and classify according to the following rules: ； Among them, the classification results A value of 0 means no hallucinations occur, and a value of 1 means hallucinations occur. represents the confidence score of the sentence, is the threshold of the enumeration; Calculate different thresholds on the validation set Next Fraction, The score calculation formula is: ； in, For accuracy, Indicates the number of true positive examples among the samples predicted by the model as positive examples; is the recall rate, It represents the proportion of all true positive examples that the model can correctly predict as positive examples. The calculation formula is as follows: ； ； in, is the true positive, the number of correctly predicted positive examples; is the number of false positive examples, FalsePositive, which is the number of false positive examples predicted as positive examples; is the number of false negative examples, which is the number of false negative examples predicted as negative examples; Next Score, so When the score is maximum Recorded as ,Will as the final static threshold.

7. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: The fact extraction is performed on sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, including: Input sentences with confidence levels lower than a set threshold into the trained BERT model, and the model outputs key sentences in the sentences, including subject, predicate, and object; For the key sentence, triple data is extracted to obtain the first entity-relationship-second entity data; wherein the first entity is the subject in the key sentence; the relationship is the predicate in the key sentence; the second entity is the object in the relationship sentence; The sentences with confidence levels lower than the set threshold are input into the trained BERT model, and the model outputs the key sentences in the sentences, including: Construct a data set; use the data set to train the BERT model. During the training process, the input value is a sentence whose confidence is lower than a set threshold, and the output value is a key sentence, which includes a subject, a predicate, and an object. After the model training is completed, call the trained model to output the key sentences in the sentence. Generate a subgraph A of each sentence based on the triple data of each sentence, including: mapping the subject and object in the triple to the subject node and object node of the subgraph A respectively, and the attributes of the subject node and the object node both include entity names; and use the predicate in the triple as a connecting edge between the subject node and the object node; Based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph, including: according to the subject in each sentence triple data, extract subgraph B from the local knowledge graph, the depth of subgraph B is 2, and subgraph B includes a first entity node, a first connecting edge, and a second entity node; the first entity node is the subject in the triple data.

8. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: The calculation of the similarity between subgraph A and subgraph B includes: calculating the similarity between subgraph A and subgraph B, first defining the nodes in subgraph A and subgraph B as , , , , the total similarity score formula is: ； in, represents the edge similarity law, Representative point similarity law; The calculation formula is: ； The calculation formula is: ； in, , This is the result of embedding the attribute value of the node. , It is the result after embedding the attribute value of the edge; Represents cosine similarity, and the calculation formula is: ； in, represent The module length, represent The mold length.

9. The method for detecting and repairing output hallucination of a large language model as claimed in claim 1, characterized in that: If the similarity is less than the set threshold, the sub-image A is repaired according to the sub-image B to obtain the repaired sub-image A, including: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, triple data whose second entity is the object of subgraph A is filtered out from subgraph B; Embed the relationship between the filtered triple data and the predicate of subgraph A, and then calculate the cosine similarity between the relationship between the filtered triple data and the predicate of subgraph A; If the cosine similarity is greater than 0.9, the predicate of subgraph A is modified and replaced based on the entity-relationship-entity data of the screened triple data to obtain the repaired subgraph A; If the similarity is less than the set threshold, the sub-image A is repaired according to the sub-image B to obtain the repaired sub-image A, further comprising: From the triple data of subgraph A, the subject of subgraph A, the predicate of subgraph A and the object of subgraph A are decomposed; According to the subject of subgraph A, select triples from subgraph B whose relations are the predicate of subgraph A; Based on the entity-relationship-entity data of the screened triple data, the object of subgraph A is modified and replaced to obtain the repaired subgraph A.

10. A detection and repair system for output hallucinations in large language models, characterized by: include: A question-answering module is configured to: obtain a question, input the question into a large language model, and obtain an answer content generated by the large language model; The confidence calculation module is configured to: divide the answer content into a number of sentences, calculate the confidence of each sentence, and determine whether the confidence of each sentence exceeds a set threshold. If so, directly output the answer content; if not, proceed to the next step; A fact extraction module is configured to: extract facts from sentences whose confidence is lower than a set threshold to obtain triple data of each sentence, wherein the triple data includes: a subject, a predicate and an object; and generate a subgraph A of each sentence based on the triple data of each sentence; The detection and repair module is configured as follows: based on the subject in each sentence triple data, extract subgraph B from the local knowledge graph; calculate the similarity between subgraph A and subgraph B, if the similarity is greater than a set threshold, it is considered that the answer content generated by the large language model is correct, and the answer content generated by the large language model is directly output; if the similarity is less than the set threshold, repair subgraph A according to subgraph B to obtain a repaired subgraph A; obtain a repaired sentence according to the triple data corresponding to the repaired subgraph A; after all sentences with confidence levels lower than the set threshold are repaired, the final answer content is obtained.

Images