A method and system for inhibiting hallucinations of llm agents and securing trustworthy outputs

Abstract

This invention discloses a method and system for suppressing hallucinations in LLM agents and ensuring credible output. The method includes: real-time monitoring of internal signals in three dimensions during generation to mark high-risk hallucination intervals; dual-path external verification after generation using multi-source knowledge cross-validation and adversarial self-validation; credibility-level labeling and visualization output; construction of a hallucination-correction preference dataset and execution of DPO fine-tuning training; extraction of high-frequency hallucination patterns and construction of a pre-interception library. The system includes a monitoring module during generation, a verification module after generation, a credibility-level output module, a hallucination feedback learning module, and a hallucination pattern interception module. This invention achieves two-stage real-time detection and dual-channel multi-dimensional verification, significantly reducing system latency and improving verification accuracy; enhances security in high-risk scenarios through a credibility-level output mechanism; and establishes a data flywheel-style closed-loop optimization system to achieve model self-evolution, which can be widely applied in fields such as medicine, finance, and law.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence security technology, and in particular to a method and system for suppressing LLM agent illusions and ensuring credible output. Background Technology

[0002] With the rapid evolution of artificial intelligence technology, intelligent agents built on Large Language Models (LLM) have been widely applied in many vertical fields such as medical consultation, legal advice, financial analysis, and intelligent customer service. However, LLM intelligent agents generally suffer from the illusion defect in the text generation process, that is, the generated content is grammatically coherent and seems reasonable on the surface, but in fact it does not conform to the facts, is fabricated out of thin air, or contradicts the input context.

[0003] In high-risk application scenarios where factual accuracy is extremely critical, hallucination outputs can severely undermine the credibility of LLM agents' outputs, misleading user decisions and potentially leading to serious liability incidents and safety risks. Existing detection and suppression schemes for LLM agents exhibit several technical shortcomings: First, they rely excessively on post-generation verification, lacking real-time intervention capabilities during the generation phase, resulting in high system response latency and failing to meet the demands of real-time interactive scenarios. Second, verification methods are limited, mostly relying on fixed external knowledge bases, leading to low coverage of long-tail knowledge, the latest information, and complex logical reasoning, and poor adaptability to open domains. Third, they lack closed-loop self-evolution and self-repair mechanisms, merely detecting and intercepting hallucinations without transforming them into effective feedback for model optimization, resulting in the recurrence of similar hallucination errors.

[0004] To address the shortcomings of the existing technologies, there is an urgent need for an LLM agent illusion suppression method that balances real-time performance and detection accuracy, possesses multi-dimensional verification capabilities, and enables continuous model optimization, so as to fundamentally ensure the credibility of the LLM agent's output. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for suppressing hallucinations in LLM agents and ensuring credible output. This method addresses the technical problems of high detection latency, limited verification methods, and lack of model self-repair mechanisms in existing LLM agent hallucination detection technologies. It enables real-time hallucination monitoring, multi-dimensional accurate verification, graded display of output credibility, and closed-loop optimization of the model based on hallucination cases during the generation process of LLM agents. Ultimately, this significantly reduces the hallucination rate of LLM agents and ensures the authenticity and credibility of their output content.

[0006] To achieve the above objectives, the present invention provides the following technical solution: According to one aspect of the present invention, a method for suppressing LLM agent hallucinations and ensuring credible output is provided, comprising the following steps: When S1 and LLM agents generate text, they monitor the internal signals of the model in real time from three dimensions: word-level uncertainty, attention pattern, and internal consistency of content. If an alert is triggered, the corresponding text is marked as a high-hallucination risk zone. S2. Perform dual-path external verification of multi-source knowledge cross-validation and adversarial self-validation on texts in high-illusion-risk intervals, and output quantitative verification results; S3. Combining internal monitoring and external verification results, generate text hierarchical labeling credibility tags and visualize the output to users; S4. Construct a hallucination-correction preference dataset by using suspected hallucination outputs as counterexamples and corresponding correct facts as positive examples. When the preset threshold is reached, use the direct preference optimization algorithm to fine-tune the LLM agent. S5. Extract high-frequency hallucination prompt word patterns to build an interception library, match new input prompt words in real time, and issue an early warning and adjust the LLM agent generation strategy if a match is successful.

[0007] According to an embodiment of the present invention, in step S1, the word-level uncertainty perception is to calculate the entropy value of the predicted probability distribution of each word output by the decoder, and to determine by a dynamic sliding window if the entropy values ​​of multiple consecutive words exceed a preset dynamic threshold, thereby triggering an early warning. In step S1, the attention pattern anomaly detection involves extracting the cross-attention weight matrix, calculating the weight distribution clustering degree of the input context entities, and triggering an early warning if the clustering degree is lower than a preset benchmark. In step S1, the internal consistency verification of the content is an embedded implicit backtracking check mechanism, which calculates the cosine similarity of the semantic vectors of the current and historically generated content. If the similarity deviates significantly in the negative direction, an early warning is triggered.

[0008] According to an embodiment of the present invention, in step S2, the multi-source knowledge cross-validation involves extracting fact statements in the form of triples from risky texts and sending them in parallel to a structured knowledge graph, an authoritative document retrieval system, and an open-domain search engine, and using a majority voting algorithm to calculate the confidence score of the fact being true; the adversarial self-validation involves instantiating an independent validating agent, generating exploratory prompts for the fact statements and re-inputting them into the LLM agent, calculating the variance of multiple responses based on the principle of self-consistency, and determining that the variance exceeds a preset threshold as a suspected hallucination.

[0009] According to an embodiment of the present invention, in step S3, the credibility labels include three categories: highly credible, to be verified, and suspected hallucination; a highly credible label is marked if there is no warning from internal monitoring and / or the external verification confidence score is higher than the first threshold; a label to be verified is marked if there is a warning from internal monitoring but external verification is passed or the confidence score is between the first and second thresholds; and a suspected hallucination label is marked if there is a warning from internal monitoring and the external verification confidence score is lower than the second threshold or the adversarial self-verification variance exceeds the threshold.

[0010] According to an embodiment of the present invention, in step S1, the hallucination-correction preference dataset is a standardized positive-negative example pairing dataset, where negative examples are the output logs of LLM agents with suspected hallucination labels, and positive examples are objective and correct factual texts corresponding to hallucination content; the direct preference optimization algorithm updates the weight parameters of the LLM agent through gradient descent to suppress the probability of its output hallucination features.

[0011] According to an embodiment of the present invention, in step S5, high-frequency hallucination cue word patterns are extracted using TF-IDF, keyword matching, or semantic similarity methods; the generation strategy is adjusted to make the LLM agent prioritize calling relevant data from external authoritative knowledge sources and force the generation of text based on objective facts.

[0012] According to one embodiment of the present invention, the LLM agent is an LLM agent adapted to the vertical fields of medical, financial, legal and industrial decision-making, and the internal signal monitoring threshold, external verification data source, credibility grading standard and hallucination mode interception rules are customized for different fields.

[0013] According to one embodiment of the present invention, the three-dimensional monitoring process in step S1 and the dual-path external verification process in step S2 both support parallel processing. The degree of parallel processing is adjusted according to the hardware resource configuration to reduce the overall system response latency.

[0014] According to one embodiment of the present invention, the preset threshold is a configurable parameter that can be adjusted according to the application scenario, model size, and accuracy requirements of the LLM agent.

[0015] On the other hand, the present invention also provides a system for suppressing LLM agent hallucinations and ensuring credible output, the system comprising: a generation monitoring module, a generation verification module, a credibility-level output module, a hallucination feedback learning module, and a hallucination pattern interception module; The generation monitoring module integrates a lexical entropy calculation unit, an attention weight analysis unit, and a semantic consistency verification unit to achieve real-time monitoring of three dimensions of signals within the model and marking of high hallucination risk intervals. The post-generation verification module integrates a fact extraction unit, a multi-source retrieval verification unit, and an adversarial self-verification unit, which are used to realize dual-path external verification of risky texts and output quantitative results. The credibility grading output module integrates a credibility scoring unit, a tag rendering unit, and a terminal interaction unit to generate text credibility grading annotations and visual output. The hallucination feedback learning module integrates a log recording unit, a dataset construction unit, and a DPO fine-tuning training unit, which are used to realize hallucination log recording, preference dataset construction, and LLM agent fine-tuning training. The hallucination pattern interception module integrates a prompt word pattern extraction unit, a pattern matching unit, and a generation strategy adjustment unit, which are used to realize high-frequency hallucination pattern extraction, interception library construction, and new input pre-warning and generation strategy adjustment.

[0016] The present invention provides a method and system for suppressing LLM agent hallucinations and ensuring credible output. Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Dual-stage real-time detection significantly reduces system response delay.

[0017] This innovative two-stage hallucination detection framework combines internal monitoring during generation with external verification after generation. By pre-filtering the internal state signals of the LLM agent during generation, external verification requests are only initiated for high-risk content, significantly reducing the number of external knowledge source searches and computational resource consumption. While ensuring a high detection rate of hallucinations, it significantly reduces the overall response latency of the system, meeting the usage requirements of real-time human-computer interaction scenarios.

[0018] (2) Dual-channel multidimensional verification improves the accuracy and coverage of hallucination detection.

[0019] A dual-channel verification system combining internal signals and external verification is established. Internal signal monitoring utilizes the model's own generative features to achieve real-time early warning without dependencies. External verification combines multi-source knowledge cross-verification and adversarial self-verification, covering multiple dimensions such as structured knowledge graphs, authoritative documents, open domain information, and model self-consistency verification. This solves the problem of the single verification method in existing technologies, significantly improves the verification capability for long-tail knowledge, the latest information, and complex logical reasoning content, and significantly enhances open domain adaptability.

[0020] (3) The credibility-level output improves the security of use in high-risk scenarios.

[0021] Breaking away from the "black box" absolute confidence output mode of LLM agents, it adds three types of structured credibility tags to the generated text: high credibility, unverified, and suspected hallucination, and realizes visual rendering display. This explicitly conveys the uncertainty of machine information generation to users, giving them the right to know and judge the quality of information, and greatly improves the security boundary and feasibility of LLM agents in high-risk scenarios such as medical, financial, and legal fields.

[0022] (4) Data flywheel closed-loop optimization can realize model self-evolution.

[0023] By automatically converting suspected hallucination cases into a "hallucination-correction" preference dataset, and combining it with the direct preference optimization algorithm to complete the regular fine-tuning of the model, and building a hallucination pattern interception library to realize the early warning of high-risk inputs and the adjustment of generation strategies, the LLM agent has the ability to "learn from mistakes" and self-repair and self-evolve, continuously reducing the occurrence rate of hallucinations from the root and achieving long-term iterative improvement of model credibility.

[0024] (5) It has strong domain adaptability and scalability, and is suitable for multiple application scenarios.

[0025] The core modules of the system are decoupled. The internal signal monitoring threshold, external verification data source, credibility grading standard, and hallucination mode interception rules in the method are all configurable parameters, which can be customized for different vertical fields such as medical, financial, legal, and industrial decision-making. At the same time, the monitoring and verification processes in the method support parallel processing, and the degree of parallelism can be adjusted according to hardware resource configuration to adapt to LLM agent application scenarios with different scales and accuracy requirements. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a method for suppressing LLM agent hallucinations and ensuring reliable output according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a system for suppressing LLM agent hallucinations and ensuring reliable output according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the application of LLM intelligent agent hallucination detection and output in an embodiment of the present invention. Detailed Implementation

[0027] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0028] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0030] like Figure 1 As shown, a flowchart of a method for suppressing LLM agent hallucinations and ensuring credible output is presented. The method includes the following steps: S101: Real-time monitoring of three-dimensional internal signals to mark high-risk hallucination zones; During the process of LLM agent decoding and generating text sequences, the internal state signals of the model are extracted and analyzed in real time. Simultaneous monitoring is carried out from three dimensions: word-level uncertainty, attention pattern, and internal consistency of content. When the monitoring result of any dimension triggers the warning condition, the currently generated text segment is immediately marked as a high hallucination risk zone, and the marking information is synchronized to the subsequent verification stage, so as to realize real-time risk identification and pre-filtering in the generation stage.

[0031] S102: Dual-channel external verification, outputting quantitative verification results; For texts in the high-illusion-risk range marked in S101, a dual-path external verification mechanism of multi-source knowledge cross-validation and adversarial self-validation is triggered. First, factual statements are extracted from the risky text. On one hand, the factual statements are sent to multiple external knowledge sources to complete cross-validation and calculate confidence scores. On the other hand, an independent verification agent asks reverse questions to the LLM agent, and the reliability of the answers is determined based on the principle of self-consistency. Finally, the quantitative verification results, including confidence scores, answer variance, and illusion determination conclusions, are output.

[0032] S103: Credibility rating labeling, with visual output to users; Combining the internal signal monitoring and early warning results of S101 with the external verification and quantification results of S102, a comprehensive credibility score is calculated for each text segment generated by the LLM agent. Based on the score range, three structured credibility tags—high credibility, unverified, and suspected hallucination—are attached to each segment. The generated content with credibility tags is then visualized and rendered through the terminal interface, explicitly conveying the uncertainty of the machine's information generation to the user.

[0033] S104: Construct a preference dataset and fine-tune the model using the direct preference optimization algorithm; The output logs of LLM agents labeled "suspected hallucination" are used as negative examples, while objectively correct factual texts corresponding to the hallucination content are obtained or generated as positive examples. These positive and negative examples are paired to construct a standardized hallucination-correction preference dataset. When the number of samples in the dataset accumulates to a preset threshold, the Direct Preference Optimization (DPO) algorithm is used to fine-tune the training of the LLM agent. Gradient descent is used to update the model's weight parameters, suppressing the probability of it outputting hallucination features.

[0034] S105: Extract high-frequency hallucination patterns and build a pre-interception library to achieve early warning.

[0035] This system mines and extracts high-frequency trigger word patterns from hallucination cases. After structuring these patterns using methods such as TF-IDF, keyword matching, or semantic similarity, the patterns are stored in a hallucination pattern interception library. When new user input is received, the input content is matched against the trigger word patterns in the interception library in real time. If a match is successful, a pre-warning message is immediately output to the terminal, and a generation strategy adjustment instruction is sent to the LLM agent, instructing it to prioritize the use of external authoritative knowledge source data to generate text, thereby reducing the probability of hallucinations from the source.

[0036] like Figure 2 The diagram shows a system schematic for suppressing LLM agent hallucinations and ensuring credible output. This system, used to implement the above method, includes: The generation monitoring module integrates a lexical entropy calculation unit, an attention weight analysis unit, and a semantic consistency verification unit. It is used to extract the internal state signals of the LLM agent during the text generation process in real time, complete the monitoring and early warning judgment in three dimensions, mark the high hallucination risk zone, and synchronize the marked data to the post-generation verification module.

[0037] The post-generation verification module communicates with the generation-time monitoring module and integrates a fact extraction unit, a multi-source retrieval verification unit, and an adversarial self-verification unit. It is used to extract fact statements from text in high-illusion-risk intervals, perform multi-source knowledge cross-verification and adversarial self-verification, output quantitative verification results, and synchronize the results to the credibility grading output module.

[0038] The credibility grading output module communicates with the generation monitoring module and the generation verification module. It integrates a credibility scoring unit, a tag rendering unit, and a terminal interaction unit. It is used to calculate the credibility score of the generated text based on the internal monitoring and external verification results, complete the grading annotation and visualization rendering, and synchronize logs with suspected hallucination tags to the hallucination feedback learning module.

[0039] The hallucination feedback learning module communicates with the credibility grading output module and integrates a log recording unit, a dataset construction unit, and a DPO fine-tuning training unit. It is used to record hallucination case logs completely, construct a hallucination-correction preference dataset, and perform DPO fine-tuning training of the LLM agent when the dataset reaches a preset threshold. At the same time, it synchronizes the high-frequency hallucination patterns discovered to the hallucination pattern interception module.

[0040] The hallucination pattern interception module is communicatively connected to the hallucination feedback learning module and the generation monitoring module. It integrates a prompt word pattern extraction unit, a pattern matching unit, and a generation strategy adjustment unit. It is used to extract and structure high-frequency hallucination prompt word patterns, build a hallucination pattern interception library, complete real-time pattern matching of new inputs, trigger advance warnings, and send generation strategy adjustment instructions to the LLM agent.

[0041] The core innovation of the invention lies in proposing a dynamic hallucination detection framework with a two-stage "generation-post-generation" and a two-channel "internal signal-external verification" approach. Combined with credibility grading and human feedback reinforcement learning, it constructs a complete closed-loop technical solution from detection and interception to model self-cleaning. Figure 3 As shown, the method includes the following steps.

[0042] The first step is to build and execute an internal signal monitoring mechanism during generation.

[0043] During the process of model decoding to generate text sequences, the internal state signals of the large language model are extracted and analyzed in real time: (1) Token-level uncertainty perception: The predicted probability distribution entropy value of each token output by the decoder is calculated in real time. A dynamic sliding window is set. When the entropy values ​​of multiple consecutive tokens in the window exceed the preset dynamic threshold, an early warning is triggered, and the currently generated text fragment is marked as a "high hallucination risk zone".

[0044] (2) Attention Pattern Anomaly Detection: Extract the weight matrix of the cross-attention mechanism when generating the current word. Calculate the distribution clustering degree of the current attention weights on the input context entities. If the clustering degree is lower than the preset benchmark, it is determined that the model has deviated from the input factual basis during generation, triggering an early warning.

[0045] (3) Internal consistency check: An implicit backtracking check mechanism is embedded in the decoding strategy. The semantic vector of the currently generated sentence is extracted and a contradiction detection calculation is performed with the semantic vector of the previously generated historical content. If the cosine similarity shows a significant negative deviation, it is judged as logical inconsistency.

[0046] The second step is to build and execute a post-generation external verification mechanism.

[0047] For text marked as "high hallucination risk zone" in the first step, trigger two-way verification: (1) Multi-source knowledge cross-validation: The relation extraction algorithm is used to extract factual statements in the form of triples from the risky text, and the factual statements are sent in parallel to the structured knowledge graph, the authoritative document retrieval system and the open domain search engine. The retrieval feedback from each data source is received, and the majority voting algorithm is used to calculate the confidence score of the fact as true.

[0048] (2) Adversarial self-verification: An independent verification agent is instantiated. This agent generates multiple exploratory prompts from different perspectives based on the extracted factual statements and re-inputs them into the large language model to ask questions. The large language model's multiple responses to the same fact are compared, and the variance of the responses is calculated based on the principle of self-consistency. If the variance is higher than a threshold, it is judged as a possible hallucination.

[0049] The third step is to perform credibility labeling and hierarchical output.

[0050] Based on the detection results of the first and second steps, structured credibility tags are attached to each text paragraph generated by the agent and rendered to the end user: High-confidence label: Internal signal monitoring did not trigger an alert, and / or the external verification confidence score is higher than the first threshold.

[0051] Labels to be verified: internal signal monitoring triggers an alert but external verification passes, or internal signals are normal but external verification confidence scores are between the first and second thresholds.

[0052] Suspected hallucination label: internal signal monitoring triggers an alert, and the external verification confidence score is below the second threshold, or the adversarial self-verification variance is above the threshold.

[0053] The fourth step is to implement the hallucination feedback learning mechanism.

[0054] The original output logs labeled "suspected hallucination" are recorded as negative examples; simultaneously, the corresponding correct factual text is obtained or generated as positive examples. These positive and negative examples are paired to construct a "hallucination-correction" preference dataset. The Direct Preference Optimization (DPO) algorithm is periodically used to fine-tune and update the large language model, suppressing the probability of its output hallucination features. At the same time, high-frequency hallucination triggering word patterns are extracted and stored in a hallucination pattern interception library.

[0055] Example 1: The LLM intelligent agent for medical and health question answering is an exemplary application.

[0056] This embodiment uses a medical and health question-answering LLM intelligent agent as an example application. The core configurable parameters of this invention are customized to suit the application scenarios and accuracy requirements of the medical and health question-answering LLM intelligent agent: Parameters monitored during generation: dynamic threshold for lexical entropy value is 1.5, preset baseline for attention weight distribution clustering is 20%, negative deviation threshold for semantic consistency cosine similarity is -0.3, and dynamic sliding window size for lexical monitoring is 5. External validation parameters: First confidence threshold is 0.8, second confidence threshold is 0.5, and adversarial self-validation response variance threshold is 0.5; Model fine-tuning parameters: The threshold for the number of accumulated data in the illusion-correction preference dataset is 1000; Pattern extraction parameters: The TF-IDF method is used to extract high-frequency hallucination cue word patterns.

[0057] The specific implementation steps of the method to suppress LLM agent hallucinations and ensure credible output are as follows: Step 1: During generation, the monitoring module performs three-dimensional monitoring of internal signals. A user entered a question on a healthcare Q&A terminal: "The patient has a history of hypertension and is currently taking amlodipine. Can the patient also take cold medicine containing pseudoephedrine because of a cold?" The LLM agent initiates token-by-token decoding to generate an answer. Simultaneously, the monitoring module starts real-time monitoring, with each unit performing its assigned tasks: The word entropy calculation unit calculates the predicted probability distribution entropy value of each word output by the decoder in real time, and determines whether the entropy value exceeds the limit through a dynamic sliding window of size 5. The attention weight analysis unit extracts the weight matrix of the cross-attention mechanism when generating the current word in real time, and calculates the clustering degree of the attention weight distribution for the input context entities "hypertension", "amlodipine", and "pseudoephedrine". The semantic consistency verification unit embeds an implicit backtracking check mechanism in the decoding strategy, extracts the semantic vector of the currently generated sentence in real time, and calculates the cosine similarity with the semantic vector of the previously generated historical content.

[0058] When the model generated the text "Pseudoephedrine promotes the metabolism of amlodipine and they can be taken together," the generation monitoring module detected the following: the average entropy of five consecutive lexical units reached 2.8 (higher than the dynamic threshold of 1.5); the attention weight for "hypertension" and "amlodipine" dropped below 10% (lower than the preset baseline of 20%); and the cosine similarity of the semantic vector between the generated content and the preceding text regarding "contraindications for hypertension medication" was -0.5 (lower than the negative deviation threshold of -0.3). All three dimensions triggered warnings. The generation monitoring module immediately marked the text fragment as a high-risk hallucination zone and synchronized the marked data to the post-generation verification module.

[0059] Step 2: Dual-path external verification of high-risk text (specifically performed by the post-generation verification module); After the generation and verification module receives the high-hallucination risk zone marker data, it immediately initiates dual-path external verification, with each unit performing its assigned tasks: The fact extraction unit uses a relation extraction algorithm to extract two core fact statements in the form of triples from the risk text: [Pseudoephedrine, promotes, amlodipine metabolism] and [Hypertensive patients + amlodipine, can be used in combination, pseudoephedrine cold medicine]. The multi-source retrieval verification unit sends the above factual statement in parallel to two external knowledge sources: a medical knowledge graph and the authoritative pharmaceutical database of the National Medical Products Administration. After receiving the retrieval feedback, it uses a majority voting algorithm to calculate the confidence score for the fact to be true. Both knowledge sources return "Pseudoephedrine has a vasoconstrictive effect, which can cause an increase in blood pressure, and there is a clear contraindication to its use with the antihypertensive drug amlodipine," therefore, the confidence score for the fact to be true is 0. An adversarial self-verification unit instantiates an independent medical verification agent. Based on the extracted factual statement, it generates an exploratory prompt: "What drug interactions will occur if a hypertensive patient takes amlodipine and a cold medicine containing pseudoephedrine simultaneously?" This prompt is then re-inputted into the medical health question-answering LLM agent. The model's second response is: "Pseudoephedrine constricts blood vessels, leading to increased blood pressure, which antagonizes the antihypertensive effect of amlodipine. Combined use will result in uncontrolled increases in blood pressure, indicating a clear contraindication." Comparing the two responses, the variance calculated based on the self-consistency principle is 0.95 (higher than the variance threshold of 0.5), thus indicating a possible hallucination.

[0060] After generation, the verification module will synchronize the quantitative verification result of "confidence score 0, self-consistency variance 0.95, judged as suspected hallucination" to the credibility grading output module.

[0061] Step 3: Credibility rating labeling and visualization output (executed by the credibility rating output module); After receiving internal monitoring data and external verification data, the credibility grading output module assigns tasks to each unit: The credibility scoring unit calculated the overall credibility score of the risky text fragment to be 0.1 based on "internal signal three-dimensional warning, external verification confidence score of 0, and self-consistency variance of 0.95"; at the same time, it calculated the overall credibility score of other risk-free text fragments in the response, such as "basic pharmacological effects of amlodipine", to be 0.98. The tag rendering unit assigns a high-confidence tag to paragraphs with a confidence score of 0.98 and a suspected hallucination tag to paragraphs with a confidence score of 0.1, based on the confidence score and tagging rules. The terminal interaction unit visualizes the tagged generated content on the user interface of the medical and health Q&A terminal: a green checkmark icon is displayed next to highly credible paragraphs, and a red exclamation mark icon and a red highlighted underline are added to paragraphs suspected of being hallucinations. A risk warning box pops up at the bottom of the interface, with the message: "Potential medication contraindication error detected: External authoritative medical databases show that pseudoephedrine has a vasoconstrictive effect and can cause elevated blood pressure. It is contraindicated with the antihypertensive drug amlodipine, and combined use may lead to uncontrolled blood pressure. Please follow your doctor's advice for specific medication use."

[0062] Step 4: Construct the preference dataset and fine-tune the DPO model (executed by the illusion feedback learning module); After the credibility grading output module synchronizes the output logs marked as "suspected hallucination" to the hallucination feedback learning module, the various units within this module will have the following tasks: The log recording unit fully records the user input of this interaction, the hallucination output of the LLM agent, the quantitative data monitored during generation, and the result data of the verification after generation, forming a standardized medical scenario hallucination case log; The dataset construction unit takes the hallucination output "Pseudoephedrine promotes the metabolism of amlodipine and they can be taken together" as a negative example (Rejected) and the correct conclusion generated based on the medical knowledge graph and authoritative medical database "It is not recommended for hypertensive patients to take amlodipine in combination with cold medicine containing pseudoephedrine. Pseudoephedrine will constrict blood vessels and cause blood pressure to rise, antagonizing the antihypertensive effect of amlodipine. There is a clear contraindication to use" as a positive example (Chosen). The positive and negative examples are then paired and added to the hallucination-correction preference dataset. When the number of samples in the hallucination-correction preference dataset reaches a preset threshold of 1000, the DPO fine-tuning training unit initiates the model fine-tuning training process. Using this dataset as training samples, the Direct Preference Optimization (DPO) algorithm is used to update the weight parameters of the medical and health question-answering LLM agent. The generation strategy of the model is optimized through gradient descent to suppress the probability of it outputting hallucination content in the "hypertensive antihypertensive drugs + pseudoephedrine" scenario.

[0063] Step 5: Extract high-frequency hallucination patterns and pre-interception (executed by the hallucination pattern interception module); After the hallucination feedback learning module synchronizes the high-frequency hallucination cases in medical scenarios discovered during the fine-tuning training to the hallucination pattern interception module, the various units within this module have their respective tasks: The cue word pattern extraction unit uses the TF-IDF method to extract high-frequency cue word patterns that trigger hallucinations from hallucination cases in medical scenarios: "hypertension + [name of antihypertensive drug] + [name of cold medicine] + combined use / taken together"; The pattern matching unit performs structured processing on the prompt word pattern, stores it in the hallucination pattern interception library, and establishes real-time matching rules based on semantic similarity. Generation Strategy Adjustment Unit: When the user inputs "Can a patient with hypertension take nifedipine and also take cold medicine containing pseudoephedrine?" is received, the pattern matching unit performs real-time matching of the input with patterns in the interception library. Upon successful matching, a pre-warning prompt is immediately output to the terminal: "This question involves the contraindication of using antihypertensive drugs and cold medicines together. It is a high-risk consultation question. The answer will be verified in conjunction with an authoritative medical database. Please refer to the advice of professional doctors." At the same time, a generation strategy adjustment instruction is sent to the generation monitoring module, causing the LLM agent to prioritize calling relevant data from the medical knowledge graph and authoritative medical database when generating the answer to this question. This forces the model to generate the answer based on objective facts, reducing the probability of hallucinations from the source.

[0064] This embodiment verifies the effectiveness of the method and system described in this invention through the practical application of a medical and health question-answering LLM intelligent agent: it achieves real-time hallucination monitoring and accurate labeling of high-risk intervals during the generation process, achieves accurate hallucination determination through dual-path external verification, provides users with intuitive information quality references through credibility-level output, and achieves model self-evolution and source hallucination suppression through DPO fine-tuning training and pre-mode interception, effectively solving the pain points of high hallucination detection latency, single verification methods, and lack of model self-repair mechanism in the prior art.

[0065] The method and system described in this invention are not only applicable to LLM agents for medical and health question answering, but also adaptable to LLM agents in any vertical field such as financial analysis, legal counsel, intelligent customer service, and industrial decision-making. By simply customizing and adjusting configurable parameters such as internal signal monitoring thresholds, external verification data sources, credibility grading standards, and hallucination mode interception rules according to the application scenarios, model size, and accuracy requirements of different fields, a good hallucination suppression effect can be achieved, which has broad application prospects.

[0066] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and other materials. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0067] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for inhibiting hallucinations of LLM agents and guaranteeing trustworthy outputs, characterized in that, Includes the following steps: When S1 and LLM agents generate text, they monitor the internal signals of the model in real time from three dimensions: word-level uncertainty, attention pattern, and internal consistency of content. If an alert is triggered, the corresponding text is marked as a high-hallucination risk zone. S2. Perform dual-path external verification of multi-source knowledge cross-validation and adversarial self-validation on texts in high-illusion-risk intervals, and output quantitative verification results; S3. Combining internal monitoring and external verification results, generate text hierarchical labeling credibility tags and visualize the output to users; S4. Construct a hallucination-correction preference dataset by using suspected hallucination outputs as counterexamples and corresponding correct facts as positive examples. When the preset threshold is reached, use the direct preference optimization algorithm to fine-tune the LLM agent. S5. Extract high-frequency hallucination prompt word patterns to build an interception library, match new input prompt words in real time, and issue an early warning and adjust the LLM agent generation strategy if a match is successful.

2. The method according to claim 1, characterized in that, In step S1, the word-level uncertainty perception is calculated by computing the entropy value of the predicted probability distribution of each word output by the decoder. The judgment is made by a dynamic sliding window. If the entropy values ​​of multiple consecutive words exceed the preset dynamic threshold, an early warning is triggered. In step S1, the attention pattern anomaly detection involves extracting the cross-attention weight matrix, calculating the weight distribution clustering degree of the input context entities, and triggering an early warning if the clustering degree is lower than a preset benchmark. In step S1, the internal consistency verification of the content is an embedded implicit backtracking check mechanism, which calculates the cosine similarity of the semantic vectors of the current and historically generated content. If the similarity deviates significantly in the negative direction, an early warning is triggered.

3. The method according to claim 1, characterized in that, In step S2, the multi-source knowledge cross-validation involves extracting fact statements in the form of triples from the risky text, sending them in parallel to a structured knowledge graph, an authoritative document retrieval system, and an open-domain search engine, and using a majority voting algorithm to calculate the confidence score for the fact to be true. The adversarial self-verification involves instantiating an independent verification agent, generating exploratory prompts for the factual statement, re-inputting them into the LLM agent, and calculating the variance of multiple responses based on the principle of self-consistency. If the variance exceeds a preset threshold, it is determined to be a suspected hallucination.

4. The method according to claim 1, characterized in that, In step S3, the credibility labels include three categories: highly credible, unverified, and suspected hallucination. The label "highly credible" is marked if there is no warning from internal monitoring and / or the external verification confidence score is higher than the first threshold. The label "unverified" is marked if there is a warning from internal monitoring but external verification is passed or the confidence score is between the first and second thresholds. The label "suspected hallucination" is marked if there is a warning from internal monitoring and the external verification confidence score is lower than the second threshold or the adversarial self-verification variance exceeds the threshold.

5. The method according to claim 1, characterized in that, In step S1, the hallucination-correction preference dataset is a standardized positive-negative example pairing dataset. Negative examples are the output logs of LLM agents with suspected hallucination labels, and positive examples are objective and correct factual texts corresponding to hallucination content. The direct preference optimization algorithm updates the weight parameters of the LLM agent through gradient descent to suppress the probability of its output hallucination features.

6. The method according to claim 1, characterized in that, In step S5, TF-IDF, keyword matching, or semantic similarity methods are used to extract high-frequency hallucination cue word patterns; the generation strategy is adjusted to make the LLM agent prioritize calling relevant data from external authoritative knowledge sources and force the generation of text based on objective facts.

7. The method according to claim 1, characterized in that, The LLM agent is adapted to vertical fields such as medical, financial, legal, and industrial decision-making. It customizes the internal signal monitoring threshold, external verification data source, credibility grading standard, and hallucination mode interception rules for different fields.

8. The method according to claim 1, characterized in that, The three-dimensional monitoring process in step S1 and the dual-path external verification process in step S2 both support parallel processing. The degree of parallel processing can be adjusted according to the hardware resource configuration to reduce the overall system response latency.

9. The method according to claim 1, characterized in that, The preset threshold is a configurable parameter that can be adjusted according to the application scenario, model size, and accuracy requirements of the LLM agent.

10. A system for suppressing LLM agent hallucinations and ensuring credible output, for implementing the method according to any one of claims 1 to 9, characterized in that, The system includes: a generation monitoring module, a generation verification module, a credibility grading output module, a hallucination feedback learning module, and a hallucination pattern interception module. The generation monitoring module integrates a lexical entropy calculation unit, an attention weight analysis unit, and a semantic consistency verification unit to achieve real-time monitoring of three dimensions of signals within the model and marking of high hallucination risk intervals. The post-generation verification module integrates a fact extraction unit, a multi-source retrieval verification unit, and an adversarial self-verification unit, which are used to realize dual-path external verification of risky texts and output quantitative results. The credibility grading output module integrates a credibility scoring unit, a tag rendering unit, and a terminal interaction unit to generate text credibility grading annotations and visual output. The hallucination feedback learning module integrates a log recording unit, a dataset construction unit, and a DPO fine-tuning training unit, which are used to realize hallucination log recording, preference dataset construction, and LLM agent fine-tuning training. The hallucination pattern interception module integrates a prompt word pattern extraction unit, a pattern matching unit, and a generation strategy adjustment unit, which are used to realize high-frequency hallucination pattern extraction, interception library construction, and new input pre-warning and generation strategy adjustment.

Images