A hallucination detection and interpretable decision method and system for model fidelity hallucinations

Abstract

The application discloses a hallucination detection and explainable judgment method and system for model fidelity hallucination, and relates to the technical field of artificial intelligence. The method comprises the following steps: automatically generating candidate samples; performing multi-dimensional filtering processing on the candidate samples to obtain high-quality explanation type detection data through screening; supervising and fine-tuning an initial detection model by using the high-quality explanation type detection data; constructing a composite reward mechanism, rewarding and feeding back the output result of the fine-tuned detection model based on the composite reward mechanism to generate a composite reward signal; iteratively optimizing and training the fine-tuned detection model based on the composite reward signal to obtain a final hallucination detection model; and inputting an input document and a text to be detected into the final hallucination detection model to output a fidelity judgment result and a corresponding natural language explanation, which helps to solve the problem that the prior art cannot automatically generate high-quality and verifiable explanation fidelity hallucination detection while ensuring low cost and high detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for detecting and determining the interpretability of model fidelity illusions. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have been widely used in numerous scenarios such as retrieval augmented generation (RAG), text summarization, intelligent question answering, and multi-hop reasoning, becoming a core tool for information processing and interaction. However, the problem of "fidelity illusion," where the content generated by the model is inconsistent with the given context or factual basis, is becoming increasingly prominent. This seriously affects the reliability and security of large language models in high-credibility scenarios, and the accurate detection and judgment of illusions has become a critical issue that the industry urgently needs to address.

[0003] Existing hallucination detection methods can be mainly divided into three categories: detection methods based on the capabilities of large language models rely on large closed-source models, resulting in high inference costs, large response delays, and insufficient stability; dedicated hallucination detection classification models, although lower in cost, only output binary judgment results, lack interpretability, and are in a "black box" state; detection methods for specific tasks or hallucination types have limited generalization capabilities and are difficult to transfer stably across tasks.

[0004] These existing technologies generally suffer from significant drawbacks: most methods lack interpretability and fail to provide clear evidence, limiting their application in high-reliability scenarios; they lack cross-task generalization ability and exhibit unstable performance; high-quality training data is scarce, manual annotation is costly and subjective, and synthetic data quality control is lacking; training objectives are disconnected from actual needs, and natural language interpretations are often lengthy and useless.

[0005] Therefore, existing technologies are unable to simultaneously meet the comprehensive requirements of low cost, high detection accuracy, strong cross-task generalization, and high availability and interpretability. There is an urgent need for a fidelity illusion detection solution that takes into account efficiency, accuracy, and transparency. Summary of the Invention

[0006] In view of this, the present invention proposes a method and system for detecting and interpreting model fidelity illusions, which can automatically generate high-quality, verifiable and interpretable fidelity illusions while ensuring low cost and high detection accuracy.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for detecting and determining the interpretability of a model-based fidelity illusion includes: Based on a large language model, candidate samples are automatically generated, including fidelity judgment results, reasoning process, and natural language explanation, using the input document and the claims of the text to be detected as conditions. The candidate samples are subjected to multi-dimensional filtering to obtain high-quality interpretable detection data; The initial detection model was supervised and fine-tuned using the high-quality interpretable detection data to obtain the fine-tuned detection model. A composite reward mechanism is constructed, and a composite reward signal is generated by providing reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism. Based on the composite reward signal, the fine-tuned detection model is iteratively optimized and trained to obtain the final hallucination detection model. The input document and the text to be detected are input into the final hallucination detection model, which outputs the fidelity judgment result and the corresponding natural language explanation.

[0008] Based on the above technical solution, the present invention can be further improved as follows: Optionally, the text to be detected may be derived from any one or more application scenarios such as text summarization, retrieval enhancement generation, question answering, and multi-hop reasoning.

[0009] Optionally, the multi-dimensional filtering process includes at least one or more of label consistency filtering, interpretation validity filtering, and data diversity filtering; The fidelity judgment results of candidate samples are verified by label consistency filtering to check the degree of matching between the results and the real labels. The effectiveness of the interpretation was used to evaluate the supporting role of natural language interpretation in the fidelity determination results. The distribution of different types of candidate samples is balanced by filtering data diversity.

[0010] Optionally, the composite reward mechanism includes at least one or more of the following: judgment correctness reward, interpretation quality reward, and output format reward; The reward for correct judgment is related to the degree of matching between the loyalty judgment result and the real label; The reward for the quality of interpretation is related to the information validity of the natural language interpretation. The output format reward is related to the degree of structure of the model output.

[0011] Optionally, the explanation quality reward is achieved through an auxiliary model evaluation. The auxiliary model takes the input document, the claim to be detected text, and the natural language explanation generated by the fine-tuned detection model as input to determine fidelity. If the fidelity determination result of the auxiliary model is correct, a positive explanation quality reward is triggered.

[0012] Optionally, the initial detection model is a small-scale deep learning model or a medium-scale deep learning model.

[0013] Optionally, the iterative optimization training employs a reinforcement learning algorithm, which includes a policy optimization algorithm based on group relative advantage.

[0014] A system for detecting and interpreting illusions related to fidelity in large language models, comprising: The sample generation module is used to automatically generate candidate samples based on a large language model, taking the input document and the text to be detected as conditions, including the fidelity judgment result, reasoning process and natural language explanation. The data filtering module is used to perform multi-dimensional filtering on the candidate samples to obtain high-quality interpretable detection data. The fine-tuning module is used to perform supervised fine-tuning of the initial detection model using the high-quality interpretable detection data to obtain a fine-tuned detection model. The reward generation module is used to construct a composite reward mechanism, and to provide reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism to generate a composite reward signal. An iterative optimization module is used to perform iterative optimization training on the fine-tuned detection model based on the composite reward signal to obtain the final hallucination detection model. The result output module is used to input the input document and the text to be detected into the final hallucination detection model, and output the fidelity judgment result and the corresponding natural language explanation.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described herein.

[0016] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.

[0017] The present invention has the following advantages: The present invention provides a method for detecting and interpreting model fidelity illusions. By automatically generating candidate samples and filtering in multiple dimensions, high-quality training data can be obtained without manual annotation, significantly reducing training costs. Through supervised fine-tuning and reinforcement learning iterative optimization, the model achieves both high detection accuracy and strong cross-task generalization. The output of fidelity judgment results and natural language explanations clearly presents the judgment criteria, solving the "black box" problem of existing technologies and improving the credibility and traceability of the results. Attached Figure Description

[0018] For illustrative purposes and not limiting, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1This is a flowchart illustrating the method for detecting and determining the interpretability of model fidelity illusions in an embodiment of the present invention. Figure 2 This is a schematic diagram of the main components of the illusion detection and interpretability determination system for fidelity illusion of large language models in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] It should be noted that, where there is no conflict, the embodiments and features of the present invention can be combined with each other. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating the method for detecting and determining the interpretability of model fidelity illusions according to an embodiment of the present invention. Figure 1 As shown, the hallucination detection and interpretability determination method for model fidelity hallucination provided in this embodiment of the invention includes the following steps S101 to S106.

[0023] S101, based on a large language model, automatically generates candidate samples including fidelity judgment results, reasoning processes, and natural language explanations, using the input document and the claims of the text to be detected as conditions.

[0024] The text to be detected claims to originate from any one or more application scenarios among text summarization, retrieval enhancement generation, question answering, and multi-hop reasoning.

[0025] Based on a large language model with strong reasoning capabilities, the system takes a pre-set input document and the text claims to be tested generated by the large language model as input. The large language model automatically performs fidelity analysis and generates an integrated candidate sample that includes "fidelity judgment result (illusion exists / illusion does not exist / unverifiable), corresponding reasoning logic process, and natural language explanation supporting the judgment". No manual annotation of judgment results, reasoning process and explanation content is required throughout the process.

[0026] S102 performs multi-dimensional filtering on candidate samples to obtain high-quality interpretable detection data.

[0027] The multi-dimensional filtering process includes at least one or more of the following: label consistency filtering, interpretation validity filtering, and data diversity filtering. The fidelity judgment results of candidate samples are verified by label consistency filtering to check the matching degree between the results and the real labels; the judgment labels given by the model in the synthetic samples are compared with the real labels already existing in the original dataset, and only samples that are consistent with each other are retained to ensure the correctness of the training signal. The effectiveness of the explanations is evaluated to assess the supporting role of natural language explanations in the fidelity determination results; the explanation content is assessed to determine whether it can improve the confidence of the detection model under training for the correct label, and low-quality explanations that cannot assist in the determination or are misleading are automatically eliminated to ensure that the retained explanations have practical information value.

[0028] By filtering data diversity to balance the distribution of different types of candidate samples, and clustering samples based on the semantic representation of documents and claims, the distribution of different types of samples is controlled, avoiding excessive concentration of training data in simple or highly similar situations, thereby enhancing the model's cross-task generalization ability.

[0029] S103, using high-quality interpretable detection data to perform supervised fine-tuning of the initial detection model, resulting in a fine-tuned detection model.

[0030] The initial detection model is a small-scale deep learning model or a medium-scale deep learning model.

[0031] A small- or medium-sized deep learning model is selected as the initial detection model. The selected high-quality interpretable detection data is input into the initial detection model for supervised fine-tuning training. This allows the model to learn the mapping relationship of "input document + claim to be detected → fidelity judgment result + natural language interpretation" and output a fine-tuned detection model with basic detection and interpretation capabilities.

[0032] S104. Construct a composite reward mechanism, and provide reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism to generate a composite reward signal.

[0033] The composite reward mechanism includes at least one or more of the following: judgment correctness reward, interpretation quality reward, and output format reward; The reward for correct judgment is related to the matching degree between the fidelity judgment result and the real label; when the fidelity judgment output by the model is consistent with the real label, a positive reward is given to directly enhance the hallucination detection capability.

[0034] The reward for the quality of interpretation is related to the information validity of the natural language interpretation. The explanation quality reward is achieved through an auxiliary model evaluation. This auxiliary model takes the input document, the claim to be detected, and the natural language explanation generated by the fine-tuned detection model as input to determine fidelity. If the auxiliary model's fidelity determination is correct, a positive explanation quality reward is triggered. A weaker auxiliary model is introduced, using the explanation generated by the detection model as one of the input conditions. If this auxiliary model can make a correct determination based on the combined effects of the document, claim, and explanation, the explanation is considered to possess sufficient informativeness and clarity, and a positive reward is given.

[0035] The output format reward is related to the degree of structure of the model output. The structure of the model output is constrained to ensure that the judgment result and the interpretation content conform to the preset format, which facilitates system parsing and actual deployment.

[0036] S105, based on the composite reward signal, iterative optimization training is carried out on the fine-tuned detection model to obtain the final hallucination detection model.

[0037] By employing reinforcement learning algorithms (such as policy optimization algorithms based on group relative advantage), the composite reward signal generated by the composite reward mechanism is used as feedback to iteratively optimize and train the fine-tuned detection model, continuously improving the model's judgment accuracy and interpretability, and finally obtaining a stable hallucination detection model.

[0038] S106: Input the input document and the text to be detected into the final hallucination detection model, and output the fidelity judgment result and the corresponding natural language explanation.

[0039] Input the input document and the text to be detected from the actual application scenario into the final hallucination detection model. The model quickly completes the fidelity analysis and outputs a clear fidelity judgment result (hallucination exists / hallucination does not exist / unverifiable) and the corresponding natural language explanation. The explanation can clearly explain the relationship between the text to be detected and the input document, and has the characteristics of evidence and traceability.

[0040] This invention ultimately yields a high-performance, robust, and well-interpretable faithfulness illusion detection model. This model does not rely on manually labeled data or manual rules, and can provide reliable and transparent faithfulness detection and analysis capabilities for content generated by large language models with low training and inference costs.

[0041] First, this invention enables high-accuracy fidelity illusion detection without the need for manual annotation and review. Through automated data synthesis, quality screening, and a two-stage training mechanism, this invention effectively avoids the problems of high cost, strong subjectivity, and difficulty in scaling manual annotation, significantly reducing training and deployment costs while ensuring detection performance.

[0042] Secondly, this invention overcomes the limitation of existing hallucination detection methods that only output binary judgment results, simultaneously generating explanatory information that is highly consistent with the judgment results, highly readable, and evidence-oriented. This explanation can clearly illustrate the consistency, contradiction, or unverifiable relationship between the generated text and the original document, making the detection results interpretable and traceable, and significantly improving the system's usability in high-reliability application scenarios.

[0043] Furthermore, this invention introduces an explanation effectiveness-oriented reinforcement learning optimization mechanism, enabling the model to not only focus on whether the judgment is correct, but also to explicitly optimize whether the explanation content is actually helpful to the judgment. This avoids the problems of redundant, vague, or irrelevant explanations in existing methods, and enhances the actual value of the explanation information.

[0044] Furthermore, this invention introduces data diversity constraints and a unified cross-task modeling approach during training, enabling the trained detection model to exhibit good generalization ability across different task types and hallucination manifestations. This model can be stably applied to various scenarios such as text summarization, retrieval enhancement generation, question answering, and multi-hop reasoning, reducing the need for repeated training and maintenance of models for different tasks.

[0045] Finally, this invention forms a plug-and-play hallucination detection system that can be deployed as an independent module. It can be directly integrated into existing large language model application processes without retraining or modifying the original generative model. It provides real-time, low-cost fidelity detection and interpretation analysis capabilities for the generated results, thereby significantly improving the overall system's security, reliability, and engineering practical value.

[0046] This invention proposes a technical solution for jointly modeling fidelity hallucination detection and the interpretation of judgment criteria. Unlike existing technologies that only output a binary judgment result of "whether a hallucination exists", this invention systematically extends the hallucination detection task into a joint output form of "judgment result + judgment interpretation". This allows the detection model to automatically generate a natural language interpretation that supports the judgment while making the judgment, fundamentally improving the transparency and credibility of the detection results.

[0047] This invention innovatively proposes a data synthesis and automatic quality screening mechanism that eliminates the need for manual annotation, for constructing high-quality, interpretable hallucination detection training data. Through multi-dimensional automatic screening strategies, including label consistency verification, interpretability evaluation, and sample diversity constraints, this invention effectively filters out noisy samples and low-quality interpretations from synthesized data without relying on manual review, solving the key problem of the difficulty in scalably acquiring high-quality, interpretable training data in existing technologies.

[0048] A rule-based reinforcement learning reward mechanism with "explanatory usability" as its core objective is designed. This invention no longer uses judgment accuracy as the optimization objective alone, but instead introduces an auxiliary model to evaluate whether the explanation can truly improve judgment ability. The information content and clarity of the explanation are directly incorporated into the optimization process of reinforcement learning, thereby training a hallucination detection model that is both accurate in judgment and effective in explanation.

[0049] This invention constructs a low-cost, highly generalizable two-stage automated training framework. By organically combining cold-start supervised fine-tuning with subsequent reinforcement learning optimization, this invention significantly improves the robustness of the model under different tasks, different hallucination modes, and different data distributions while ensuring training efficiency and stability, enabling it to flexibly adapt to various practical application scenarios as a unified hallucination detection module.

[0050] In summary, this invention has achieved substantial innovations in task modeling methods, training data construction methods, and optimized target design, effectively making up for the shortcomings of existing hallucination detection technologies in terms of interpretability, generalization ability, and automation.

[0051] Figure 2 This diagram illustrates the main components of the illusion detection and interpretability determination system for fidelity illusions in large language models, as described in an embodiment of the present invention. Figure 2 As shown, the illusion detection and interpretability determination system 1 for the fidelity illusion of large language models provided in this embodiment of the invention includes a sample generation module 10, a data filtering module 20, a fine-tuning module 30, a reward generation module 40, an iterative optimization module 50, and a result output module 60.

[0052] The sample generation module 10 is used to automatically generate candidate samples, including fidelity judgment results, reasoning process and natural language explanation, based on the large language model, with the input document and the text to be detected as conditions. Data filtering module 20 is used to perform multi-dimensional filtering on the candidate samples to obtain high-quality interpretable detection data; Fine-tuning module 30 is used to perform supervised fine-tuning of the initial detection model using the high-quality interpretable detection data to obtain a fine-tuned detection model; The reward generation module 40 is used to construct a composite reward mechanism, and to provide reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism to generate a composite reward signal. The iterative optimization module 50 is used to perform iterative optimization training on the fine-tuned detection model based on the composite reward signal to obtain the final hallucination detection model. The result output module 60 is used to input the input document and the text to be detected into the final hallucination detection model, and output the fidelity judgment result and the corresponding natural language explanation.

[0053] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 70 includes: a processor 701, a memory 702, and a bus 703; The processor 701 and the memory 702 communicate with each other via the bus 703. The processor 701 is used to call program instructions in the memory 702 to execute the methods provided in the above-described method embodiments, and to execute the methods provided in the embodiments of the present invention.

[0054] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions, which cause a computer to execute the method provided in this embodiment of the invention.

[0055] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0056] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting and determining the interpretability of a model fidelity illusion, characterized in that, include: Based on a large language model, candidate samples are automatically generated, including fidelity judgment results, reasoning process, and natural language explanation, using the input document and the claims of the text to be detected as conditions. The candidate samples are subjected to multi-dimensional filtering to obtain high-quality interpretable detection data; The initial detection model was supervised and fine-tuned using the high-quality interpretable detection data to obtain the fine-tuned detection model. A composite reward mechanism is constructed, and a composite reward signal is generated by providing reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism. Based on the composite reward signal, the fine-tuned detection model is iteratively optimized and trained to obtain the final hallucination detection model. The input document and the text to be detected are input into the final hallucination detection model, which outputs the fidelity judgment result and the corresponding natural language explanation.

2. The method for detecting and interpreting model fidelity illusions according to claim 1, characterized in that, The text to be detected claims to originate from any one or more application scenarios among text summarization, retrieval enhancement generation, question answering, and multi-hop reasoning.

3. The method for detecting and determining the interpretability of model fidelity illusions according to claim 1, characterized in that, The multi-dimensional filtering process includes at least one or more of the following: label consistency filtering, interpretation validity filtering, and data diversity filtering. The fidelity judgment results of candidate samples are verified by label consistency filtering to check the degree of matching between the results and the real labels. The effectiveness of the interpretation was used to evaluate the supporting role of natural language interpretation in the fidelity determination results. The distribution of different types of candidate samples is balanced by filtering data diversity.

4. The method for detecting and determining the interpretability of model fidelity illusions according to claim 1, characterized in that, The composite reward mechanism includes at least one or more of the following: judgment correctness reward, interpretation quality reward, and output format reward; The reward for correct judgment is related to the degree of matching between the loyalty judgment result and the real label; The reward for the quality of interpretation is related to the information validity of the natural language interpretation. The output format reward is related to the degree of structure of the model output.

5. The method for detecting and interpreting model fidelity illusions according to claim 4, characterized in that, The explanation quality reward is achieved through an auxiliary model evaluation. The auxiliary model takes the input document, the claim in the text to be detected, and the natural language explanation generated by the fine-tuned detection model as input to determine fidelity. If the fidelity determination result of the auxiliary model is correct, a positive explanation quality reward is triggered.

6. The method for detecting and interpreting model fidelity illusions according to claim 1, characterized in that, The initial detection model is a small-scale deep learning model or a medium-scale deep learning model.

7. The method for detecting and interpreting model fidelity illusions according to claim 1, characterized in that, The iterative optimization training employs a reinforcement learning algorithm, which includes a policy optimization algorithm based on group relative advantage.

8. A system for detecting and interpreting illusions related to fidelity illusions in large language models, characterized in that, include: The sample generation module is used to automatically generate candidate samples based on a large language model, taking the input document and the text to be detected as conditions, including the fidelity judgment result, reasoning process and natural language explanation. The data filtering module is used to perform multi-dimensional filtering on the candidate samples to obtain high-quality interpretable detection data. The fine-tuning module is used to perform supervised fine-tuning of the initial detection model using the high-quality interpretable detection data to obtain a fine-tuned detection model. The reward generation module is used to construct a composite reward mechanism, and to provide reward feedback to the output of the fine-tuned detection model based on the composite reward mechanism to generate a composite reward signal. An iterative optimization module is used to perform iterative optimization training on the fine-tuned detection model based on the composite reward signal to obtain the final hallucination detection model. The result output module is used to input the input document and the text to be detected into the final hallucination detection model, and output the fidelity judgment result and the corresponding natural language explanation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images