Large-model fine-grained illusion detection and correction method and product
By constructing entity sets and calculating support scores to locate suspicious entities, and combining this with minimal correction using a large language model, the illusion problem in content generated by the large language model is solved, improving the accuracy and efficiency of detection and correction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MILITARY SCI INFORMATION RES CENT ACAD OF MILITARY SCI OF THE CHINESE PEOPLES LIBERATION ARMY
- Filing Date
- 2025-12-29
- Publication Date
- 2026-05-08
AI Technical Summary
Existing large language models are prone to illusions when generating content, and existing detection methods rely solely on information element matching or require complex internal analysis, resulting in high correction costs and difficulty in accurately locating entities at the entity level.
By constructing entity sets of reference information and test content, support scores are calculated to locate suspicious entities, and a large language model is used for minimal correction to avoid in-depth analysis within the model.
It significantly improves factual consistency and error correction accuracy without changing the original sentence structure and writing logic, while reducing engineering complexity and resource consumption and maintaining text stability.
Smart Images
Figure CN121996997A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hallucination detection and correction in specific domain knowledge question answering, and particularly relates to a method and product for fine-grained hallucination detection and correction of large models. Background Technology
[0002] With the rapid development of Natural Language Processing (NLP), Large Language Models (LLMs) have made significant progress on various tasks. However, unlike traditional task-specific systems, LLMs are trained on large-scale online text, which, while giving them excellent language generation capabilities, inevitably absorbs biases from the data. When faced with ambiguous or misunderstood cues, they may exhibit reasoning biases, or even fabricate details to maintain contextual coherence. This resulting "hallucination" manifests as generating content irrelevant to the given context or contradicting the facts. More critically, most LLMs lack built-in mechanisms to trace their output back to external evidence, which weakens their reliability in practical applications and becomes a major obstacle to achieving secure and reliable deployment.
[0003] Existing technologies for hallucination detection mainly focus on two categories of methods. The first category verifies generated content by retrieving external facts (such as Wikipedia, knowledge graphs, etc.) to determine the presence of hallucinations. However, these methods typically require first identifying information elements (such as entities and concepts) in the generated content, then using these elements to generate questions, using the questions to retrieve external knowledge, and finally using that knowledge to verify the factuality of the generated content. While these methods can detect the degree of hallucination in the generated content to some extent, they rely solely on the matching degree of information elements for existence detection, lacking contextual semantic judgment, making the detection too one-sided. The second category of methods attempts to detect hallucinations by analyzing the internal state or behavior of the model. These methods typically use the model's word prediction probabilities and entropy values during the prediction process to capture abnormal behavior in the generated content. While these methods can reveal the model's hallucination tendency to some extent, they usually require in-depth analysis of the model structure.
[0004] For illusion correction, this refers to locating and correcting factual errors in answers based on external high-confidence information sources to ultimately obtain the correct answer. Existing technologies mainly focus on two types of methods. The first type employs large-model prompting engineering techniques. Relying on the capabilities of large language models in natural language understanding and generation, it typically constructs example-based prompt words through few-sample prompt learning to diagnose and correct erroneous content. However, this type of method generally suffers from high costs of large-model invocation, slow response times, and complex prompt design. The second type employs small-model instruction fine-tuning techniques. Related research transforms content correction into an instruction-following task, fine-tuning a small-scale model to solidify its correction capabilities. Although the above methods have achieved good results in the factual correction task of LLM-generated content, factual errors are often deeply bound to erroneous entities, and their correction should be refined to the entity level, such as errors at the conceptual noun level or errors in entity relationships within a sentence. Summary of the Invention
[0005] To address the shortcomings of existing methods, such as one-sided reliance on information element matching, the need for complex internal analysis, high correction costs, and difficulty in accurate positioning at the entity level, the present invention aims to overcome the above-mentioned defects of the prior art and proposes a large-model fine-grained illusion detection and correction method and product.
[0006] In view of this, the present invention proposes a large-scale fine-grained hallucination detection and correction method, comprising: Step 1: Construct reference information based on the title and description of the references, extract entities from the reference information using a large language model, obtain the encoded representation of each entity, and construct a reference information entity set; Step 2: Extract entities from the content to be tested using a large language model, obtain the encoded representation of each entity, and construct the entity set of the content to be tested; Step 3: For each entity in the entity set of the content to be tested, calculate the support score between it and the entity in the entity set of the reference information. Based on the support score, determine whether the entity in the entity set of the content to be tested has reference support and locate suspicious entities. Step 4: Based on the located suspicious entities, construct hallucination correction prompts, utilize the text understanding capabilities of the large language model to determine whether the suspicious entities involve hallucination content, perform minimal correction on the involved hallucination content, and output the corrected content.
[0007] As an improvement to the above method, step 1 includes: Step 101: Construct reference information based on the titles and descriptions of the references provided by the user; Step 102: Using preset entity extraction prompts, extract entities from the references using a large language model; the large language model is the Qwen3-32B model. Step 103: Remove duplicates from the extracted entities and construct a reference information entity set; Step 104: Encode the entities in the reference information entity set using a pre-trained large language model to obtain the hidden layer representation of the entity text and construct the reference information encoding set; the pre-trained large language model is the Sentence-BERT model.
[0008] As an improvement to the above method, step 2 includes: Step 201: When the content to be tested is in English, calculate the position of the space and establish an index mapping from character to word; when the content to be tested is in Chinese, perform word segmentation using a bidirectional maximum matching algorithm. Step 202: The content to be tested is segmented into sentences according to punctuation, and entity extraction prompts are used to extract entities from each sentence using a large language model; the large language model is the Qwen3-32B model. Step 203: Perform deduplication and validity checks on the extracted entities to construct the entity set of the content to be tested; Step 204: Encode entities in the entity set of the content to be tested using a pre-trained large language model to obtain the hidden layer representation of the entity text and construct the encoding set of the content to be tested; the pre-trained large language model is the Sentence-BERT model.
[0009] As an improvement to the above method, the support score in step 3 includes a weighted average of the implication score and the BM25 score, with both preset to have a weight of 0.5.
[0010] As an improvement to the above method, step 3 further includes: performing a hard match between entities in the entity set of the content to be tested and entities in the entity set of the reference information, or judging whether the support score exceeds a preset threshold, in order to locate suspicious entities.
[0011] As an improvement to the above method, step 4 includes: Step 401: For each sentence in the test content that contains a suspicious entity, combine the original sentence with the suspicious entity to form a hallucination correction prompt word; Based on the hallucination correction prompts, the large language model is used to determine whether the sentence to be tested is hallucinating. If it is, the sentence to be tested is corrected with minimal correction, and the corrected sentence is output. If it is not, it is marked as not requiring correction. The large language model is the Qwen3-32B model. Step 402: Replace the corresponding sentence in the original test content with the corrected sentence to form the final corrected non-hallucination content.
[0012] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0013] Compared with the prior art, the advantages of the present invention are: 1. This invention designs a fine-grained "detection-correction" method based on entity extraction and support / conflict measurement, which solves the problems of existing methods that rely solely on information element matching, ignore contextual semantics, and have excessively large error correction magnitudes. By identifying the content to be detected at the entity level and measuring its support / conflict relationships in conjunction with the context, suspicious entities are first accurately located. Then, following the principle of minimum magnitude, only necessary replacements are made to words / phrases directly related to the entity. This significantly improves factual consistency, increases the error correction hit rate, and maintains text stability without changing the original sentence structure and writing logic, avoiding semantic drift and style distortion caused by large-scale rewriting.
[0014] 2. This invention does not rely on the internal state of the model. It adopts a combination of external evidence-driven and threshold decision-making, which takes into account the replaceability of black-box models and low deployment costs, significantly reducing engineering complexity and large model call overhead. Without needing to access probability distributions or internal representations, it completes the judgment based on external evidence and provides stable suspicious entity location decisions through thresholds, thereby reducing the number of calls and latency, reducing resource consumption, and facilitating rapid integration and maintenance in different models and environments. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the large-scale fine-grained hallucination detection and correction method of the present invention; Figure 2 This is a detailed flowchart of the large-scale fine-grained hallucination detection and correction method of the present invention. Detailed Implementation
[0016] To address the limitations of existing technologies, this application proposes a fine-grained hallucination detection and correction method for a large model. By writing entity extraction prompts, the large model is guided to extract entities from the content to be detected and from references, and support is calculated to locate suspicious entities in the content to be detected. Furthermore, hallucination correction prompts are written to guide the large model to further determine whether suspicious entities involve hallucination representations and to perform minimal correction. This method provides a fine-grained approach to hallucination detection and correction. In hallucination detection, it avoids relying solely on one-sided judgments based on hallucination entities without requiring in-depth analysis of the model's internal representation. In hallucination correction, by locating suspicious entities, it not only avoids excessive model calls but also refines the correction magnitude. This makes the method more versatile and convenient, effectively identifying and correcting hallucination content output by the model.
[0017] To achieve the above objectives, this invention proposes a large-model fine-grained hallucination detection and correction method, including the construction of a reference information entity set, the construction of a test content entity set, support calculation to locate suspicious entities, and fine-grained hallucination correction. The method mainly includes: Step 1) Construct reference information using the titles and descriptions of the references, extract entities from the reference information using a large model and obtain the encoded representation of each entity to construct a reference information entity set; Specifically, it includes: Step 101) Construct reference information based on the titles and descriptions of the references provided by the user; Step 102) Extract entities from the references using a large model. If no entities are extracted, a result marked as unvalidated is returned because there are no reference entities available for comparison. In one embodiment, the large language model is the Qwen3-32B model; Step 103) Deduplicate entities and construct a reference information entity set; Step 104) The entities extracted from the references are input into the pre-trained language model for encoding to obtain the hidden layer representation of the entity text, and a reference information encoding set is constructed; in one embodiment, the pre-trained large language model is the Sentence-BERT model. Step 2) Use a large model to extract entities from the content to be tested, and obtain the encoded representation of each entity to construct the entity set of the content to be tested; Specifically, it includes: Step 201) When the content to be tested is in English, calculate the position of the space and establish an index mapping from character to word. This is for subsequent deduplication and suspicious entity location. When the content to be tested is in Chinese, perform word segmentation using the bidirectional maximum matching algorithm. Step 202) The content to be tested is segmented into sentences according to punctuation, and the entity extraction prompts are extracted using preset entity extraction prompts. Entities are extracted from each sentence using a large language model. In one embodiment, the large language model is the Qwen3-32B model. Step 203) Deduplicate and perform preliminary verification on the entities, verify whether the extracted entities are valid, and construct the entity set of the content to be tested; Step 204) Encode entities in the entity set of the content to be tested using a pre-trained large language model to obtain the hidden layer representation of the entity text and construct the encoding set of the content to be tested. In one embodiment, the pre-trained large language model is the Sentence-BERT model.
[0018] Step 3) Calculate the implication score and BM25 score of each entity in the target content entity set and each entity in the reference information entity set using the support score. Then, determine whether each entity in the generated content entity set has reference support by direct hard matching or by using the scores obtained above, and screen out and locate suspicious entities.
[0019] Specifically, it includes: Step 301) Input each entity in the entity set of the content to be tested and each entity in the entity set of the reference information into the support calculation tool to obtain the implication score and BM25 score, and obtain the degree to which each entity of the generated content is supported by the reference information. Step 302) For each sentence of the content to be tested, generate a detailed result indicating which entities in the sentence are suspicious entities.
[0020] Step 4) Write large-scale hallucination correction prompts, use the large-scale model's text understanding capabilities to determine whether the selected suspicious entities are involved in hallucination content, analyze the scope of correction, and perform the minimum correction.
[0021] Specifically, it includes: Step 401) For each sentence in the content to be tested that contains a suspicious entity, combine the original sentence with the suspicious entity to form a correction prompt word, prompting the large model to make a minimum correction. If the large model determines that the suspicious entity in the sentence is not a hallucination, then no correction is made. Step 402) Replace the corresponding original sentence in the original test content with the corrected sentence to obtain the corrected non-hallucination content.
[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0023] Example 1 like Figure 1 As shown, the fine-grained hallucination detection method of this invention consists of four steps: constructing reference information using the title and description information of references; extracting entities from the reference information using a large model and obtaining the encoded representation of each entity, thus constructing a reference information entity set and a reference information encoding set; extracting entities from the generated content using a large model and constructing a generated content entity set; calculating the implication score and BM25 score of each entity in the generated content entity set and each entity in the reference information entity set using support, determining whether each entity in the generated content entity set is supported by references, and locating suspicious entities; and finally, calling the large model to perform fine-grained hallucination correction.
[0024] For detailed procedures, please refer to [link / reference]. Figure 2 It needs to be explained that... Figure 2 The specific structure of the pre-trained language model in the example is for illustrative purposes only. The main steps include: Step 1) Reference Content Entity Extraction and Encoding: This step extracts all entities present in the reference content and encodes them using a pre-trained language model. Specifically, it includes the following steps: Leveraging the powerful semantic analysis and information extraction capabilities of a large model, more high-quality entity information is extracted from complex references, ensuring that the constructed reference information entity set has sufficient breadth and depth to provide a solid foundation for subsequent support calculations. Through large model entity extraction, entities are extracted from the reference content, duplicate entities are removed, and a reference content entity set is formed.
[0025] The following specific examples will be used to describe in detail the solution of Embodiment 1 of the present invention: This embodiment describes a complete process for detecting hallucinatory information in content generated by a large language model. Assuming the large language model generates new research content about a certain scientific field, the specific process is as follows: Below is a sample output from a large model (such as Qwen3-8B) based on the retrieved references: { References information: [ { "Title":"Deep Learning Model Accuracy in Image Recognition", "Description": "This paper discusses the accuracy of various deeplearning models in image recognition tasks, benchmarking them against industry standards." }, { "Title":"Advancements in Natural Language Processing", "Description": "In this research, we explore recent advancements innatural language processing, including BERT, GPT, and their applications." } ], "Generating text": "The BERT model has shown significant advancements innatural language understanding, achieving state-of-the-art results in multiple NLP benchmarks." } The prompt template for entity extraction is as follows: You are an information extraction expert. You will extract entities from the provided text that fall into the following categories: [Location, Country, Institution, Person, Technology, Model, Indicator, Project, Document, Event, Time (Specific Time), Time (Duration), Quantifier (Ordinal Number), Quantifier (Unit Quantity), World Region (Country), Organization (Government Agency)] ### Output format: If the provided text contains entities, it will be output in this format: [('Entity 1', 'Entity Category'),...] If the provided text contains no entity, then only output: No entity. ###Example User input text: The World Health Organization (WHO) launched the GlobalHealth Initiative in Nairobi, Kenya on June 15, 2024, led by Dr. MariaGonzalez, focusing on applying AI and big data technologies to improve publichealth services across East Africa.. Your output: [{('World Health Organization','Institution'),('WHO','Institution'),('Global HealthInitiative','Project'),('Nairobi','Location'),('Kenya','World Region_Country'),('June 15,2024','Time_Specific Time'),('Dr. Maria Gonzalez','Person'),('AI','Technology'),('bigdata','Technology'),('East Africa','Location')}] Now, extract the relevant entities and their corresponding category labels from the following text: Text: {text} Strictly adhere to the output format; do not include your thought process or any other content! The descriptive information of the two references in the example is placed into {text}, and the extracted entities are obtained by calling the large model (such as Qwen3-32B): [{('Deep Learning Model','Model'),('Accuracy','Indicator'),('ImageRecognition','Technology')}, {('NLP','Technology'),('BERT','Model'),('GPT','Model')}] Next, duplicate entities are extracted, and a reference content entity set is constructed: [{('Deep Learning Model','Model'),('Accuracy','Indicator'),('ImageRecognition','Technology'),('Natural Language Understanding','Technology'),('BERT','Model'),('GPT','Model')}] After entity extraction is complete, the extracted entities are input into a pre-trained language model (such as the BERT model) for encoding to obtain the hidden layer representation of the entities.
[0026] Step 2) Entity extraction and encoding of the content to be detected. This step is used to extract all entities present in the reference content and generate a set of entities to be detected. Specifically, it includes the following steps: extracting entities from multiple entities in the content to be detected, removing duplicate entities, and forming a set of entities in the content to be detected.
[0027] The generated text from the example is placed into the detection prompt word template {text} to obtain the detection prompt words. Calling a large model (such as Qwen3-32B) yields the extracted entities, including: [{('BERT model','model'),('natural language understanding','technology'),('state-of-the-art results','metrics'),('NLP benchmarks','metrics')}] Step 3) Support Calculation for Suspicious Entities: This step uses a support calculation tool to evaluate the matching degree between each entity and the reference information entity set. Specifically, it includes the following steps: For each entity in the content to be detected, check whether it matches an entity in the reference content. This step can use semantic-based search (such as BM25 combined with Sentence-BERT) to achieve soft matching, identifying matching entities even in cases of slight differences (synonyms or slightly different expressions). If an entity in the content to be detected does not find a match in the reference content, then these entities may be considered "illusions," i.e., generating incorrect or inaccurate information in the content.
[0028] For each clause, based on whether there are unsupported entities, output a tagged result. If unsupported entities exist, the output will include details about these entities and their type in the text. This example will produce the following output: { "Content":"The BERT model has shown significant advancements in natural language understanding, achieving state-of-the-art results in multiple NLPbenchmarks.", "Marked": "Hallucination may exist" "Literature": ["Deep Learning Model Accuracy in Image Recognition","Advancements in Natural Language Processing"], "Suspicious Entities": [ { "Entity": "state-of-the-art results", "Starting position": 95, "End position": 119, Type: "Indicator" } ] } In this example, the main entities in the content to be detected, such as “BERT model”, “natural language understanding”, “state-of-the-art results”, and “NLP benchmarks”, were compared with the main entities in the references. The entity “state-of-the-art results” was found to have failed to reach the support threshold and was therefore identified as a suspicious entity.
[0029] Step 4) Fine-grained hallucination correction: This step is used to further determine the relevant contextual facts of suspicious entities, obtain the hallucination detection results for each sentence in the text to be detected, correct the hallucination sentences, and finally output the corrected complete text. Specifically, it includes the following steps: Segmenting the text to be detected into sentences, and then for each sentence containing a suspicious entity, constructing prompt words according to the following template: ##You are a natural language processing expert who can always accurately correct errors in sentences based on reference text.
[0030] #The following is reference text: {ref} #The following is a sentence: {sen} You need to determine if the sentence contains information that contradicts the references, focusing on whether the entities in the sentence are supported by the references: {ents} ## Output Requirements: #If the sentence is not an error, output "No error"; #If the sentence is incorrect, correct it to the minimum and output: "The corrected sentence is: ......" ##Strictly adhere to the output requirements; no other content may be output! In this example, the reference description information is placed in the "{ref}" field, sentences with suspicious entities are placed in the "{sen}" field, and suspicious entities are placed in the "{ents}" field to form clue words. A large model (such as Qwen3-32B) is then called to obtain the hallucination-corrected sentence. The original sentence is then replaced with the sentence that has actually undergone hallucination correction to obtain the corrected result.
[0031] This process provides a method to verify potentially erroneous or inaccurate information, fully performing factual accuracy checks and corrections on the content to be tested.
[0032] Based on the above steps, this invention proposes a fine-grained hallucination detection method based on entity extraction, the method comprising: Step 1) Construct reference information using the titles and descriptions of the references, extract entities from the reference information using a large model and obtain the encoded representation of each entity to construct a reference information entity set; Step 2) Use a large model to extract entities from the content to be tested, and obtain the encoded representation of each entity to construct the entity set of the content to be tested; Step 3) Calculate the implication score and BM25 score of each entity in the target content entity set and each entity in the reference information entity set using the support score. Then, determine whether each entity in the generated content entity set has reference support by direct hard matching or by using the scores obtained above, and screen out and locate suspicious entities.
[0033] Step 4) Write large-scale hallucination correction prompts, use the large-scale model's text understanding capabilities to determine whether the selected suspicious entities are involved in hallucination content, analyze the scope of correction, and perform the minimum correction.
[0034] Example 2 Embodiments of the present invention may also provide a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the various steps in the above method embodiments can be implemented.
[0035] Overview: This invention uses entities as the core processing unit, making judgments based on entity extraction and support, thereby accurately locating suspicious entities without relying on complex introspection. After location, this invention emphasizes minimal correction, making only necessary modifications to content directly related to the suspicious entity, avoiding significant rewriting and style shifts caused by correction. Through these strategies, the overall factual consistency and processing efficiency of generated content are improved, and reliance on cumbersome prompts is reduced, making the correction process more controllable, economical, and stable.
[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A large-scale model fine-grained hallucination detection and correction method, comprising: Step 1: Construct reference information based on the title and description of the references, extract entities from the reference information using a large language model, obtain the encoded representation of each entity, and construct a reference information entity set; Step 2: Extract entities from the content to be tested using a large language model, obtain the encoded representation of each entity, and construct the entity set of the content to be tested; Step 3: For each entity in the entity set of the content to be tested, calculate the support score between it and the entity in the entity set of the reference information. Based on the support score, determine whether the entity in the entity set of the content to be tested has reference support and locate suspicious entities. Step 4: Based on the located suspicious entities, construct hallucination correction prompts, utilize the text understanding capabilities of the large language model to determine whether the suspicious entities involve hallucination content, perform minimal correction on the involved hallucination content, and output the corrected content.
2. The large-scale fine-grained hallucination detection and correction method according to claim 1, characterized in that, Step 1 includes: Step 101: Construct reference information based on the titles and descriptions of the references provided by the user; Step 102: Using preset entity extraction prompts, extract entities from the references using a large language model; the large language model is the Qwen3-32B model. Step 103: Remove duplicates from the extracted entities and construct a reference information entity set; Step 104: Encode the entities in the reference information entity set using a pre-trained large language model to obtain the hidden layer representation of the entity text and construct the reference information encoding set; the pre-trained large language model is the Sentence-BERT model.
3. The method for large-scale fine-grained hallucination detection and correction according to claim 1, characterized in that, Step 2 includes: Step 201: When the content to be tested is in English, calculate the position of the space and establish an index mapping from character to word; when the content to be tested is in Chinese, perform word segmentation using a bidirectional maximum matching algorithm. Step 202: The content to be tested is segmented into sentences according to punctuation, and entity extraction prompts are used to extract entities from each sentence using a large language model; the large language model is the Qwen3-32B model. Step 203: Perform deduplication and validity checks on the extracted entities to construct the entity set of the content to be tested; Step 204: Encode entities in the entity set of the content to be tested using a pre-trained large language model to obtain the hidden layer representation of the entity text and construct the encoding set of the content to be tested; the pre-trained large language model is the Sentence-BERT model.
4. The method for detecting and correcting large-scale fine-grained hallucinations according to claim 1, characterized in that, The support score in step 3 includes a weighted average of the implied score and the BM25 score, with both preset to have a weight of 0.
5.
5. The large-model fine-grained hallucination detection and correction method according to claim 4, characterized in that, Step 3 further includes: performing a hard match between entities in the entity set of the content to be tested and entities in the entity set of the reference information, or judging whether the support score exceeds a preset threshold, in order to locate suspicious entities.
6. The method for detecting and correcting large-scale fine-grained hallucinations according to claim 1, characterized in that, Step 4 includes: Step 401: For each sentence in the test content that contains a suspicious entity, combine the original sentence with the suspicious entity to form a hallucination correction prompt word; Based on the hallucination correction prompts, the large language model is used to determine whether the sentence to be tested is hallucinating. If it is, the sentence to be tested is corrected with minimal correction, and the corrected sentence is output. If it is not, it is marked as not requiring correction. The large language model is the Qwen3-32B model. Step 402: Replace the corresponding sentence in the original test content with the corrected sentence to form the final corrected non-hallucination content.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-6.