Method for reducing model output illusion based on knowledge retrieval

Abstract

The embodiment of the invention provides a method for reducing model output illusion based on knowledge retrieval. The method comprises the following steps: constructing a knowledge base for storing knowledge entries; receiving a question text input by a user, converting the question text into vector representation, performing matching retrieval on the vector representation in the knowledge base, and retrieving and outputting candidate knowledge entries; obtaining an output answer text of the question text generated by the large model, comparing the candidate knowledge entries with the output answer text, and judging whether the candidate knowledge entries violate the output answer text or not; and performing illusion detection on the output answer text, correcting the output answer text according to an illusion detection result, and outputting an answer corresponding to the question input by the user. According to the method, real-time or near-real-time verification and intervention can be carried out on model output, and illusion is inhibited from the source or the early stage.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (Generative AI) technology, and in particular to a method for reducing model output hallucinations based on knowledge retrieval. Background Art

[0002] With the rapid development of generative AI models and their widespread application across various fields, the quality and reliability of model outputs have become critical issues. Hallucinations are particularly prominent, where models generate seemingly plausible content that is in fact inconsistent with the facts, logically contradictory, or completely fabricated. These hallucinations seriously affect the accuracy and credibility of model responses, and can potentially mislead users, posing potential risks.

[0003] Currently, many existing solutions for suppressing hallucinations primarily rely on post-processing correction, which involves checking and correcting the model after it has completed all output. While this approach can detect and correct errors to a certain extent, it cannot intervene in real time during the model generation process. This is especially true for streaming output, where users may first see parts of the content that contain hallucinations, impacting the user experience, and the cost of correction is relatively high. Therefore, the industry urgently needs a method that can perform real-time or near-real-time verification and intervention on model outputs to suppress hallucinations at the source or in the early stages. Summary of the Invention

[0004] The present invention provides a method for reducing model output hallucinations based on knowledge retrieval to provide a smoother and more reliable user interaction experience.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] A method for reducing model output hallucination based on knowledge retrieval, comprising:

[0007] Build a knowledge base to store knowledge items;

[0008] Receive a question text input by a user, convert the question text into a vector representation, perform a matching search on the vector representation in the knowledge base, and retrieve and output candidate knowledge items;

[0009] Obtaining the output answer text of the question text generated by the large model, comparing the candidate knowledge items with the output answer text, and determining whether there is any contradiction between the candidate knowledge items and the output answer text;

[0010] Perform hallucination detection on the output answer text, modify the output answer text according to the hallucination detection result, and output the answer corresponding to the question input by the user.

[0011] Preferably, the construction of a knowledge base storing knowledge items includes:

[0012] Collect historical user input data and its corresponding big model to generate output data sets, as well as real-time interaction data between online users and the big model. Perform preprocessing operations on the collected data, including text cleaning, deduplication, and formatting.

[0013] For the preprocessed raw data, one or more pre-trained or automated models specially trained for hallucination detection tasks are used to perform preliminary hallucination risk screening on the preprocessed raw data to obtain deep-corrected raw data. The deep-corrected raw data is converted into a vector embedding representation, and the vector embedding representation is imported into a vector database. The vector database is used as a knowledge base, which stores and manages vector data of various knowledge items, wherein the knowledge items include question-answer pairs or fact items.

[0014] Preferably, the large model receives a question text input by a user, converts the question text into a vector representation, performs a matching search on the vector representation in the knowledge base, and retrieves and outputs candidate knowledge items, including:

[0015] Receive the question text input by the user, convert the question text input by the user into a vector representation through the text encoder model, and use the retrieval algorithm provided by the vector database to search the vector database for candidate knowledge items that are most similar to the user input question vector. The candidate knowledge items include question-answer pairs or fact item vectors. The retrieval methods of the vector database include: vector similarity retrieval, keyword matching retrieval and hybrid retrieval.

[0016] For each candidate knowledge item retrieved, calculate the similarity score between it and the data entered by the user. If the similarity score of the candidate knowledge item reaches or exceeds the set threshold, the candidate knowledge item is considered to have been retrieved successfully. The user question and the retrieved candidate knowledge item are recorded.

[0017] If the similarity scores of all retrieved candidate knowledge items are lower than the set threshold, the knowledge base retrieval is considered to have failed.

[0018] Preferably, the step of obtaining the output answer text of the question text generated by the large model, comparing the candidate knowledge items with the output answer text, and determining whether there is any contradiction between the candidate knowledge items and the output answer text includes:

[0019] The key information in the output answer text of the question text generated by the current large model is compared with the candidate knowledge item to determine whether there is any contradiction between the output answer text and the candidate knowledge item. When it is determined that there is no contradiction, step S40 is executed; when it is determined that there is a contradiction, the output answer text of the question text generated by the large model is reused, and the regenerated output answer text is compared with the candidate knowledge item again until there is no contradiction between the candidate knowledge item and the output answer text, and step S40 is executed.

[0020] Preferably, performing hallucination detection on the output answer text, correcting the output answer text according to the hallucination detection result, and outputting the answer corresponding to the question input by the user includes:

[0021] The output answer text is detected for hallucinations based on the result of user input recognition. When it is determined that hallucination detection is required for the output answer text, the output answer text is detected for hallucinations using a hallucination detection model. The detection timing and strategy of the hallucination detection model include:

[0022] For non-streaming output: If a large model generates the complete answer text in a non-streaming manner, hallucination detection will be performed once the entire output text is generated. The system will analyze the complete output text as a whole.

[0023] For streaming output: If a large model generates answers gradually in a streaming manner, a phased triggering strategy is used to trigger hallucination detection when the model output reaches the preset detection threshold;

[0024] Based on character count threshold: whenever the output reaches a certain number of characters, a hallucination detection is triggered;

[0025] Based on the token quantity threshold: a detection is triggered every time the output reaches a certain number of tokens (such as 30 tokens);

[0026] Based on sentence endings: Whenever the output encounters a sentence ending, a hallucination detection is triggered;

[0027] Based on specific keywords or phrases: trigger a hallucination detection when the output contains certain keywords or phrases that may be related to the facts;

[0028] After any hallucination detection opportunity is triggered, the current model output fragment or complete output is cross-checked and judged by comprehensively utilizing the following information: hallucination detection model analysis, fact verification model, logical consistency analysis model, language pattern recognition model and hallucination judgment conclusion transmission.

[0029] Preferably, the step of correcting the output answer text according to the hallucination detection result and outputting the answer corresponding to the question input by the user includes:

[0030] The user input, the knowledge base search results, and the complete model output with hallucinations are provided as input to the hallucination processing model. The hallucination processing model identifies inaccuracies, factual inconsistencies, or logical contradictions in the text and corrects them. The correction methods include: directly editing the text, rewriting part or all of the text based on the knowledge base content, or inserting correct information from the knowledge base into the model output in a natural and fluent manner.

[0031] When a hallucination is detected at a certain checkpoint during the model's answer generation process, the system stops generating the current large model and backtracks to the last checkpoint. The hallucination processing model then takes over the subsequent generation process, using user input, knowledge base retrieval results, and the model's valid output before the hallucination as context to restart the process from the backtracking point.

[0032] After the entire modification is completed, the answer corresponding to the question input by the user is output.

[0033] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the method of the present invention can realize real-time or near real-time verification and intervention of model output, thereby suppressing the generation of hallucinations from the source or early stage.

[0034] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A schematic diagram illustrating a method for reducing model output hallucinations based on knowledge retrieval provided by an embodiment of the present invention;

[0037] Figure 2 A processing flow chart of a method for reducing model output hallucination based on knowledge retrieval provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0039] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0041] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0042] The embodiment of the present invention provides a method for reducing model output hallucination based on knowledge retrieval. Figure 1 The specific processing flow is as shown in Figure 2 As shown, the processing steps include the following:

[0043] Step S10: Construct a knowledge base to store knowledge items.

[0044] Step S20: The large model receives the question text input by the user, converts the question text into a vector representation, performs a matching search on the vector representation in the above-mentioned knowledge base, and retrieves and outputs candidate knowledge items, which include question-answer pairs and / or fact items.

[0045] Step S30: Obtain the output answer text of the above question text generated by the large model, compare the candidate knowledge items with the output answer text, and determine whether there is any contradiction between the candidate knowledge items and the output answer text.

[0046] Step S40: performing hallucination detection on the output answer text, correcting the output answer text according to the hallucination detection result, and outputting the answer corresponding to the question input by the user.

[0047] Specifically, the above step S10 includes:

[0048] The knowledge base is a core component used in embodiments of the present invention to provide factual evidence and suppress model hallucinations. The quality and coverage of the knowledge base directly impact the effectiveness of hallucination detection and mitigation. Building a knowledge base is an ongoing optimization process, aiming to continuously accumulate and update high-quality, hallucination-free question-answer pairs and related factual information.

[0049] The construction of the knowledge base is mainly achieved through the following processing:

[0050] 1: Historical data screening

[0051] This process aims to systematically extract high-quality question-answer pairs from accumulated historical user interaction data, and identify and correct hallucinations in model output, thereby enriching the knowledge base content. The specific implementation steps are described in more detail below:

[0052] Data Collection and Preprocessing: The system automatically collects large amounts of historical user input (Query) and its corresponding large model-generated output (Response). This data can come from question-and-answer pairs between users and models, conversation logs, Q&A platform data, and model testing data. A series of preprocessing operations are performed on the collected raw data, including but not limited to: text cleaning (removing noise and special characters), deduplication (identifying and removing duplicate Q&A pairs), and formatting (unifying text encoding and punctuation). In addition, preliminary quality filtering can be performed, such as filtering out short, meaningless, or obviously low-quality Q&A pairs.

[0053] Online data sampling and monitoring: The system intelligently samples and monitors real-time data from online users interacting with large models. Sampling strategies can be based on a variety of factors, such as user feedback (e.g., users explicitly flagging inaccurate model outputs), low-confidence features of model outputs, questions and answers on specific topics or domains, or random sampling to uncover unknown issues. Furthermore, real-time monitoring mechanisms can be implemented to focus on high-risk users or specific types of questions and answers.

[0054] Online Search Enhancement and Fact Verification: Using pre-processed raw data, an advanced online search module retrieves relevant references from a variety of external sources based on key entities, key phrases, or complete questions in user input or model output. These external sources can include authoritative websites, news reports, academic papers, and other reliable internet resources. Online search can utilize a variety of techniques, such as keyword-based full-text search and semantic similarity search based on vector embeddings. The retrieved references are used to conduct preliminary verification of the factual information in the model output, introducing external perspectives and addressing the limitations of the model's internal knowledge.

[0055] Automated hallucination detection screening: Use one or more pre-trained or specially trained automated models for hallucination detection tasks to perform preliminary hallucination risk screening on historical model outputs. These models can adopt different technical approaches:

[0056] Factual consistency-based models: These models compare model outputs with online references to determine whether statements in the output contradict the references. For example, a Natural Language Inference (NLI) model can be used to determine whether the model outputs imply the references or whether the model outputs contradict the references.

[0057] Linguistic feature-based models: These models analyze the linguistic features of the model output itself, such as overconfidence (using absolute terms), fictitious references (non-existent literature or sources), and logical leaps. These features may indicate the presence of hallucinations. The initial screening results in a preliminary hallucination risk score or label, which is used to identify question-answer pairs most likely to harbor hallucinations.

[0058] This knowledge base takes the form of a vector database. The deeply rectified raw data is converted into a vector embedding representation and imported into the vector database. Vector databases are specifically designed to store and manage high-dimensional vector data, supporting efficient similarity searches. Updating the knowledge base is an ongoing, streamlined process, employing either periodic batch updates or real-time incremental updates to ensure the timeliness and accuracy of the knowledge base content. Furthermore, a knowledge base maintenance mechanism is required to regularly review, deduplicate, and perform quality checks on the knowledge base content.

[0059] These online samples with hallucinations and their corrected versions can be used as negative samples or corrected samples for subsequent model training or fine-tuning, forming an online feedback mechanism for the large model and helping the model itself reduce hallucinations.

[0060] Specifically, the above step S20 includes:

[0061] User input recognition is the starting point of the entire hallucination suppression process. Its core function is to determine whether there is highly relevant known information in the knowledge base based on the specific question entered by the user. This judgment determines whether hallucination detection should be triggered in the model output.

[0062] For the specific question input by the user, the question text is converted into a vector representation, the vector representation is matched and searched in the above knowledge base, and a recall operation is performed to retrieve and output candidate question-answer pairs and / or fact entries. The specific implementation details are as follows:

[0063] User input representation and encoding: When a user enters a question, the question text is first converted into a high-dimensional vector representation, or question vector embedding, through a high-quality text encoder model (e.g., Transformer-based Sentence-BERT, SimCSE, etc.). This vector representation captures the semantic information of the user input.

[0064] Knowledge base retrieval and similarity calculation: Using the retrieval algorithm provided by the vector database, the candidate knowledge items that are most similar to the user input question vector are retrieved from the vector database. The candidate knowledge items include question-answer pairs or fact item vectors. Common retrieval methods include:

[0065] Vector Similarity Search: Calculate the similarity between the user's question vector and the vectors of all candidate knowledge items in the knowledge base (cosine similarity is the most commonly used) in the vector database. The vector database can efficiently perform large-scale vector similarity searches and quickly retrieve the items with the highest similarity scores as candidate results.

[0066] Keyword matching search (optional): This method can be combined with traditional keyword-based search methods as an auxiliary method, using technologies such as inverted indexing to quickly match keywords in user input with keywords in knowledge base entries. This method can make up for the shortcomings of pure vector search in terms of precise matching.

[0067] Hybrid search (optional): Combines the advantages of vector similarity search and keyword matching search, and improves the accuracy and coverage of search results by fusing the search results or scores of the two methods.

[0068] Similarity threshold judgment and retrieval success determination: For each retrieved candidate knowledge item, the similarity score between it and the user-entered data is calculated. A preset similarity threshold (SimilarityThreshold) is set. This threshold is determined based on experience or through experimental tuning. If the similarity score of the retrieved candidate knowledge item reaches or exceeds this threshold, the knowledge base is considered to have been retrieved successfully. This indicates that the knowledge base may contain reference information that is highly relevant to the user's question and can be used for fact-checking.

[0069] Retrieve result records and process branch control:

[0070] Successful retrieval: The system records the user's question and the retrieved candidate knowledge items in detail (including the retrieved question-answer pair / fact item content and its similarity score). The system internally marks this interaction as requiring model output hallucination detection and passes the retrieved candidate knowledge items as important factual reference information to the model output detection module.

[0071] Retrieval failure: If the similarity scores of all retrieved candidate knowledge items fall below a preset threshold, the knowledge base retrieval is considered a failure. This typically means that there is no known, high-quality information directly addressing the question in the current knowledge base. In this case, the system internally marks this interaction as not requiring mandatory hallucination detection based on knowledge base comparison. The user's input question is directly sent to the large model, where the model generates an answer. While the model may still experience hallucinations at this point, it primarily relies on its own generalization capabilities, making detection through knowledge base comparison less effective. Therefore, this stage is skipped to improve response efficiency.

[0072] Specifically, the above step S30 includes: using the candidate knowledge items (including question-answer pairs and / or fact items) retrieved in step S20 as important fact references. The system will carefully compare the key information, entities, factual statements, etc. in the output answer text (or streaming output fragments) generated by the current large model with these retrieved knowledge items. Through this comparison, it is determined whether the content output by the model is consistent with the knowledge base information, and whether there are any contradictions or inconsistencies. For example, factual statements in the model output can be extracted and compared semantically or structurally with the retrieved knowledge base content.

[0073] Based on the comparison results, a determination is made as to whether the candidate knowledge item and the output answer text conflict. If no conflict is found, step S40 is executed. If no conflict is found, the output answer text generated from the question text using the large model is re-generated and compared again with the candidate knowledge item until no conflict is found between the candidate knowledge item and the output answer text, at which point step S40 is executed.

[0074] Specifically, the above step S40 includes: based on the result of user input recognition, detecting whether the output answer text contains hallucinations. When the output of the user input recognition judgment model requires hallucination detection (i.e., the knowledge base retrieval is successful and there is potential factual reference information), the hallucination detection mechanism will be immediately triggered. The specific timing and method of detection will be adapted according to the output form of the large language model, and the results of the knowledge base retrieval and the special hallucination detection model will be used for judgment. The specific implementation is as follows:

[0075] Testing timing and strategy:

[0076] For non-streaming output: If a large model generates the entire answer text in a non-streaming manner, hallucination detection will be performed once and globally after the entire output text is generated. The system will analyze the entire output text as a whole.

[0077] For streaming output: If a large model generates answers gradually using a streaming approach, a phased triggering strategy is used to achieve near-real-time hallucination detection and intervention. A detection can be triggered once the model output reaches a preset detection threshold. The detection threshold can be set based on actual needs, for example:

[0078] Character count threshold-based: A detection is triggered whenever the output reaches a certain number of characters (such as 50 characters).

[0079] Based on the token quantity threshold: a detection is triggered every time the output reaches a certain token quantity (such as 30 tokens).

[0080] Based on sentence endings: A detection is triggered whenever the output encounters a sentence ending (such as a period, question mark, exclamation mark, semicolon, etc.) to ensure that the detection is based on complete sentence fragments.

[0081] Based on specific keywords or phrases: Detection is triggered when the output contains certain keywords or phrases that may be related to the facts. This staged detection can identify hallucinations at an early stage, preventing the model from continuing to generate a large amount of erroneous content and providing an earlier opportunity for subsequent intervention.

[0082] Detection Process and Multi-Source Verification: After any detection opportunity is triggered, the system will use the following information to cross-check and judge the current model output fragment (for streaming output, the output between the last detection point and the current detection point) or the complete output (for non-streaming output):

[0083] Hallucination detection model analysis: Call one or more specially trained hallucination detection models to perform in-depth analysis of the current model output fragments or complete output text. These models can use a variety of techniques to determine hallucinations:

[0084] Fact-checking models: These models focus on determining the authenticity of textual statements and can be combined with knowledge base comparisons or leverage the model's own learned factual knowledge for verification.

[0085] Logical consistency analysis model: Analyzes whether there are logical contradictions or jumps within the output text.

[0086] Language Pattern Recognition Model: Identifies language patterns associated with hallucinations, such as overconfidence, ambiguity, and fabricated references. By integrating knowledge base comparison results with analysis and judgment by the hallucination detection model, the system draws a conclusion on whether the current model output fragment or the entire output is hallucinatory. This conclusion can be a binary judgment (yes / no hallucination) or a hallucination risk score.

[0087] Hallucination judgment conclusion transmission: The hallucination judgment conclusion is transmitted to the answer writing module. If the hallucination is judged to exist, the subsequent hallucination processing mechanism is triggered.

[0088] After hallucinations are detected during model output detection, the model output is corrected or regenerated to eliminate the hallucinations and ensure that the answers ultimately output to the user are accurate and reliable.

[0089] If hallucinations are detected in the model output detection phase, the hallucination handling mechanism will be immediately activated. One or more specialized hallucination handling models or specially fine-tuned large models are involved in this phase. Their core goal is to combine existing information (including user input, knowledge base question retrieval results, and the model's effective output content before the hallucination occurred) to correct or continue the answer, eliminating the hallucination information and generating a high-quality final answer. The specific implementation details are as follows:

[0090] Processing strategy and process control: The hallucination processing strategy will be finely controlled based on the model output form and hallucination detection results:

[0091] For hallucinations in non-streaming outputs: When the model generates a complete answer at one time and hallucinations are detected, the hallucination processing model will analyze and correct the entire output text with hallucinations. The system will provide user input, knowledge base retrieval results, and the complete model output with hallucinations as input to the hallucination processing model. The processing model will identify inaccurate, factually inconsistent, or logically contradictory parts of the text and attempt to correct them. Correction methods may include: directly editing the text (for example, deleting incorrect sentences, replacing incorrect words), rewriting part or all of the text based on the knowledge base content, or inserting the correct information in the knowledge base into the model output in a natural and fluent manner. After the entire modification is completed, the corrected complete answer will be output.

[0092] For hallucinations in streaming output: When a hallucination is discovered at a certain checkpoint during the model's answer generation process, the system immediately stops the generation process of the current large model. The system then coordinates backtracking to the last checkpoint (that is, the valid output position before the hallucination occurred). This means discarding the hallucinated content generated by the model between the last checkpoint and the current checkpoint. The hallucination processing model then takes over the subsequent generation process. It uses user input, knowledge base retrieval results, and the model's valid output content before the hallucination occurred as context to rewrite from the backtracking point. The goal of the rewriting is to generate an answer that is free of hallucinations, coherent with the previous valid output, and accurate. This process may involve the hallucination processing model replanning and generating subsequent content to correct the previous incorrect direction. In order to ensure the fluency of the rewriting, the processing model needs to have good context understanding and generation capabilities.

[0093] Hallucination processing model capabilities: The hallucination processing model can be a standalone model, or a model fine-tuned for the hallucination correction task based on the original large model. The model needs to have the following capabilities:

[0094] Understanding user intent and context: Accurately understand the user's question and previous conversation history (if applicable).

[0095] Utilize external knowledge: Able to effectively use the knowledge base to retrieve factual information in the results as the basis for generation.

[0096] Identifying and localizing hallucinations: Ability to accurately identify the hallucinated parts of the input text (the model’s hallucinated output).

[0097] Produce accurate and coherent text: Ability to produce revisions or continuations that are factually correct, logically coherent, and linguistically fluent.

[0098] Keep the style consistent with the original text (optional): When revising or continuing, try to maintain consistency with the original output style of the model to provide a more natural user experience.

[0099] Outputting the Corrected Answer: The system outputs the corrected or regenerated answer to the user, as corrected by the hallucination processing model. For streaming output, the user interface displays the corrected answer, typically overwriting the previously erroneous output or seamlessly following the erroneous output.

[0100] In summary, this paper proposes a method for suppressing model hallucinations that combines dynamic knowledge retrieval with a multi-node verification mechanism. By performing verification at key nodes where the model generates answers and leveraging external dynamic knowledge (such as authoritative information in a knowledge base) as a reference, hallucinations in model output can be detected and intervened in real time or near real time, ensuring that the final output is consistent with existing knowledge and improving the accuracy and reliability of answers.

[0101] The embodiments of the present invention timely detect and intervene in hallucinations for streaming output, and backtrack and correct them when necessary, thereby avoiding displaying erroneous information to users and providing a smoother and more reliable interactive experience.

[0102] Through historical data screening and manual online review, the system can continuously learn from actual applications, discover new illusion patterns and knowledge blind spots, and thus continuously optimize and expand the knowledge base, forming a positive cycle.

[0103] Combining knowledge base recall, a specialized hallucination detection model, and a hallucination processing mechanism, this solution can promptly detect and correct inaccurate or fictitious information in the model output, significantly improving the authenticity and credibility of the model output.

[0104] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0105] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0106] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0107] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for reducing model output hallucination based on knowledge retrieval, characterized in that: include: Build a knowledge base to store knowledge items; Receive a question text input by a user, convert the question text into a vector representation, perform a matching search on the vector representation in the knowledge base, and retrieve and output candidate knowledge items; Obtaining the output answer text of the question text generated by the large model, comparing the candidate knowledge items with the output answer text, and determining whether there is any contradiction between the candidate knowledge items and the output answer text; Perform hallucination detection on the output answer text, modify the output answer text according to the hallucination detection result, and output the answer corresponding to the question input by the user.

2. The method according to claim 1, characterized in that The construction of a knowledge base for storing knowledge items includes: Collect historical user input data and its corresponding big model to generate output data sets, as well as real-time interaction data between online users and the big model. Perform preprocessing operations on the collected data, including text cleaning, deduplication, and formatting. For the preprocessed raw data, one or more pre-trained or automated models specially trained for hallucination detection tasks are used to perform preliminary hallucination risk screening on the preprocessed raw data to obtain deep-corrected raw data. The deep-corrected raw data is converted into a vector embedding representation, and the vector embedding representation is imported into a vector database. The vector database is used as a knowledge base, which stores and manages vector data of various knowledge items, wherein the knowledge items include question-answer pairs or fact items.

3. The method according to claim 1, characterized in that The large model receives a question text input by a user, converts the question text into a vector representation, performs a matching search on the vector representation in the knowledge base, and retrieves and outputs candidate knowledge items, including: Receive the question text input by the user, convert the question text input by the user into a vector representation through the text encoder model, and use the retrieval algorithm provided by the vector database to search the vector database for candidate knowledge items that are most similar to the user input question vector. The candidate knowledge items include question-answer pairs or fact item vectors. The retrieval methods of the vector database include: vector similarity retrieval, keyword matching retrieval and hybrid retrieval. For each candidate knowledge item retrieved, calculate the similarity score between it and the data entered by the user. If the similarity score of the candidate knowledge item reaches or exceeds the set threshold, the candidate knowledge item is considered to have been retrieved successfully. The user question and the retrieved candidate knowledge item are recorded. If the similarity scores of all retrieved candidate knowledge items are lower than the set threshold, the knowledge base retrieval is considered to have failed.

4. The method according to claim 3, characterized in that The step of obtaining the output answer text of the question text generated by the large model, comparing the candidate knowledge items with the output answer text, and determining whether there is any contradiction between the candidate knowledge items and the output answer text includes: Compare the key information in the output answer text of the question text generated by the current large model with the candidate knowledge items to determine whether there is any contradiction between the output answer text and the candidate knowledge items. When it is determined that there is no contradiction, execute step S40; when it is determined that there is a contradiction, use the hallucination processing model to continue writing the answer and execute step S40.

5. The method according to claim 4, characterized in that The performing hallucination detection on the output answer text, correcting the output answer text according to the hallucination detection result, and outputting the answer corresponding to the question input by the user includes: The output answer text is detected for hallucinations based on the result of user input recognition. When it is determined that hallucination detection is required for the output answer text, the output answer text is detected for hallucinations using a hallucination detection model. The detection timing and strategy of the hallucination detection model include: For non-streaming output: If a large model generates the complete answer text in a non-streaming manner, hallucination detection will be performed once the entire output text is generated. The system will analyze the complete output text as a whole. For streaming output: If a large model generates answers gradually in a streaming manner, a phased triggering strategy is used to trigger hallucination detection when the model output reaches the preset detection threshold; Based on character count threshold: whenever the output reaches a certain number of characters, a hallucination detection is triggered; Based on the token quantity threshold: a detection is triggered every time the output reaches a certain number of tokens (such as 30 tokens); Based on sentence endings: Whenever the output encounters a sentence ending, a hallucination detection is triggered; Based on specific keywords or phrases: trigger a hallucination detection when the output contains certain keywords or phrases that may be related to the facts; After any hallucination detection opportunity is triggered, the current model output fragment or complete output is cross-checked and judged by comprehensively utilizing the following information: hallucination detection model analysis, fact verification model, logical consistency analysis model, language pattern recognition model and hallucination judgment conclusion transmission.

6. The method according to claim 5, characterized in that The step of correcting the output answer text according to the hallucination detection result and outputting the answer corresponding to the question input by the user includes: The user input, the knowledge base search results, and the complete model output with hallucinations are provided as input to the hallucination processing model. The hallucination processing model identifies inaccuracies, factual inconsistencies, or logical contradictions in the text and corrects them. The correction methods include: directly editing the text, rewriting part or all of the text based on the knowledge base content, or inserting correct information from the knowledge base into the model output in a natural and fluent manner. When a hallucination is detected at a certain checkpoint during the model's answer generation process, the system stops generating the current large model and backtracks to the last checkpoint. The hallucination processing model then takes over the subsequent generation process, using user input, knowledge base retrieval results, and the model's valid output before the hallucination as context to restart the process from the backtracking point. After the entire modification is completed, the answer corresponding to the question input by the user is output.

Images