Large model enhanced retrieval generation method and device based on real-time fact verification, equipment and medium

Abstract

The embodiment of the invention provides a large model enhanced retrieval generation method and device based on real-time fact verification, equipment and a medium, and relates to the technical field of retrieval generation. The method comprises: in response to a user query, starting a text generation process to generate a reply text stream unit by unit; the generated text content is monitored in real time; when it is monitored that the triggering condition is met, text generation is interrupted, and the to-be-verified content is extracted; obtaining a verification evidence set related to the to-be-verified content; verifying the to-be-verified content based on the verification evidence set to obtain a corresponding verification result; generating a correction instruction based on the verification result; and adjusting the text content generated in the text generation process based on the correction instruction. According to the embodiment of the invention, the method can achieve the real-time interception and correction of a fact error in a text generation process through the construction of a generation-monitoring-interruption-verification-feedback dynamic closed-loop generation process, thereby remarkably improving the accuracy and reliability of the generated content.

Description

Technical Field

[0001] This application relates to the field of retrieval generation technology, and more specifically, to a method, apparatus, device, and medium for generating large-model enhanced retrieval based on real-time fact verification. Background Technology

[0002] Retrieval-enhanced generation techniques aim to improve the factual accuracy of large language model outputs by incorporating external knowledge retrieval before generation. Current mainstream techniques employ a sequential, static "retrieval-generation" process: first, relevant documents are retrieved from a knowledge base based on the query, and then these documents are used as fixed context input to the model to generate the final response.

[0003] However, this traditional technique has significant drawbacks: First, fact verification relies entirely on the initial search results. If the search information is incomplete or outdated, the model will generate "illusory" content based on an incorrect context, lacking the ability to intervene in the generation process in real time. Second, the system has difficulty tracing specific factual statements in the output back to the exact source of evidence in the searched document, resulting in insufficient interpretability. Finally, it cannot detect and correct spontaneous deviations and fabrications that occur in subsequent paragraphs when generating long texts in real time.

[0004] In summary, a solution is needed to improve the accuracy and reliability of retrieved content. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and medium for generating large-model enhanced retrieval based on real-time fact verification, so as to improve the accuracy and reliability of the retrieval generated content.

[0006] In a first aspect, embodiments of this application provide a large model-enhanced retrieval generation method based on real-time fact verification, including: In response to a user query, initiate a text generation process to generate a response text stream unit by unit; During the text generation process, the generated text content is monitored in real time; When the text content is detected to meet the preset triggering conditions, the text generation process is interrupted, and the content to be verified is extracted based on the currently generated text content. Obtain a set of verification evidence related to the content to be verified; The content to be verified is verified based on the set of verification evidence to obtain the corresponding verification result. A correction instruction is generated based on the verification result; The text content generated during the text generation process is adjusted based on the correction instructions.

[0007] In this embodiment of the application, by constructing a dynamic closed-loop generation process of generation-monitoring-interruption-verification-feedback, factual errors can be intercepted and corrected in real time during the text generation process, thereby significantly improving the accuracy and reliability of the generated content.

[0008] In some embodiments, the preset triggering condition includes at least one of the following: Identify new named entities in the generated text content; Based on semantic role tagging, a complete proposition that conforms to a preset semantic structure is identified; The entropy value of the probability distribution output during the text generation process exceeds a preset threshold when generating the current unit.

[0009] In this application embodiment, by setting diversified triggering conditions based on entities, complete propositions or uncertainties, the key fact points that need to be verified can be accurately captured, thereby efficiently starting the verification process while ensuring smoothness.

[0010] In some embodiments, the identification of a complete proposition that has formed a preset semantic structure based on semantic role tags includes: Syntactic analysis and semantic role labeling are performed on the generated text content to identify a semantic framework centered on predicates; When the semantic framework contains the core argument of the predicate, it is determined that a complete proposition conforming to the preset semantic structure has been formed.

[0011] In this embodiment of the application, by judging the completeness of the core arguments in the semantic framework to trigger the verification, it is possible to ensure that complete and verifiable propositions are verified, thereby avoiding invalid verification of incomplete expressions and further improving the efficiency of the overall system.

[0012] In some embodiments, obtaining the set of verification evidence related to the content to be verified includes: Query instructions are constructed based on atomic facts extracted from the current text content; Based on the query command, relevant document fragments are retrieved from an external knowledge base to form the set of verification evidence.

[0013] In this embodiment of the application, by constructing precise queries based on atomic facts and performing secondary searches, evidence fragments that are highly relevant to the facts to be verified can be obtained, thereby providing high-quality and targeted information input for subsequent comparisons and further improving the efficiency and accuracy of fact verification.

[0014] In some embodiments, the step of verifying the content to be verified based on the set of verification evidence to obtain the corresponding verification result includes: The content to be verified is compared with each piece of evidence in the set of verification evidence to detect whether there is a contradiction; If contradictions exist, the credibility of each contradictory piece of evidence shall be assessed based on the authority and timeliness of its source. The verification result corresponding to the content to be verified is determined based on the credibility of each contradictory piece of evidence.

[0015] In this embodiment of the application, the comprehensive verification process that combines contradiction detection and evidence credibility assessment can objectively quantify the degree of conflict between the generated content and external evidence, thereby providing a structured and measurable basis for subsequent decision-making and improving the interpretability of fact verification.

[0016] In some embodiments, comparing the content to be verified with each piece of evidence in the verification evidence set to detect whether there is a contradiction includes: Calculate the semantic vector similarity between the content to be verified and each piece of evidence, and determine that there is a contradiction when the similarity is lower than a preset threshold.

[0017] In this embodiment of the application, contradictions are quantified by calculating semantic vector similarity, which enables efficient and automated comparison between generated content and massive amounts of evidence, thereby quickly and accurately identifying potential factual inconsistencies.

[0018] In some embodiments, the assessment of the credibility of each contradictory piece of evidence based on the authority and timeliness of its source includes: The authority score of the evidence source is determined based on a preset authority mapping dictionary, and the timeliness score of the evidence is determined based on the information release time. The credibility of the contradictory evidence is obtained by weighted summation of the authority score and the timeliness score.

[0019] In this embodiment of the application, by combining the authority of the source and the timeliness of the information to quantify the credibility of evidence, the reliability of contradictory information can be assessed more scientifically, thereby giving priority to the acceptance of highly credible evidence in conflict adjudication and further improving the reliability of fact verification.

[0020] In some embodiments, determining the verification result corresponding to the content to be verified based on the credibility of each of the contradictory pieces of evidence includes: Obtain the initial confidence level of the text generation model when generating the content to be verified; The initial confidence level is calibrated and compared with the credibility of contradictory evidence; The decision is based on the comparison results to determine the tendency to accept external evidence versus the original generated content. The verification result corresponding to the content to be verified is determined based on the aforementioned acceptance tendency.

[0021] In the embodiments of this application, by calibrating and comparing the model's generated confidence level with the credibility of external evidence, a balanced decision can be made between the model's "subjective" judgment and external "objective" facts, thereby improving the rationality and robustness of the final decision.

[0022] In some embodiments, adjusting the text content generated by the text generation process based on the correction instruction includes at least one of the following methods: The correction instruction is injected as a prefix constraint into the subsequent generation context of the text generation model to adjust the content of the subsequently generated text. The text generation process is controlled to revert to the target node corresponding to the content to be verified, and the text generation process is restarted upon prompting the addition of the correction instruction to correct the generated text content.

[0023] In this embodiment, by providing two instruction execution methods—prefix constraint and rollback regeneration—the most suitable correction strategy can be flexibly selected according to the error type, thereby achieving efficient and accurate adjustment of subsequent and generated text content.

[0024] In some embodiments, the step of initiating a text generation process in response to a user query to generate a response text stream unit by unit includes: In response to a user query, an initial search is performed from a pre-defined knowledge base to obtain an initial set of reference documents; Based on the user query and the initial set of reference documents, a text generation process is initiated to generate a response text stream unit by unit.

[0025] In this embodiment of the application, by performing an initial retrieval and providing background documents before performing dynamic verification, a rich knowledge base can be laid for the text generation process, thereby improving the overall relevance and contextual coherence of the generated content.

[0026] Secondly, embodiments of this application provide a large model-enhanced retrieval generation device based on real-time fact verification, comprising: The text generation initiation module is used to respond to user queries and initiate the text generation process to generate a response text stream unit by unit; The real-time content monitoring module is used to monitor the generated text content in real time during the text generation process. The fact verification trigger module is used to interrupt the text generation process when the text content is detected to meet the preset trigger conditions, and extract the content to be verified based on the currently generated text content. The verification evidence collection module is used to acquire a set of verification evidence related to the content to be verified; The evidence comparison and verification module is used to verify the content to be verified based on the verification evidence set and obtain the corresponding verification result. A correction instruction generation module is used to generate correction instructions based on the verification results; The text generation and adjustment module is used to adjust the text content generated during the text generation process based on the correction instructions.

[0027] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any embodiment of the first aspect.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0029] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, wherein when the computer program is executed by a processor, it can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a large model-enhanced retrieval generation method based on real-time fact verification, provided for embodiments of this application; Figure 2 This is a schematic diagram of the existing technology's retrieval process. Figure 3 This is a schematic diagram of the search generation process provided in the embodiments of this application; Figure 4 A schematic diagram of a large model-enhanced retrieval generation device based on real-time fact verification is provided for an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0033] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0034] It should be noted that existing search enhancement generation technologies typically employ a static "search-generation" chain approach, with a typical process as follows: Figure 2 As shown, specifically, when the system receives a user query, it first understands or rewrites it to optimize the retrieval; then, it retrieves several document fragments most relevant to the query from a preset external knowledge base (such as a professional database) through the vector retrieval module; finally, it concatenates the user query and the retrieved documents together as context input to the large language model, which then integrates all the information to generate the final answer (response text).

[0035] However, this traditional retrieval enhancement generation technique has the following drawbacks: 1. Delayed and passive fact verification: Fact verification relies entirely on the initially retrieved documents. The system does not proactively and in real-time verify the content that the model is about to produce during the generation process. If the search results themselves are incomplete, outdated, or contain errors, the large model will generate answers based on flawed context, resulting in the generation of incorrect information.

[0036] 2. Lack of fine-grained evidence support: After the answer is generated, it is usually impossible to trace which specific sentence or piece of evidence in the search document each specific fact statement in the answer (such as a certain data, date, event) comes from, resulting in insufficient credibility and interpretability.

[0037] 3. Inability to address "illusions" during the generation process: Existing static retrieval methods cannot intervene and correct illusions in real time. When a large model generates long content (such as analysis reports or stories), it may deviate from the scope of the retrieved document in subsequent paragraphs, spontaneously generating "illusions" (i.e., fabricated information).

[0038] To address at least one technical problem in the existing technology, this application provides a large-model-enhanced retrieval generation method based on real-time fact verification. The core of this method is to replace the traditional serial static "retrieval-generation" approach with a dynamic closed-loop interactive mode of "generation-verification-regeneration." By embedding an interruptible, decision-making, and feedback-enabled real-time fact verification and decision-making module into the inference loop of a large language model (LLM), this module dynamically interacts with an external knowledge base, thereby effectively improving the accuracy and reliability of the retrieved and generated content.

[0039] like Figure 1 As shown, this application provides a method for generating large-model enhanced retrieval based on real-time fact verification, which may include the following steps: S1. In response to a user query, initiate a text generation process to generate a response text stream unit by unit.

[0040] For example, the system receives query content input by the user through the system interface, such as "Please introduce the driving range of a certain model of electric vehicle".

[0041] First, the system preprocesses and understands the query and initiates an initial retrieval: the user query is converted into a vector, and vector retrieval technology is used to retrieve several of the most relevant initial document fragments from external knowledge bases (such as product databases, authoritative technical document libraries, etc.). These fragments are then used as background knowledge and concatenated with the user query to form an initial prompt context. Subsequently, based on this prompt context, the large language model begins to generate a response text stream word by word in an autoregressive manner.

[0042] Understandably, the generation process is streaming; each time a word is generated, the model state is updated and ready to generate the next word.

[0043] S2. During the text generation process, the generated text content is monitored in real time.

[0044] It should be noted that the generation process does not operate using a traditional "black box" model, but is continuously monitored by the real-time fact verification and decision-making module. In other words, an independent real-time monitoring sub-module (real-time fact verification and decision-making module) is embedded in the reasoning loop of the large language model.

[0045] For example, the real-time monitoring submodule is invoked after each lexical unit is generated, and its input is the entire sequence of text generated up to the current time (i.e., the currently generated text content in the response text stream). The goal of monitoring is to analyze this text that is being generated and determine whether fact verification needs to be triggered. The monitoring behavior and the generation process can be performed synchronously, achieving the goal of true "real-time" monitoring.

[0046] S3. When the text content is detected to meet the preset trigger conditions, the text generation process is interrupted, and the content to be verified is extracted based on the currently generated text content.

[0047] It should be noted that the monitoring submodule can be configured with various trigger condition judgment logics according to requirements. Once any condition is met, the system immediately sends an interrupt signal to the inference engine to suspend the generation of subsequent words and "freeze" the generation process in the current state.

[0048] For example, after the text generation process is interrupted, the system can accurately extract the core statement that triggered the verification from the "frozen" text stream (the currently generated text content) as the content to be verified. For instance, if it is triggered by "the driving range is 560 kilometers", this short sentence can be directly extracted as an atomic fact for verification.

[0049] S4. Obtain a set of verification evidence related to the content to be verified.

[0050] Understandably, the system can initiate a targeted secondary search process. First, the extracted "content to be verified" (such as "range of 560 kilometers") is parsed to construct a precise search query (rather than the original user query), such as "query - electric vehicle model - range - official data".

[0051] Subsequently, based on the constructed precise query statement, a search is performed on a preset external knowledge base (which may be the same as or different from the external knowledge base in step S1, and the two knowledge bases may be in a relationship of inclusion or inclusion) to obtain relevant document fragments (such as text similarity higher than the threshold), forming a set of verification evidence.

[0052] For example, the secondary retrieval process can employ a hybrid retrieval strategy: combining sparse retrieval (BM25) to ensure the recall of relevant documents, and dense retrieval (vector retrieval based on qwen-embedding) to ensure semantic similarity. Specifically, sparse retrieval (such as keyword-matching algorithms) is used to match keywords across the entire knowledge base, ensuring recall; dense retrieval (semantic vector-based retrieval) is used for semantic similarity matching, ensuring retrieval accuracy. Then, the two retrieval results are deduplicated, sorted, and merged to form a set of verification evidence. Each piece of evidence can contain a text fragment and its metadata (such as source, publication time, and authority tags).

[0053] S5. Verify the content to be verified based on the set of verification evidence to obtain the corresponding verification results.

[0054] For example, the verification process can be a process of comparing multiple pieces of evidence and making intelligent decisions. The core of this step is that the system compares and analyzes the set of externally obtained verification evidence with the internally generated content to be verified, and makes a credibility judgment accordingly, and finally outputs a structured verification result.

[0055] Specifically, this step can automatically execute the following logic: Consistency comparison: The system compares the content to be verified with the information in the verification evidence set to analyze whether the two are consistent at the semantic and factual levels.

[0056] Result determination: Based on comparative analysis, the system can determine which of the following situations the content to be verified belongs to: (a) consistent with credible evidence, considered correct; (b) contradictory to credible evidence, considered incorrect; (c) insufficient evidence or conflict, considered questionable.

[0057] Structured output results: Regardless of the judgment, the system can generate a corresponding, machine-readable verification result. This result can include at least one conclusive identifier (such as "Correct", "Incorrect: Recommended correction to [Correct value]", "Questionable"), and optionally associate it with the identifiers of the key evidence on which it is based.

[0058] Understandably, this step completes the crucial transition from obtaining evidence to generating a clear verification conclusion, providing a direct decision-making basis for subsequent instruction generation and content adjustment. Its specific implementation methods (such as comparison algorithms, credibility quantification models, conflict resolution rules, etc.) can have various choices, and this solution is not limited to this example.

[0059] S6. Generate correction instructions based on the verification results.

[0060] Based on the structured verification results from the previous step, the instruction generation module can convert them into natural language correction instructions that the language model can understand and execute. These instructions can be generated using a template-based approach. For example, if the verification result determines that "560 kilometers" is incorrect and the correct value should be "approximately 550 kilometers," the generated instruction might be: "Note: Regarding 'the driving range of this vehicle model,' it should be stated as 'approximately 550 kilometers' according to official data, not '560 kilometers.' Please continue generating or correcting based on this." S7. Adjust the text content generated during the text generation process based on the correction instructions.

[0061] It should be noted that the system can feed back the generated correction instructions to the interrupted text generation process, thereby enabling targeted adjustments to the output text content.

[0062] Specifically, the system can adjust the generated content according to the following logic: Instruction Injection: The system injects correction instructions into the context of the text generation process in a way that the model can understand.

[0063] Process Resumption and Guidance: After the injection command, the system resumes the interrupted text generation process. Upon receiving this command, the subsequent behavior of the generation process will be guided or constrained.

[0064] Content adjustment implementation: Ultimately, this involves the actual adjustment of the text content. Its effects may manifest in one or two ways: first, guiding and influencing subsequent text content; second, correcting or rewriting specific text content that has already been generated.

[0065] Understandably, this step completes the "verification-feedback" closed loop, enabling the verification results based on external evidence to directly and effectively influence and optimize the final text output. The specific implementation methods (such as the specific format of the instruction, the timing and location of the injection, and the specific technical path for guidance or correction) can have various choices, but all fall within the scope of this step.

[0066] like Figure 3 As shown, this application embodiment constructs a dynamic closed-loop generation process of generation-monitoring-interruption-verification-feedback, which can intercept and correct factual errors in real time during the text generation process, thereby significantly improving the accuracy and reliability of the generated content.

[0067] In some embodiments, the preset triggering conditions include at least one of the following: Identify new named entities in the generated text content; Based on semantic role tagging, a complete proposition that conforms to a preset semantic structure is identified; The entropy value of the probability distribution output during the text generation process exceeds a preset threshold when generating the current unit.

[0068] For example, the preset triggering condition can be implemented in one or more of the following ways: 1. Named Entity Recognition Trigger: The system can integrate a lightweight named entity recognition model (such as StanfordNER or a BERT-based sequence labeling model). During real-time monitoring, whenever a new word is generated, the named entity recognition model is run on the current complete text sequence.

[0069] When the named entity recognition model identifies a new entity belonging to a preset category (such as people, places, organizations, time, numerical quantifiers, etc.), and this entity appears for the first time in the current dialogue context, a verification is immediately triggered. For example, if "a new type of processor" appears for the first time in the text and is identified as a "product" entity, a verification of its related description is triggered.

[0070] 2. Semantic integrity trigger: The system can integrate a semantic role labeling (SRL) model, which can analyze the newly generated sentences in real time and identify the predicates and their argument structures.

[0071] For example, the predefined rule for judging the completeness of semantic structure can be: when the identified semantic framework contains the core arguments necessary for the predicate (such as agent ARG0, patient ARG1, etc.), it is determined that a complete and verifiable proposition has been formed, triggering verification. For example, if the system generates the text "This drug inhibits the replication of the virus," SRL identifies the predicate "inhibits," ARG0 "this drug," and ARG1 "the replication of the virus," and all the core arguments are present, meaning a complete proposition has been formed, triggering verification of this fact.

[0072] 3. Model uncertainty triggering: During the generation step of each word in the language model, the system can access the output vocabulary probability distribution in real time. Then, the entropy of its probability distribution is calculated using the information entropy formula. :

[0073] in V For vocabulary size, For the first i The probability of each word.

[0074] At each moment of word-by-word generation by the model t The system will calculate the probability distribution of its output. P t And calculate its entropy value. .

[0075] The system calculates an entropy threshold θ (e.g., set to 2.0 through experimental calibration) based on a preset threshold θ. When the value is greater than θ, the model is determined to be in a state of "high uncertainty," which may produce factual ambiguity or "illusionary" content, indicating that it is uncertain what facts to be stated next. In this case, fact verification is immediately triggered; if A value less than 0 indicates that the model is very certain and the generated content has a low factual risk. In this case, no verification is triggered, and the model continues to generate.

[0076] Based on this, by setting diverse triggering conditions based on entities, complete propositions, or uncertainties, it is possible to accurately capture the key fact points that need to be verified, thereby efficiently starting the verification process while ensuring smoothness.

[0077] In some embodiments, identifying a complete proposition that has formed a preset semantic structure based on semantic role tags includes: Syntactic analysis and semantic role labeling are performed on the generated text content to identify a semantic framework centered on predicates; When the semantic framework contains the core argument of the predicate, it is determined that a complete proposition conforming to the preset semantic structure has been formed.

[0078] For example, the system invokes a pre-trained semantic role labeling tool (such as a deep learning-based SRL parser) to process the current text segment to be analyzed (usually a sentence). The processing flow may include: a) word segmentation and part-of-speech tagging; b) dependency parsing; c) predicate identification; d) identifying the related arguments for each predicate and labeling them with semantic roles [such as ARG0 (prototype agent), ARG1 (prototype patient), ARGM-TMP (time modifier), etc.].

[0079] The system makes judgments based on a predefined core argument rule base (for example, for the predicate "investment," its core arguments typically include investor [ARG0] and investment object [ARG1]). After analysis, the system checks the output semantic framework: if there is at least one predicate, and all the predefined core argument roles corresponding to that predicate have been identified and filled with specific text content, then a "complete proposition conforming to the predefined semantic structure" is determined to have been formed.

[0080] For example, when analyzing the sentence "The company released a new product last year", if the predicate "released" is identified, and ARG0 "company", ARG1 "new product" and ARGM-TMP "last year" are also identified, then the trigger condition is met.

[0081] Based on this, by triggering verification by judging the completeness of core arguments in the semantic framework, we can ensure that complete and verifiable propositions are verified, thereby avoiding invalid verification of incomplete statements and further improving the efficiency of the overall system.

[0082] In some embodiments, obtaining a set of verification evidence related to the content to be verified includes: Query instructions are constructed based on atomic facts extracted from the current text content; Relevant document fragments are retrieved from an external knowledge base based on query commands to form a set of verification evidence.

[0083] Specifically, firstly, the system extracts atomic facts and constructs queries: it performs syntactic parsing on the content to be verified, extracts its core subject-verb-object or subject-verb-complement structure, removes modifying elements, and forms a concise "atomic fact" statement. For example, from "Authoritative reports show that the average energy consumption of this device is only 25 watts," it extracts "The average energy consumption of this device is 25 watts."

[0084] Then, this atomic fact is transformed into a query instruction suitable for retrieval, which may include keyword extraction, synonym expansion, etc., for example, generating the query: "Device-Average Energy Consumption-25 Watts-Specifications".

[0085] Subsequently, the system performs a hybrid search: using the constructed query command, the following two types of searches are executed in parallel: Sparse retrieval: Using algorithms such as BM25, keyword matching retrieval is performed in the full-text index of external knowledge bases, focusing on recall rate.

[0086] Dense retrieval: Query instructions are encoded into vectors using pre-trained text embedding models (such as Sentence-BERT), and approximate nearest neighbor search is performed in the vector index of the knowledge base, emphasizing semantic similarity.

[0087] Finally, the two search results are merged, deduplicated, and sorted by relevance (e.g., combining BM25 score and vector similarity score) to form the final set of verification evidence.

[0088] Based on this, by constructing precise queries based on atomic facts and performing secondary searches, it is possible to obtain evidence fragments that are highly relevant to the facts to be verified, thereby providing high-quality and targeted information input for subsequent comparisons and further improving the efficiency and accuracy of fact verification.

[0089] In some embodiments, the content to be verified is verified based on the set of verification evidence to obtain the corresponding verification result, including: The content to be verified is compared with each piece of evidence in the verification evidence set to detect whether there are any contradictions; If contradictions exist, the credibility of each contradictory piece of evidence shall be assessed based on the authority and timeliness of its source. The verification result corresponding to the content to be verified is determined based on the credibility of each contradictory piece of evidence.

[0090] It should be noted that the system can verify the content to be verified based on the set of verification evidence obtained through secondary retrieval. First, it performs contradiction detection: the system traverses each piece of evidence in the set of verification evidence. E i Compare it with the content to be verified C Each sentence is input into an encoder to obtain a corresponding vector pair representation. V ei and V c Then calculate the cosine similarity between each pair of vectors:

[0091] Based on a set similarity threshold δ (e.g., 0.75), if < δ Then the evidence is judged. E i content to be verified C There is a semantic "contradiction" or significant inconsistency.

[0092] Then, a credibility assessment is performed: For all evidence marked as contradictory, the system activates the credibility assessment module, which reads the metadata attached to each piece of contradictory evidence, mainly including: a) Source Authority: Based on a predefined and configurable authority score mapping table (e.g., official institutions, top academic journals = 1.0, authoritative media, core journals = 0.9, well-known commercial media, authoritative websites in professional fields = 0.7, mainstream news portals = 0.5, personal blogs, forums, social media = 0.3, anonymous sources, unverified user-generated content = 0.1, etc.), obtain the corresponding authority score: .

[0093] b) Timeliness: Extract the timestamp of the published evidence, calculate the time difference Δt between it and the current time, and use an exponential decay function to calculate the timeliness score. ,in λ This is the attenuation coefficient (which can be set based on domain knowledge, such as the news field). λ =2, Basic Science Field λ =0.5), to control the rate of information aging.

[0094] evidence E i Credibility Cred(E i ) The following formula can be used for calculation:

[0095] in w a and w t As preset weights, and w a + w t = 1, the weight can be configured according to the application scenario, for example, in the technology field, the timeliness weight. w t It may be relatively high.

[0096] Subsequently, the system can determine the verification result based on the credibility score. For example, the system integrates the results of contradiction detection and credibility assessment. If there is no contradictory evidence, the verification result is marked as "consistent". If there is contradictory evidence, the verification result may include at least the following information: a) marked as "conflict"; b) the content of the contradictory evidence with the highest credibility; c) optionally, a recommended conclusion based on credibility (such as "recommend adopting evidence X").

[0097] Based on this, a comprehensive verification process that combines contradiction detection with evidence credibility assessment can objectively quantify the degree of conflict between generated content and external evidence, thereby providing a structured and measurable basis for subsequent decision-making and improving the interpretability of fact verification.

[0098] In some embodiments, the content to be verified is compared with each piece of evidence in the verification evidence set to detect whether there is a contradiction, including: Calculate the semantic vector similarity between the content to be verified and each piece of evidence, and determine that there is a contradiction when the similarity is lower than a preset threshold.

[0099] For example, the system can use a sentence encoding model (such as SimCSE or a similar model) trained on a natural language reasoning task to generate semantic vectors. This model maps sentences to a semantic space, where the cosine distance between vectors directly reflects the degree of semantic consistency. After calculation, the system compares the similarity with a preset threshold (e.g., determined by balancing precision and recall on a validation set) to achieve automated binary classification of contradictions.

[0100] Based on this, by calculating semantic vector similarity to quantify contradictions, it is possible to achieve efficient and automated comparison between generated content and massive amounts of evidence, thereby quickly and accurately identifying potential factual inconsistencies.

[0101] In some embodiments, the credibility of each contradictory piece of evidence is assessed based on the authority and timeliness of its source, including: The authority score of the evidence source is determined based on the pre-set authority mapping dictionary, and the timeliness score of the evidence is determined based on the information release time. The credibility of contradictory evidence is obtained by weighted summation of the authority score and the timeliness score.

[0102] For example, the system can maintain a configurable "source-authority score" mapping dictionary. This dictionary uses the characteristics of the data source (such as website domain, publishing institution name, and document type) as keys and preset authority scores (usually between 0 and 1) as values. For example: official institutions, top academic journals = 1.0, authoritative media, core journals = 0.9, well-known commercial media, authoritative websites in professional fields = 0.7, mainstream news portals = 0.5, personal blogs, forums, social media = 0.3, anonymous sources, unverified user-generated content = 0.1, etc.

[0103] When evaluating evidence, the system queries this dictionary to obtain the corresponding authority score based on its source URL or the publisher field in the metadata. Auth Score ).

[0104] Then, the timeliness score is calculated: the publication timestamp of the evidence extracted by the system.t publish , with the current time t current Calculate the time difference Δ t = t current - t publish (Usually in years). For example, the timeliness score can be calculated using an exponential decay model, with the following formula:

[0105] in, λ This is a decay factor that can be adjusted according to the speed of information updates, such as in fields with fast update speeds (e.g., technology news). λ A value of 2.0 is acceptable; for fields with slow update rates (such as basic science), λ can be 0.5. For example, the score range is (0, 1], with newer evidence scores closer to 1.

[0106] For example, the credibility of evidence can be obtained based on a weighted summation; specifically, the evidence... E i Credibility Calculate using the following formula:

[0107] in w a and w t As preset weights, and w a + w t = 1, the weight can be configured according to the application scenario, for example, in the technology field, the timeliness weight. w t It may be relatively high.

[0108] This calculation result is a quantified confidence value, which is used for subsequent conflict resolution adjudication.

[0109] Based on this, by combining the authority of the source with the timeliness of the information to quantify the credibility of evidence, we can more scientifically assess the reliability of contradictory information, thereby giving priority to the adoption of highly credible evidence in conflict adjudication and further improving the reliability of fact verification.

[0110] In some embodiments, determining the verification result corresponding to the content to be verified based on the credibility of each contradictory piece of evidence includes: Obtain the initial confidence level of the text generation model when generating content to be verified; The initial confidence level was calibrated and compared with the credibility of contradictory evidence; The decision is based on the comparison results to determine the tendency to accept external evidence versus the original generated content. The verification result corresponding to the content to be verified is determined based on the acceptance bias.

[0111] For example, embodiments of this application may introduce model confidence for adjudication, and the specific process is as follows: At the moment the verification is triggered, the system not only extracts the content to be verified. C It can also record the generation of language models. C The probability distribution of key parts (such as predicates or values) is calculated, and their entropy is also calculated. H(P) t ） As a measure of uncertainty, and through the formula Convert entropy into the model's initial confidence. ( (This is a scaling factor, such as 1.0). Understandably, the lower the entropy (the more certain the generation), the higher the initial confidence.

[0112] When contradictory evidence exists, the system performs a calibration comparison: for each piece of evidence with a credibility level of... Calculate the "challenge strength" (penalty strength) of contradictory evidence on the model's confidence level: ( β (This is the penalty coefficient). Then, based on this, the calibrated confidence level is obtained: .

[0113] Finally, the system makes decisions based on pre-set thresholds. γ (For example, 0.1) Make a ruling: if evidence exists E i Make < γ If the credibility of the evidence is deemed sufficient to overturn the model's confidence level, then the external evidence is accepted, and the verification result can indicate "Content error, correction recommended". E i The statement continues: "Otherwise, if the model's confidence is sufficient to withstand the challenge of the current contradictory evidence, or if the evidence is not credible enough, the ruling will uphold the original content generated by the model, and the verification result may indicate that "the content is credible, but there are inconsistent voices."

[0114] Based on this, by calibrating and comparing the model's generated confidence with the credibility of external evidence, a balanced decision can be made between the model's "subjective" judgment and external "objective" facts, thereby improving the rationality and robustness of the final decision.

[0115] In some embodiments, adjusting the text content generated during the text generation process based on correction instructions includes at least one of the following methods: The correction instructions are injected as prefix constraints into the subsequent generation context of the text generation model to adjust the content of the subsequently generated text. The text generation process is controlled to revert to the target node corresponding to the content to be verified, and the text generation process is restarted upon prompting for the addition of a correction instruction to correct the generated text content.

[0116] For example, the text content can be adjusted in the following two ways: 1. Prefix Constraint Adjustment: The system appends a correction instruction (e.g., "[Correction: The energy consumption value mentioned above should be 30 watts, not 25 watts.]") as a special prefix to the end of the currently generated text. The system then unblocks the language model, which continues generating subsequent text based on this expanded context containing the correction information. This method seamlessly guides the direction of subsequent content.

[0117] 2. Rollback and Regeneration Adjustment: During the generation process, the system maintains a lightweight stack of state checkpoints. When it determines that the generated content needs correction, the system rolls back from the stack to the last checkpoint before generating the content to be verified (i.e., the target node, which saves the model's hidden state, context window, etc. at that time). Then, the system inserts the correction instruction into the prompt information at the rollback point and uses this new prompt combination to re-execute the generation process from that checkpoint, thereby overwriting the part of the text stream that originally contained errors. This method can directly correct historical output content.

[0118] Based on this, by providing two instruction execution methods—prefix constraint and rollback regeneration—the most suitable correction strategy can be flexibly selected according to the error type, thereby achieving efficient and accurate adjustment of subsequent and generated text content.

[0119] In some embodiments, in response to a user query, a text generation process is initiated to generate a response text stream unit by unit, including: In response to a user query, an initial search is performed from a pre-defined knowledge base to obtain an initial set of reference documents; Based on the user query and the initial set of reference documents, the text generation process is initiated to generate a response text stream unit by unit.

[0120] For example, before initiating the text generation process, the system can perform an initialization and warm-up step: the user query is first sent to an initial retrieval module, which performs an initial document retrieval using the query itself, either through vector retrieval or keyword retrieval. A set of the most relevant documents or paragraphs (e.g., 5) is returned from a pre-defined knowledge base, forming an initial reference document set. After brief cleaning and formatting, these documents, along with the original user query, constitute the initial prompt for this text generation task. The language model then begins generating the response text stream unit by unit based on this initial prompt rich in relevant background knowledge.

[0121] Based on this, by performing an initial retrieval and providing background documents before performing dynamic verification, a rich knowledge base can be laid for the text generation process, thereby improving the overall relevance and contextual coherence of the generated content.

[0122] It should be noted that, compared with the prior art, the embodiments of this application have the following beneficial effects: 1. Significantly improves the factual accuracy of generated text: By embedding "checkpoints" during the generation process, inaccurate content generated by the model based on "illusions" arising from incomplete initial information or its own parameters can be effectively intercepted and corrected, ensuring that the final output is consistent with external reliable knowledge sources. Experiments show that in tasks requiring the generation of long texts with accurate facts, this solution can reduce the factual error rate by more than 40%.

[0123] 2. Enhanced System Interpretability and Credibility: The system can automatically link each key fact point in the final answer to specific evidence documents retrieved during the verification process, forming a traceable chain of evidence. This meets the needs of application scenarios in fields such as medicine, law, and finance where high credibility of the answer source is required.

[0124] 3. Improve adaptability to dynamically changing knowledge sources: Even if the initially retrieved documents are outdated, as long as real-time verification is triggered on the key fact points generated subsequently, the system can obtain the latest information from the knowledge base and make corrections, making the system more robust to knowledge updates.

[0125] Please refer to Figure 4 , Figure 4 The diagram illustrates a block diagram of a large model-enhanced retrieval generation apparatus based on real-time fact-checking, provided in some embodiments of this application. It should be understood that this large model-enhanced retrieval generation apparatus based on real-time fact-checking is similar to the one described above. Figure 1 Corresponding to the method embodiments, it is able to execute the various steps involved in the above method embodiments. The specific functions of the large model-enhanced retrieval generation device based on real-time fact verification can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0126] Figure 4The large model augmented retrieval generation device based on real-time fact verification includes at least one software functional module that can be stored in a memory or embedded in the large model augmented retrieval generation device in the form of software or firmware. The large model augmented retrieval generation device based on real-time fact verification includes: The text generation initiation module 410 is used to initiate the text generation process in response to a user query to generate a response text stream unit by unit; The real-time content monitoring module 420 is used to monitor the generated text content in real time during the text generation process. The fact verification trigger module 430 is used to interrupt the text generation process and extract the content to be verified based on the currently generated text content when the text content is detected to meet the preset trigger conditions. The verification evidence collection module 440 is used to acquire a set of verification evidence related to the content to be verified. The evidence comparison and verification module 450 is used to verify the content to be verified based on the verification evidence set and obtain the corresponding verification result. The correction instruction generation module 460 is used to generate correction instructions based on the verification results; The text generation adjustment module 470 is used to adjust the text content generated during the text generation process based on correction instructions.

[0127] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention. The large model-enhanced retrieval generation device based on real-time fact verification provided by the embodiments of the present invention can realize the large model-enhanced retrieval generation method based on real-time fact verification provided by any one of the method embodiments of the present invention.

[0128] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0129] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement any of the methods included in the above-described large model augmented retrieval generation method based on real-time fact verification.

[0130] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0131] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0132] Some embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, describes the method described in the method embodiments.

[0133] Some embodiments of this application also provide a computer program product that, when run on a computer, causes the computer to perform the methods described in the method embodiments.

[0134] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0135] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0136] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0137] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for generating large-scale model-enhanced retrieval based on real-time fact verification, characterized in that, include: In response to a user query, initiate a text generation process to generate a response text stream unit by unit; During the text generation process, the generated text content is monitored in real time; When the text content is detected to meet the preset triggering conditions, the text generation process is interrupted, and the content to be verified is extracted based on the currently generated text content. Obtain a set of verification evidence related to the content to be verified; The content to be verified is verified based on the set of verification evidence to obtain the corresponding verification result. A correction instruction is generated based on the verification result; The text content generated during the text generation process is adjusted based on the correction instructions.

2. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 1, characterized in that, The preset triggering conditions include at least one of the following: Identify new named entities in the generated text content; Based on semantic role tagging, a complete proposition that conforms to a preset semantic structure is identified; The entropy value of the probability distribution output during the text generation process exceeds a preset threshold when generating the current unit.

3. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 2, characterized in that, The identification of a complete proposition that conforms to a preset semantic structure based on semantic role labeling includes: Syntactic analysis and semantic role labeling are performed on the generated text content to identify a semantic framework centered on predicates; When the semantic framework contains the core argument of the predicate, it is determined that a complete proposition conforming to the preset semantic structure has been formed.

4. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 1, characterized in that, The acquisition of the set of verification evidence related to the content to be verified includes: Query instructions are constructed based on atomic facts extracted from the current text content; Based on the query command, relevant document fragments are retrieved from an external knowledge base to form the set of verification evidence.

5. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 1, characterized in that, The step of verifying the content to be verified based on the set of verification evidence to obtain the corresponding verification result includes: The content to be verified is compared with each piece of evidence in the set of verification evidence to detect whether there is a contradiction; If contradictions exist, the credibility of each contradictory piece of evidence shall be assessed based on the authority and timeliness of its source. The verification result corresponding to the content to be verified is determined based on the credibility of each contradictory piece of evidence.

6. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 5, characterized in that, The step of comparing the content to be verified with each piece of evidence in the verification evidence set to detect whether there is a contradiction includes: Calculate the semantic vector similarity between the content to be verified and each piece of evidence, and determine that there is a contradiction when the similarity is lower than a preset threshold.

7. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 5, characterized in that, The assessment of the credibility of each contradictory piece of evidence based on the authority and timeliness of its source includes: The authority score of the evidence source is determined based on a preset authority mapping dictionary, and the timeliness score of the evidence is determined based on the information release time. The credibility of the contradictory evidence is obtained by weighted summation of the authority score and the timeliness score.

8. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 5, characterized in that, Determining the verification result corresponding to the content to be verified based on the credibility of each contradictory piece of evidence includes: Obtain the initial confidence level of the text generation model when generating the content to be verified; The initial confidence level is calibrated and compared with the credibility of contradictory evidence; The decision is based on the comparison results to determine the tendency to accept external evidence versus the original generated content. The verification result corresponding to the content to be verified is determined based on the aforementioned acceptance tendency.

9. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 1, characterized in that, The adjustment of the text content generated by the text generation process based on the correction instruction includes at least one of the following methods: The correction instruction is injected as a prefix constraint into the subsequent generation context of the text generation model to adjust the content of the subsequently generated text. The text generation process is controlled to revert to the target node corresponding to the content to be verified, and the text generation process is restarted upon prompting the addition of the correction instruction to correct the generated text content.

10. The large model-enhanced retrieval generation method based on real-time fact verification according to claim 1, characterized in that, The step of responding to a user query by initiating a text generation process to generate a response text stream unit by unit includes: In response to a user query, an initial search is performed from a pre-defined knowledge base to obtain an initial set of reference documents; Based on the user query and the initial set of reference documents, a text generation process is initiated to generate a response text stream unit by unit.

11. A large-model enhanced retrieval generation device based on real-time fact verification, characterized in that, include: The text generation initiation module is used to respond to user queries and initiate the text generation process to generate a response text stream unit by unit; The real-time content monitoring module is used to monitor the generated text content in real time during the text generation process. The fact verification trigger module is used to interrupt the text generation process when the text content is detected to meet the preset trigger conditions, and extract the content to be verified based on the currently generated text content. The verification evidence collection module is used to acquire a set of verification evidence related to the content to be verified; The evidence comparison and verification module is used to verify the content to be verified based on the verification evidence set and obtain the corresponding verification result. A correction instruction generation module is used to generate correction instructions based on the verification results; The text generation and adjustment module is used to adjust the text content generated during the text generation process based on the correction instructions.

12. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the large model-enhanced retrieval generation method based on real-time fact verification as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the large model-enhanced retrieval generation method based on real-time fact verification as described in any one of claims 1-10.

Images