An enterprise knowledge base question-answering version-consistent evidence constraint retrieval enhancement generation method and system

Abstract

This application discloses a method and system for enhancing retrieval and generation based on version consistency evidence constraints for enterprise knowledge base question answering, belonging to the retrieval enhancement generation optimization technology in the application of large-scale artificial intelligence models. The electronic device receives user questions, query time, user identity, and business scope to determine a knowledge base snapshot; it segments enterprise documents into evidence blocks and writes version identifiers, effective start and end times, substitution edges, applicable scope, and permission tags to form a versioned evidence index and document genealogy diagram; it retrieves candidate evidence and recovers new and old evidence with the same anchor point or substitution chain, generating a version-consistent evidence set, an excluded evidence set, and conflicting evidence pairs based on time, scope, substitution, and conflict relationships; it constructs an answer key evidence coverage table, and when the coverage and consistency conditions are met, it inputs the evidence package into the large language model to generate a cited answer, and verifies the validity of the evidence sentence by sentence after generation. This scheme can reduce the risks of citing old version evidence, mixing new and old versions, and answering without evidence.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence large model applications, natural language processing, enterprise knowledge bases, retrieval enhancement generation, large language model question answering, model context invocation, evidence material citation constraints, document version management, evidence verification and answer traceability, and in particular to a method and system for version consistency evidence constraint retrieval enhancement generation for enterprise knowledge base question answering. Background Technology

[0002] Enterprise knowledge bases typically consist of product manuals, customer service knowledge entries, after-sales policies, operation and maintenance procedures, interface documentation, pricing rules, compliance policies, and temporary announcements. These knowledge bases are characterized by frequent updates, multiple versions, fragmented application scope, and many rules being valid only for specific products, regions, customer types, or business systems.

[0003] In customer service, internal operations, and compliance Q&A sessions, users often ask questions that are not general knowledge inquiries, but rather requests for actionable current rules from the system. For example, they might ask whether a certain interface still supports older parameters in the current version, whether a certain customer type can still renew with the old plan, whether a certain region is covered by a temporary announcement, or whether a certain approval process has switched to the new process on a specified date. If the system mixes old manuals, new announcements, and clauses with different scopes of application, even if the answers appear to be cited, they may not be feasible in business operations.

[0004] Retrieval-enhanced generation techniques typically first retrieve relevant fragments from a knowledge base, then use a large language model to generate the answer. This approach reduces the risk of the model relying solely on parameter memory for answers and facilitates providing citation sources. However, the core challenge of enterprise knowledge bases lies not only in retrieving relevant fragments but also in determining whether the retrieved fragments can be used together in the same answer. An old clause, a new announcement, and a regional exception rule may all be semantically related to the question, but their effective dates, substitution relationships, and applicable scopes differ, and they cannot be simply concatenated and fed into the model for generation.

[0005] In large-scale AI model applications, enterprise knowledge bases typically serve as external knowledge sources before the large language model generates answers. If the model context invocation phase does not distinguish whether evidence blocks are currently valid, or if the generation phase does not restrict the materials that can be cited, the model may organize expired, inapplicable, or conflicting materials into the answer, resulting in a semantically coherent answer but lacking currently valid evidence. Therefore, computable constraint rules need to be added to the entire process of the model invoking the knowledge base, constructing the context, generating the answer, and verifying citations.

[0006] Existing enterprise knowledge base systems typically focus on document entry, vector retrieval, keyword recall, answer generation, answer citation, and access control. Some systems also record document versions or update the knowledge base, but version records often remain only at the document management level and do not become constraints in the question-and-answer generation process. Common problems include: search results containing both expired and currently valid clauses; temporary announcements covering only a portion of the official manual, yet the system still treats the entire manual as the current basis; a model citing one evidence block but including values, dates, or operational steps from another evidence block in the answer; and answers containing citation numbers, but the cited evidence is not applicable to the current query time or current business scope.

[0007] In scenarios such as legal, financial, customer service, operations and maintenance, and product support, incorrectly referencing outdated documents can pose significant business risks. The system needs to perform permission, time, scope of application, and substitution relationship checks before data is entered into the model, and then verify the answer sentence by writing it back to the evidence block one by one after the model is generated. When evidence is insufficient, there are version conflicts, or the reference is inapplicable, the system should output a clarification prompt, a "no response for now" prompt, or a record of manual review, rather than continuing to generate a seemingly fluent response for the sake of completeness.

[0008] Therefore, there is a need for a retrieval enhancement generation method and system that integrates enterprise document version identification, effective start and end time, substitution relationship, scope of application, permission tags, answer key coverage, and post-generation verification into the same technical chain, in order to reduce the impact of expired evidence being cited, mixing of old and new versions, and no evidence answers on the enterprise knowledge base question-and-answer results. Summary of the Invention

[0009] The purpose of this application is to provide a version-consistent evidence-constrained retrieval enhancement method and system for enterprise knowledge base question answering, in order to solve the problem that existing enterprise knowledge base question answering schemes easily mix old versions of documents, new versions of documents, and document fragments with different applicable scopes, and it is difficult to determine whether each answer point has current valid evidence to support it.

[0010] This application focuses on the engineering application of large-scale artificial intelligence models in enterprise-specific knowledge base question answering. Based on the retrieval-enhanced generation architecture, it adds constraint control logic to the knowledge base retrieval and filtering, model context organization, answer generation input control, evidence material citation, and post-generation verification stages. This ensures that the large language model is subject to the joint constraints of version, validity period, scope of application, permissions, and evidence coverage conditions when calling internal enterprise data.

[0011] To achieve the above objectives, this application provides a method executed by an electronic device. The electronic device receives a user question, user identity, query time, and business scope, and determines a knowledge base snapshot corresponding to the query time; it segments and indexes documents in the enterprise knowledge base to form a versioned evidence index with version identifiers, valid start and end times, substitution edges, applicable scope, permission tags, and structured fields; it breaks down the user question into answer points and evidence requirements; it retrieves candidate evidence from the evidence blocks after permission filtering and applicable scope filtering, and retrieves currently valid evidence and historically adjacent evidence along the document genealogy graph with the same anchor point or substitution chain; based on query time, applicable scope, validity period, substitution edges, and content conflict relationships, it obtains a set of version-consistent evidence, a set of excluded evidence, and conflicting evidence pairs; it constructs an evidence coverage table of answer points and evidence blocks; when the evidence coverage and version consistency conditions are met, it calls a large language model to generate a cited answer; after generation, it verifies the correspondence between the answer sentence and the cited evidence block, the consistency of values ​​and dates, the validity period of the evidence, the applicable scope, and the permission tags sentence by sentence.

[0012] This application also provides a system, including a document access module, a versioned evidence index module, a document genealogy module, a question splitting module, a candidate evidence retrieval module, a version consistency filtering module, an evidence coverage module, an answer generation module, an answer verification and output module, and an audit record module.

[0013] This application also provides an electronic device, a computer-readable storage medium, and a computer program product. When executed by a processor, the electronic device, computer-readable storage medium, or computer program product implements the aforementioned version consistency evidence constraint retrieval enhancement generation method for enterprise knowledge base question answering.

[0014] Compared with existing technologies, this application has at least the following beneficial effects: 1. By writing document version, effective start and end time, substitution edge, scope of application, and permission tag into the evidence block, the question-answering system can distinguish between currently valid evidence, historical evidence, evidence that has not yet taken effect, substituted evidence, and inapplicable evidence during the retrieval stage; 2. By retrieving historical adjacent evidence at the same anchor point through the document genealogy diagram, it can proactively discover the differences between old and new versions of paragraphs, avoiding the large language model from treating both as citationable materials simultaneously; 3. By using the evidence coverage table between the answer key points and the evidence block, it can determine whether key conclusions, values, dates, conditions, operation steps, interface parameters, and exception rules are supported by valid evidence; 4. By using conflicting evidence pairs and conflict risk quantities, it outputs clarification prompts, prompts to temporarily not answer, or manual review records when evidence is contradictory or can only be supported by mutually exclusive versions, reducing incomplete answers that cannot be implemented; 5. By verifying sentence by sentence after generation, it can discover problems such as invalid evidence cited in the answer sentence, inconsistencies between the answer value and the evidence, inapplicable citation scope, unauthorized citation, or mismatched citation numbers, and trigger regeneration or downgraded output; 6. By saving knowledge base snapshot identifiers, evidence records, verification results, and final output types, it is convenient for enterprise customer service, operations, compliance, or product teams to review Q&A results and maintain document substitution relationships and scope of application in reverse. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system architecture. Version fields, validity periods, alternative edges, scope of application, and permission tags are integrated throughout the retrieval, generation, and validation processes, avoiding the need to store version information only at the document management level.

[0016] Figure 2 This provides a versioned evidence index and document genealogy. The age of the evidence at the same anchor point, the coverage of evidence by announcements, and the query time all jointly determine whether an evidence block is citationable.

[0017] Figure 3 This is a schematic diagram of the candidate evidence retrieval and version consistency screening process. Relevant but invalid, unauthorized, inappropriate, or substituted evidence is not directly included in the generation context.

[0018] Figure 4 This diagram illustrates the control of evidence coverage and answer generation. The evidence coverage table is used to determine whether each necessary answer point is supported by currently valid evidence, rather than simply checking whether the answer contains citations.

[0019] Figure 5 This is a flowchart illustrating the process of line-by-line validation, downgraded output, and audit log generation. Line-by-line validation ensures that each executable conclusion in the answer is written back to the valid evidence block.

[0020] Figure 6This diagram illustrates the deployment of a knowledge base for enterprise customer service, operations, and compliance. Question and answer results and evidence records are fed back to the knowledge base maintenance process to supplement documentation, correct scope of application, and maintain alternative relationships.

Claims

1. A method for enhancing version consistency evidence constraint retrieval generation for enterprise knowledge base question answering, characterized in that, Performed by an electronic device, including the following steps: Step S1: Receive user question, user identity, query time, and business scope, and determine the knowledge base snapshot corresponding to the query time; the business scope includes at least one of the following: applicable products, applicable regions, business system identifier, customer type, document type, and user permissions; Step S2: Parse and segment the documents in the enterprise knowledge base to obtain multiple evidence blocks; record the document identifier, evidence block identifier, version identifier, paragraph or table anchor point, original content, structured fields, effective start and end time, substitution edge, applicable scope, permission tag, content hash, keyword representation and vector representation for each evidence block to form a versioned evidence index, and establish a document genealogy based on the substitution edge; Step S3: Break down the user question into at least one answer point, and generate evidence requirements and necessity markers for each answer point; the evidence requirements include at least one of the following: object, condition, conclusion, value, date, operation steps, interface parameters, approval role, exception rules, or scope of application; Step S4: Perform semantic retrieval, keyword retrieval, and structured field matching on the evidence blocks that meet the user's permissions and business scope to obtain candidate evidence blocks; and based on the document identifier, anchor point, version identifier, and substitution edge of the candidate evidence blocks, retrieve the currently valid evidence blocks and historical adjacent evidence blocks with the same anchor point or the same substitution chain from the document genealogy graph to obtain candidate evidence groups. Step S5: Based on the query time, effective start and end time, applicable scope, substitution edge and content conflict relationship, determine the citation status of each evidence block in the candidate evidence group, and generate version consistent evidence set, excluded evidence set and conflicting evidence pair. Step S6: Construct an evidence coverage table between the answer key points and the version consistency evidence set. The evidence coverage table records the answer key point number, necessity marker, evidence block number, evidence block version identifier, validity period, coverage type, coverage strength and citation status. Based on the evidence coverage table, calculate the evidence coverage amount, version consistency amount and conflict risk amount. Step S7: When the necessary answer points lack valid evidence, the version consistency is lower than the consistency threshold, or the conflict risk is higher than the conflict threshold, output a clarification prompt, a prompt not to answer for the time being, or a manual review record; when the coverage threshold, consistency threshold, and conflict threshold conditions are met, input the version consistency evidence set, citation number, validity period, scope of application, and prohibited citation mark into the large language model to generate candidate answers with citations. Step S8: Perform post-generation validation on the candidate answers, split the candidate answers into answer sentences, and verify the consistency of each answer sentence with the cited evidence block in terms of object, condition, value, date, scope of application, permission tag, and validity period; if the validation passes, output the cited answer, version description, and evidence record; if the validation fails, regenerate based on the validation failure items, downgrade to a summary response, or output a manual review record.

2. The method according to claim 1, characterized in that, In step S2, the evidence block includes at least one of the following: body paragraph evidence block, table row evidence block, interface parameter evidence block, title level evidence block, attachment reference evidence block, and announcement coverage evidence block; wherein, the table row evidence block retains the table title, column name, row number, and cell value; the interface parameter evidence block retains the interface name, parameter name, parameter value, interface version, and compatibility description; and the announcement coverage evidence block retains the covered document, covered chapter, coverage period, and coverage scope.

3. The method according to claim 1, characterized in that, The structured fields include at least one of the following: rule object, applicable conditions, rule conclusion, numerical threshold, date field, operation action, approval role, interface parameter name, allowed status, and exception conditions; the structured fields are obtained by document parsing rules, table column name mapping, named entity recognition, or manually maintained fields.

4. The method according to claim 1, characterized in that, The alternative edge includes at least one of the following: a new version of the same document replacing an old version; a new clause of the same terms replacing an old clause; a document being split and replaced by multiple documents; multiple documents being merged into a new document; a temporary announcement covering part of the official manual during the coverage period; and a new version of the interface replacing the old version of the parameter description.

5. The method according to claim 1, characterized in that, The document genealogy is established as follows: evidence blocks are used as nodes, and substitution edges, edges with the same anchor point, edges with the same table row, edges with the same interface parameters, and edges with manual confirmation are used as edges; when two evidence blocks belong to the same document identifier or have a title similarity higher than the genealogy threshold, and the chapter path, paragraph anchor point, table title, interface parameter name, or content hash satisfy the corresponding relationship, the two evidence blocks are marked as comparable new and old version evidence blocks.

6. The method according to claim 1, characterized in that, In step S4, the evidence block is... Retrieval score relative to user question q Calculate as follows: in, Indicates semantic similarity. Indicates the number of keyword matches. Indicates the number of structured field matches. Indicates evidence block With business scope The applicable scope and matching quantity Indicates evidence block Relative to query time The effectiveness quantity, Indicates evidence block For users The amount of permissions available. to These are the weighting coefficients.

7. The method according to claim 1, characterized in that, In step S5, the electronic device is categorized according to the query time. Calculate evidence blocks Time effective quantity ,like If the evidence block is within the valid start and end time of the evidence block and has not been replaced by a new evidence block valid within the query time, then... Take the first value; if the evidence block has not yet taken effect, has expired, or has been effectively replaced, then... Take the second value; when the user's question points to a historical time point, use that historical time point as... And the valid evidence block corresponding to that historical time point is used as a citation candidate.

8. The method according to claim 1, characterized in that, In step S5, the two candidate evidence blocks are... and Calculate version consistency : in, This indicates the two evidence blocks relative to the query time. Time phase capacity, This indicates that the two evidence blocks are relative to the scope of business. Applicable range and capacity, This represents the phylogenetic capacity of two evidence blocks in the document phylogenetic diagram. Indicates the number of content conflicts. to For weighting coefficients; when When the threshold for consistency is lower than the threshold, two evidence blocks will not enter the same version of the consistent evidence set at the same time.

9. The method according to claim 8, characterized in that, Content conflict volume It is obtained by combining rule comparison and semantic comparison; rule comparison at least compares date, version number, numerical threshold, allowed status, approval role, interface parameters and operation order, and semantic comparison is used to determine whether two evidence blocks give mutually exclusive conclusions for the same rule object and applicable conditions.

10. The method according to claim 1, characterized in that, In step S6, the evidence coverage of candidate answers Calculate as follows: C o v e r = ∑ k ∈ K w k ⋅ m a x e i ∈ E v S u p p o r t k e i ∑ k ∈ K w k in, This represents the set of key points in the answer. Indicate the key points of the answer The weight, Indicates evidence block Key points of the answer The number of supporting evidences; when any answer point marked with necessity does not have a valid supporting evidence block, the candidate answer is marked as an answer lacking evidence.

11. The method according to claim 1, characterized in that, The coverage type includes at least one of direct support, conditional support, exception support, numerical support, point-in-time support, and reverse exclusion; wherein, reverse exclusion means that the evidence block is used to explain that an old rule, old parameter, or old process should not be used as the basis for the answer at the time of query.

12. The method according to claim 1, characterized in that, In step S7, the context of the input large language model is organized into an evidence package, which includes a set of version-consistent evidence, the citation number of each evidence block, the original text content, the validity period, the scope of application, the evidence block version identifier, and the answer key number; historical evidence blocks that have been replaced and are invalid for the query time are not used as citationable evidence input, or are only used as conflict description input with a prohibition on citation.

13. The method according to claim 1, characterized in that, In step S8, the correspondence between the answer sentence and the referenced evidence block is verified as follows: the rule object, applicable conditions, numerical value, date, operation verb, interface parameter, and reference number in the answer sentence are identified; if the answer sentence is not associated with at least one valid evidence block, or the numerical value, date, applicable scope, permission label, and operation action in the answer sentence are inconsistent with the referenced evidence block, then the answer sentence is marked as failing the verification.

14. The method according to claim 1, characterized in that, In step S8, the version description includes the document name on which the answer is based, version identifier, evidence block anchor point, effective start and end time, scope of application, and reasons for excluding historical evidence; the reasons for exclusion include at least one of the following: not yet effective, expired, replaced, scope inapplicable, permissions unavailable, and conflict with currently valid evidence.

15. The method according to claim 1, characterized in that, The evidence record includes at least the user question, user identity identifier, query time, knowledge base snapshot identifier, business scope, answer key points, evidence block number entering the version-consistent evidence set, excluded evidence block number, conflicting evidence pair, candidate answer verification result, final output type, large language model version number, and evidence index version number.

16. A version consistency evidence constraint-based enhanced retrieval and generation system for enterprise knowledge base question answering, characterized in that, include: The document access module is used to receive corporate policies, product manuals, operation and maintenance procedures, customer service knowledge items, interface documents and temporary announcements. The versioned evidence index module is used to divide documents into evidence blocks and write document identifier, evidence block identifier, version identifier, anchor point, structured fields, effective start and end time, substitution edge, scope of application, permission tag, content hash, keyword representation and vector representation for each evidence block; the document genealogy module is used to build a document genealogy graph based on substitution edge, edge with the same anchor point and edge with manual confirmation; the question decomposition module is used to decompose user questions into answer points and evidence requirements; The candidate evidence retrieval module is used to perform retrieval in evidence blocks after permission filtering and scope of application filtering, and retrieve currently valid evidence blocks and historical adjacent evidence blocks with the same anchor point or the same substitution chain. The version consistency filtering module generates a set of version-consistent evidence, a set of excluded evidence, and conflicting evidence pairs. The evidence coverage module constructs an evidence coverage table and calculates the evidence coverage, version consistency, and conflict risk. The answer generation module generates citation-based candidate answers by calling a large language model when evidence constraints are met. The answer verification and output module verifies the consistency between candidate answers and citation evidence blocks and outputs citation-based answers, version descriptions, clarification prompts, prompts for not answering temporarily, or records of manual review. The audit log module saves evidence records and verification results.

17. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the computer program, implements the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.

19. A computer program product, characterized in that, It includes a computer program or computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 15.

Images