A large language model question and answer method and system based on multi-source knowledge retrieval
By using a lightweight, complex classifier and an entity-driven iterative expansion mechanism, combined with a versioned index structure, the problems of insufficient retrieval depth and untimely knowledge base updates in existing technologies are solved. This enables rapid response to simple questions and evidence integrity for complex questions, ensuring the auditability and freshness of the answers.
Patent Information
- Application Number
- CN202611494359.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-23
- Publication Date
- 2026-11-10
AI Technical Summary
Existing RAG systems suffer from problems such as insufficient retrieval depth, incomplete evidence chains, and untimely knowledge base updates when handling user questions of varying complexity. They struggle to simultaneously ensure low-latency responses for simple questions and multi-hop evidence integrity for complex questions.
A lightweight classifier is used to classify user questions, dynamically determining whether to perform single-hop or iterative multi-hop retrieval. Combined with an entity-driven iterative expansion mechanism and a versioned index structure, the retrieval trajectory of each hop is recorded, and only the changed blocks are locally re-embedded when the knowledge base is updated.
It enables rapid response to simple questions and complete evidence for complex questions, reduces wasted computing power and knowledge update delays, and ensures the auditability and freshness of answers.
Smart Images

Figure CN123070341A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model technology, and in particular to a large language model question answering method and system based on multi-source knowledge retrieval. Background Technology
[0002] Retrieval-enhanced generation (RAG) technology significantly improves the accuracy and interpretability of generated results by dynamically retrieving relevant information from external knowledge bases and inputting it into a large language model as context, thus becoming a mainstream solution for knowledge-intensive question-answering tasks. However, existing RAG systems still have significant shortcomings when handling user questions of varying complexity.
[0003] In the prior art, patent application CN121960739A discloses an "Adaptive Question Answering Method and Intelligent Large Language Model Based on a Large Language Model." This method determines the complexity of the query text through a query analysis model. Low-complexity queries are directly input into the large language model to generate answers, while high-complexity queries undergo a retrieval process, inputting both the reference document and the query text into the large language model. While this approach achieves query splitting, its complexity classification is relatively coarse-grained, and it only performs a single retrieval for high-complexity queries, lacking a multi-hop iterative retrieval mechanism. When faced with complex problems requiring bridging multiple entities or hop-by-hop reasoning, the retrieval depth is insufficient, making it difficult to guarantee the integrity of the evidence chain.
[0004] In the prior art, patent application CN121388189A discloses a "retrieval generation method and system based on the fusion of large models and knowledge graphs". This method utilizes a large model to extract triple structures to construct a vectorized knowledge base and knowledge graph, and generates results through GraphRAG retrieval and multi-hop path reasoning. This approach relies heavily on the quality of triple extraction, resulting in a cumbersome pipeline and a gradual accumulation of extraction errors. Furthermore, its multi-hop reasoning is performed on the fixed structure of the knowledge graph, making it difficult to dynamically expand the evidence space around newly discovered entities.
[0005] In the prior art, patent application CN115757715A discloses "a multi-hop intelligent question answering method for complex questions based on knowledge graph representation learning." This method learns vector representations of entities and relationships using a knowledge graph representation learning algorithm, searches for reachable entities of the question's subject entity, establishes reasoning paths for candidate answers, and finally selects the entity with the highest score as the answer. This approach is primarily geared towards structured knowledge graphs, relies on the quality of representation learning, and the construction of reasoning paths is limited by the completeness of the knowledge graph, thus having limited adaptability to multi-hop question answering scenarios in unstructured knowledge bases.
[0006] The aforementioned existing technologies typically employ periodic full reconstruction or deletion-then-insertion methods for knowledge base updates. These methods suffer from several drawbacks, including time-consuming full reconstruction, the coexistence of old and new versions during updates leading to the recall of outdated content, and significant delays in the effective implementation of knowledge changes. Furthermore, the retrieval process lacks a record of hop-by-hop evidence chains, making it difficult to trace answer assertions back to specific knowledge fragments, and there is a lack of clear grounds for resolving conflicts between old and new evidence after knowledge updates.
[0007] In summary, existing technologies struggle to simultaneously address both low-latency responses to simple problems and the integrity of multi-hop evidence for complex problems. Furthermore, the retrieval scheduling and knowledge base update mechanisms lack effective coordination and urgently require improvement. Summary of the Invention
[0008] To overcome the problems mentioned in the background art, this invention proposes a large language model question answering method and system based on multi-source knowledge retrieval.
[0009] The technical solution of this invention is: a question-answering method for a large language model based on multi-source knowledge retrieval, comprising the following steps: S1: Receive user questions, input the user questions into a pre-trained lightweight complexity classifier, and obtain the complexity level and corresponding confidence level. The complexity level includes at least single entity direct answer type, multi-entity association type and reasoning calculation type. S2: Determine the retrieval scheduling strategy based on the complexity level; when the complexity level is single-entity direct answer type, execute single-hop retrieval; when the complexity level is multi-entity association type or requires reasoning calculation type, execute entity-driven iterative multi-hop retrieval: construct a query with the entity set of the current hop and the user question, and retrieve the knowledge fragment set of the current hop; perform named entity recognition on the knowledge fragment set of the current hop to obtain a candidate entity set; after performing entity linking and coreference resolution on the candidate entity set, remove the entities in the visited entity set, and use the remaining entities as the entity set of the next hop, until the preset termination condition is met; S3: In response to knowledge base change events, only perform local re-embedding on the knowledge blocks that have been changed, write the re-embedding results to the incremental index, and update the version number of the corresponding entity in the entity registry; delete class changes are written with tombstone records carrying incremental version numbers. S4: In each hop search, dense vector search and sparse inverted index search are performed on the incremental index and the stock index respectively. The recall results of each index and each method are grouped according to entity identifier and block identifier. The entry with the largest version number is retained in the same group. When there is a tombstone record with a higher version number in the same group, the group is removed. After merging and deduplication, the knowledge fragment set of this hop is obtained. S5: Record the entities, knowledge fragments, and version information of each knowledge fragment hit in each hop to form a retrieval trajectory; S6: Combine the user question, the complexity level, and the retrieval trajectory into the prompt template, input it into the large language model, and generate an answer carrying evidence citations.
[0010] Preferably, step S1 includes the following sub-steps: S11: After encoding the user question, the lightweight complexity classifier outputs the probability distribution of each complexity level through the lightweight classification network, and takes the level with the highest probability as the complexity level. S12: The lightweight complexity classifier is trained by distillation of training samples, which are automatically labeled by a large language model and manually checked and corrected. S13: When the maximum probability in the probability distribution is lower than the preset confidence threshold, the complexity level is conservatively upgraded and subsequent steps are executed, wherein the single-entity direct answer type is upgraded to the multi-entity association type, and the multi-entity association type is upgraded to the reasoning calculation type.
[0011] Preferably, step S1 includes the following sub-steps: S14: Perform entity recognition on the user question, extract the number of entities, and detect the hit status of comparison words, aggregation words, ordinal words and conjunction words; S15: Determine the complexity level based on the number of entities and the hit rate: direct-answer questions containing only a single entity and without comparison and aggregation features are determined as single-entity direct-answer questions; associative questions that require bridging across two or more entities are determined as multi-entity associative questions; questions that require aggregation, comparison, or numerical and time operations are determined as reasoning-computation-required questions. In step S2, the retrieval scheduling strategy further includes: when the complexity level is single-entity direct response type, the retrieval depth is 1 hop and the number of recalled knowledge fragments does not exceed a first preset value; when the complexity level is multi-entity association type or reasoning calculation type, the upper limit of the retrieval depth is a preset maximum depth.
[0012] Preferably, in step S2, each hop of the entity-driven iterative multi-hop retrieval includes the following sub-steps: S21: Construct a query for this jump using the set of entities in the current jump and the user question; S22: Perform a retrieval based on the jump query to obtain a jump knowledge fragment set, and incorporate the jump knowledge fragment set into the evidence set and record it in the retrieval trajectory; S23: Perform named entity recognition on the knowledge fragment set of this jump to obtain a candidate entity set; S24: Using the user question and the knowledge fragment set of this jump as context, perform entity linking and coreference resolution on the candidate entity set and map it to standard entities; S25: Remove the entities from the visited entity set from the linked candidate entity set to obtain a new entity set; S26: Sort the new entity set according to its relevance to the user's question, retain entities whose number does not exceed a preset expansion width, and remove entities whose novelty relative to the collected evidence set is lower than a preset novelty threshold; S27: Add the remaining entities to the visited entity set as the entity set for the next hop.
[0013] Preferably, in step S2, the preset termination condition is determined in sub-step S28, which includes: The system determines whether the newly obtained entity set is empty, whether the search depth has reached the preset maximum depth, and whether the novelty of all entities in the new entity set relative to the collected evidence set is lower than the preset novelty threshold. If any of these determinations are true, the iterative multi-hop search is terminated. The novelty is defined as the difference between 1 and the maximum semantic similarity between the entity and the most similar knowledge fragment in the collected evidence set.
[0014] Preferably, in step S2, when the complexity level is of the reasoning computation type, the following sub-steps are also included: S29: Decompose the user problem into subproblems to obtain aggregation subproblems, comparison subproblems, or numerical and time operation subproblems, and perform structured queries and numerical calculations on the subproblems to obtain the subproblem results; S210: The results of the sub-question are incorporated as synthetic evidence into the retrieval trajectory and used to generate the final answer.
[0015] Preferably, step S5 includes sub-step S51: organizing the retrieval trajectory by jump, with each trajectory entry carrying at least the jump number, the entity set used by the jump, the original text of the hit knowledge fragment, the source identifier, the version number, and the timestamp; Step S6 includes the following sub-steps: S61: Inject constraints into the prompt template such that each factual assertion in the answer references the evidence number of the corresponding knowledge fragment in the retrieval trajectory; S62: When multiple knowledge fragments provide conflicting information about the same fact, the knowledge fragment with the larger version number or the more recent timestamp shall be given priority.
[0016] Preferably, in step S3, the processing of the change event includes the following sub-steps: S31: Receive change events, process the changed documents according to the parsing and splitting strategy consistent with the existing index, and obtain each block; S32: Calculate the content hash of each block and compare it with the existing blocks. Only perform embedding calculations on blocks whose content has changed. S33: Using the combination of entity identifier and version number as an idempotent key, write the re-embedding result into the incremental index and the entity registry; S34: Verify that the version number carried by the change event is not less than the current version number in the entity registry; otherwise, discard the change event and record the audit log. S35: Retry the write failure change event. After the preset number of retries is exceeded, transfer it to the dead letter queue.
[0017] Preferably, step S3 also includes the maintenance of the incremental index, comprising the following sub-steps: S36: Determine whether the merge triggering conditions are met. The merge triggering conditions include any one of the following: the number of incremental index entries exceeds a preset threshold, the proportion of tombstone records exceeds a preset ratio, or the time elapsed since the last merge exceeds a preset duration. S37: When the merge triggering condition is met, the incremental index is merged with the most recent existing index segment during a preset off-peak period to generate a new read-only index segment. During the merging process, tombstone records and old entries replaced by higher versions are physically cleared. S38: After the merging is completed, the index segment list is atomically updated, and subsequent searches are directed to the new read-only index segment; S39: The old index segment that has been replaced is retained for a preset time and then recycled for rollback in case of failure.
[0018] A large language model question-answering system based on multi-source knowledge retrieval includes: The complexity classification module is used to: receive user questions and output complexity level and confidence level, wherein the complexity level includes at least single entity direct answer type, multi-entity association type and reasoning calculation type; The entity recognition and linking module is used to extract question entities from the user questions, and to perform named entity recognition, entity linking and coreference resolution on the recalled knowledge fragments; The versioned knowledge index module is used to perform the following actions: responding to knowledge base change events, partially re-embedding and writing incremental indexes only to the changed blocks, updating the version number of entities in the entity registry, deleting class changes to write tombstone records carrying incremental version numbers; and merging the incremental index with the most recent existing index segment during off-peak hours when the merge trigger condition is met, generating a new read-only index segment and atomically updating the index segment list; the versioned knowledge index module includes existing index segments, incremental index segments, and an index segment list; A multi-way retrieval engine is used to perform: dense vector retrieval and sparse inverted index retrieval on the incremental index and the existing index respectively, and to perform structured query; and to group the recall results according to entity identifier and block identifier, retain the entry with the largest version number in the same group, and remove the group when there is a tombstone record with a higher version number in the same group, and merge and deduplicate to obtain the knowledge fragment set of this jump; An iterative retrieval scheduler is used to execute: determine a retrieval scheduling strategy based on the complexity level; when the complexity level is a single-entity direct response type, execute a single-hop retrieval; when the complexity level is a multi-entity association type or a reasoning calculation type, execute an entity-driven iterative multi-hop retrieval, and feed back new entities appearing in the knowledge fragment set of this hop into the next hop query after processing by the entity recognition and linking module, until a preset termination condition is met. The trajectory assembly and generation module is used to: record the entities, knowledge fragments and version information hit in each hop to form a retrieval trajectory, and combine the user question, the complexity level and the retrieval trajectory into a prompt template, input it into a large language model, and generate an answer with evidence citations.
[0019] The beneficial effects of this invention are: 1. Compared to existing technologies that use fixed retrieval depth or select retrieval depth based on coarse-grained rules, which suffer from drawbacks such as wasting computational resources by performing excessively deep retrievals for simple questions and resulting in incomplete evidence by performing shallow retrievals for complex questions, this invention uses a lightweight complexity classifier to classify user questions into three levels: single-entity direct-response type, multi-entity related type, and reasoning-computation type. Based on this level, it dynamically determines whether to perform a single-hop retrieval or an entity-driven iterative multi-hop retrieval. Simple questions are retrieved only once and the number of recalls is limited, while complex questions expand the evidence space hop by hop until a preset termination condition is met, so that the retrieval overhead matches the difficulty of the question. The response latency of simple questions does not increase significantly due to the introduction of classification, and the completeness of the answer to complex cross-entity questions is improved.
[0020] 2. Compared to existing technologies that rely on the quality of triple extraction or repeatedly call the retrieval machine with the same query during the knowledge graph construction process, making it difficult to truly expand the evidence space, this invention suffers from drawbacks such as heavy pipeline, progressive accumulation of extraction errors, and inability to delve deeply into newly discovered entities in multi-hop retrieval. This invention adopts an entity-driven iterative expansion mechanism. Each hop performs named entity recognition, entity linking, and coreference resolution on the recalled knowledge fragments. After removing visited entities from the identified new entities, the next hop query is built back. The set of visited entities is used to prevent loops, the expansion width is used to control the size of each hop, and the novelty threshold is used to determine information gain. The iteration terminates when there are no new entities, the depth limit is reached, or the information gain is insufficient. This allows multi-hop retrieval to gradually discover and utilize intermediate entities along the reasoning chain, while ensuring that the number of retrievals in the worst case has a clear upper limit and the computational power consumption is predictable.
[0021] 3. Compared to existing technologies that use periodic full reconstruction or a combination of deletion and insertion with soft deletion for knowledge base updates, which suffer from drawbacks such as time-consuming and labor-intensive full reconstruction, the coexistence of old and new versions during the update period leading to the recall of expired entities, and long delays in the effectiveness of changes, this invention adopts a versioned index structure that combines existing read-only index segments with incremental writable index segments. When knowledge changes, only the blocks whose content has changed are partially re-embedded and written to the incremental index. Deletion-type changes are expressed by tombstone records carrying incremental version numbers. During retrieval, the results of dual-index dual-path recall are grouped by entity identifier and block identifier, retaining the entry with the largest version number within each group and removing groups covered by tombstones. Then, during off-peak hours, the incremental index is merged into new read-only segments and the changes are made effective through atomic switching of the index segment list. This significantly shortens the delay from knowledge change to retrieval compared to full reconstruction. Moreover, the entire process does not require rebuilding the full index, retraining the embedding model, or fine-tuning the question-answering model. Expired content will not continue to participate in answer generation.
[0022] 4. Compared to existing technologies that directly append search results to prompts without recording the hop-by-hop evidence chain, which suffers from drawbacks such as the invisibility of multi-hop reasoning processes, the inability to trace answer assertions back to specific evidence, and the lack of a basis for resolving conflicts between old and new evidence after knowledge updates, this invention organizes and records the entity hit in each hop, the original evidence text, the source identifier, the version number, and the timestamp to form a search trajectory. The trajectory and generation constraints are injected into the prompt template, requiring each factual assertion in the answer to reference the corresponding evidence number. When multiple pieces of evidence provide conflicting information on the same fact, the evidence with the larger version number or the newer timestamp is given priority. At the same time, consistency checks are performed on the generated answer, ensuring that each assertion in the answer can be traced back to a specific knowledge fragment. Evidence conflicts have clear and consistent rules for selection, and the auditability and timeliness of the answer are guaranteed.
[0023] 5. Compared to existing technologies where retrieval strategies and index update mechanisms are independent, and multi-hop inference chains struggle to guarantee the use of the latest knowledge, resulting in drawbacks such as retrieval scheduling optimization and knowledge freshness optimization operating independently, and the possibility of outdated intermediate evidence cited in complex questions, this invention embeds the merging and deduplication logic of versioned incremental indexes into the execution path of each hop retrieval. This ensures that each hop in the complexity-adaptive scheduling undergoes version filtering, and retrieval trajectory entries carry version numbers and timestamps for conflict resolution during the generation phase. Furthermore, the update pipeline is monitored using the time difference between a change event entering the pipeline and the first time the changed content can be retrieved as a quantitative indicator of update effectiveness delay. This allows retrieval depth scheduling and index updates to cooperate at the system level, ensuring both the integrity and freshness of multi-hop evidence. These two improvements form a unified question-answering system rather than isolated local optimizations. Attached Figure Description
[0024] Figure 1This is an overall flowchart of the large language model question answering method based on multi-source knowledge retrieval of the present invention; Figure 2 This is an architecture diagram of the large language model question-answering system based on multi-source knowledge retrieval of the present invention. Detailed Implementation
[0025] This invention provides a question-answering method and system based on a large language model using multi-source knowledge retrieval. To make the objectives, technical solutions, and advantages of this invention clearer, the implementation methods of this invention are described in detail below with reference to specific embodiments and step numbers. The numerical values, parameters, datasets, scales, and experimental data appearing in each embodiment are illustrative and intended to help those skilled in the art understand and reproduce this invention, and are not intended to limit the scope of protection of this invention.
[0026] Please see Figure 1 and Figure 2 The overall concept of this invention can be summarized in two points: First, the retrieval depth is used as a scheduling variable determined by the complexity of the problem. The problem is first classified by a lightweight complexity classifier, and then a single-hop retrieval or entity-driven iterative multi-hop retrieval is performed accordingly. This solves the problems of "deep retrieval of simple problems wasting computing power and shallow retrieval of complex problems resulting in incomplete answers" at the same time. Second, the retrieval index of the knowledge base is designed as a versioned structure of "existing read-only segments + incremental writable segments". When the knowledge base changes, only the changed blocks are locally re-embedded and written to the incremental index. During retrieval, dual-path recall is performed and deduplication is merged according to the version number, so that knowledge updates take effect with a minute-level delay. The entire process does not require rebuilding the full index, retraining the embedding model, or fine-tuning the question-answering model. To make the explanation more intuitive, the following uses a smart customer service knowledge base Q&A system of a home appliance manufacturing company as a general scenario: The company's knowledge base contains approximately 1.2 million documents, including product manuals, repair manuals, service policies, parts lists, and historical work order information. After being segmented according to a predetermined strategy, these documents are divided into approximately 5 million knowledge blocks. The knowledge base adds or modifies approximately 500 documents daily; the peak concurrent QPS for the Q&A service is approximately 200 QPS; new policies requiring business requirements can be retrieved within 5 minutes of being released. This scenario is merely an example for ease of understanding; the invention is also applicable to scenarios such as government consultation, medical knowledge bases, financial compliance, industrial operations and maintenance, and legal and regulatory retrieval. The scenario itself does not constitute a limitation.
[0027] The system described in this invention comprises six main modules. The complexity classification module executes step S1, outputting the complexity level and confidence level of the user's question; the entity recognition and linking module extracts question entities from the user's question and performs named entity recognition, entity linking, and coreference resolution on the knowledge fragments recalled at each hop, providing standard entities for the iterative expansion in step S2; the versioned knowledge indexing module executes step S3, including existing index segments, incremental index segments, and an index segment manifest, responsible for change capture, local re-embedding, version registration, tombstone writing, and low-peak merging; the multi-way retrieval engine executes step S4, performing dense vector retrieval and sparse inverted index retrieval on the incremental and existing indexes respectively, and supports structured queries, performing grouping, version filtering, and deduplication merging on the recalled results; the iterative retrieval scheduler executes step S2, determining the scheduling strategy based on the complexity level, maintaining the visited entity set, evidence set, and retrieval trajectory, and controlling the iteration until the termination condition is met; the trajectory assembly and generation module executes steps S5 and S6, assembling the question, complexity level, and retrieval trajectory into a prompt template, which is then handed over to the large language model to generate an answer carrying evidence references. An exemplary operating environment is as follows: a server configured with 8 NVIDIA A10 GPUs (24GB VRAM each) deploys the embedded model, complexity classifier, and large question-answering model; the vector library uses a Milvus 2.x cluster (equivalently replaceable with pgvector, Qdrant, etc.); the inverted index search engine uses Elasticsearch (replaceable with Lucene); the entity registry is stored in PostgreSQL (replaceable with relational databases such as MySQL); original documents and index snapshots are stored in object storage; change events enter the processing pipeline via a Kafka message queue. The online link consists of S1 hierarchical structure, S2 scheduling, S4 retrieval for each hop, S5 trajectory recording, and S6 answer generation; S3 is an asynchronous background link that continuously writes knowledge base changes to the incremental index and merges them during off-peak hours. The two links are linked by "each hop retrieval uses dual indexes, and trajectory entries carry version numbers".
[0028] S1 is the complexity grading step. It receives the user's question and uses a lightweight complexity classifier to obtain the complexity level and confidence score. The levels include single-entity direct answer type, multi-entity relational type, and reasoning computation type. S11 shows the specific structure of the classifier: the classifier consists of a sentence encoder, pooling layers, and a lightweight classification head. The sentence encoder uses a lightweight pre-trained model with less than 100 million parameters (such as a 6-layer small BERT-like model or a lightweight text embedding model) to encode the question text into a 768-dimensional sentence vector. The classification head is a two-layer multilayer perceptron that maps the sentence vector into 3-dimensional logistic values. After softmax normalization, the probability distributions of three complexity levels are obtained, in the form of: ; The numbers 0, 1, and 2 are assigned to correspond to single-entity direct-response, multi-entity association, and reasoning-required calculation types, respectively. The level with the highest probability is taken as the classification result. In the actual operating environment of this embodiment, the single-card inference latency of this classifier is measured to be no more than 20 milliseconds, which is the technical basis for "no increase in latency for simple questions". S12 describes the establishment of training samples and training methods for the classifier: the training samples are established according to the cold start process of "automatic labeling of large models + manual sampling and correction + knowledge distillation". A total of 20,000 seed questions are constructed, of which about 8,000 are extracted from SQuAD2.0 released by Stanford University to cover single-entity direct-response types, about 6,000 are bridging and comparison questions from HotpotQA, about 3,000 are combined multi-hop questions from MuSiQue, about 1,000 are cross-document multi-hop questions from 2WikiMultiHopQA to cover multi-entity association types, about 1,000 are DROP numerical reasoning questions from the Allen Institute for Artificial Intelligence to cover reasoning-required calculation types, and 1,000 anonymized questions are extracted from the enterprise customer service historical question logs. The annotation process involves supplementing the domain distribution. Annotations are guided by a template, prompting any general-purpose language model with strong instruction-following capabilities (e.g., GPT, Claude, or Qwen series models) to complete each annotation line. The template specifies three levels of difficulty and provides examples of judgment. For instance, "What is the height of Mount Everest?" is a single-entity direct answer type; "What international awards has the author of *The Three-Body Problem* received?" is a multi-entity association type; and "What is the difference in the repair rate of product A between 2023 and 2024?" is a reasoning calculation type. Subsequently, 20% of the annotation results are randomly selected and independently reviewed by two annotators. A Kappa coefficient of at least 0.8 for consistency between the two annotators is considered passing. Inconsistent samples are arbitrated by a third party. The final set consists of 16,000 training records, 2,000 validation records, and 2,000 test records. During training, cross-entropy is used as the main loss, and knowledge distillation is employed to help the student model fit the probability output of the larger model. The distillation temperature is set to 3, the distillation loss weight is set to 0.7, and the optimizer is AdamW with a learning rate of [missing information]. The batch size is 32, and the training is conducted in 3 rounds. In each round, the evaluation is performed on the validation set, and the optimal weights are retained using an early stopping strategy. After deployment, low-confidence samples are continuously fed back from the online logs, and the training set is expanded through the same process, with regular small-batch retraining to form a self-evolutionary closed loop. S13 describes the confidence handling method: on the validation set, the threshold is scanned with a step size of 0.05 from 0.50 to 0.80, selecting a balance between classification accuracy and conservative upgrade trigger rate (the proportion of samples that actually require deeper retrieval after upgrade). In this embodiment, the confidence threshold is used. When the maximum probability output by the classifier is lower than When the complexity level is conservatively upgraded, the subsequent steps are executed. The single-entity direct response type is upgraded to the multi-entity related type, and the multi-entity related type is upgraded to the reasoning and calculation type. The upgrade is carried out in one direction, and it is better to search one more hop than to miss key evidence. The upgraded sample is logged at the same time for S12 feedback. S14 describes the processing of the judgment features: In addition to the original question text, the classifier input also includes a set of low-cost pre-extracted features, including the number of entities obtained by named entity recognition (values of 1, 2, or not less than 3), and the hit counts of four types of vocabulary in the question, namely, comparison vocabulary (than, more, exceeding, lower than, difference, higher than, fastest, largest, smallest, which one), aggregation vocabulary (total, cumulative, average, percentage, total, growth rate, decline rate, average), ordinal vocabulary (first, first time, recent, continuous, before, after), and connection and nesting vocabulary (of, its, where, subordinate, company, author, place of origin); the vocabulary hit count and the number of entities are concatenated as sparse features after the sentence vector and sent to the classification head to reduce the training difficulty and inference latency of the classifier. The vocabulary content can be adjusted according to the domain corpus. S15 gives the rules for determining the level and scheduling parameters: direct-answer questions containing only a single entity and without comparison and aggregation features are determined as single-entity direct-answer type. In this case, the retrieval depth is 1 hop, and the number of knowledge fragments recalled per hop is... The first preset value is 3; problems that require bridging between more than two entities are classified as multi-entity relational problems, and problems that require aggregation, comparison, or numerical and temporal operations are classified as inference-computation problems. The upper limit of the retrieval depth for these two categories is the preset maximum depth. Each jump Example of categorized output: The probability distribution of the question "What is the warranty period for the A300 washing machine?" is approximately {0.87, 0.10, 0.03}. The maximum probability of 0.87 is greater than 0.6, so it is judged as a single-entity direct answer type.
[0029] To support S23 and S24, the entity recognition and linking modules are established as follows: The named entity recognition model is fine-tuned using a BERT-based Chinese pre-trained model with sequence labeling. Training data comes from the OntoNotes 5.0 Chinese subset (including named entities and common index annotations) released by the Language Data Consortium, the MSRA Chinese NER public dataset, and 10,000 manually annotated domain corpora from enterprise corpora. Annotation uses the BIO format, with entity category examples including product model, organization / company, person name, place name, time, value, and clause (7 categories in total). The training loss is token-level cross-entropy, and an F1 score of 0.85 or higher on the development set is required for deployment. An alias table is established for entity links: using the entity registry standard name as the primary key, and incorporating redirect pages from Chinese Wikipedia and Baidu Encyclopedia, as well as a manually maintained abbreviation lookup table from enterprises (e.g., "Three Guarantees Regulations → National Regulations on Repair, Replacement, and Return of Certain Commodities"). Candidate entity generation uses a two-pronged approach: an AC automaton performs exact prefix and full name matching, and an approximate retrieval of the top-20 entities using entity name vectors. Candidate ranking score is defined as: ; in, For the problem, For the current jump evidence text, and For candidate entity names and descriptions, Indicates text concatenation, Encoding for embedded models, To normalize the edit distance and other name string similarities, Set the value to 0.8; select the highest score that is not lower than the disambiguation threshold. Candidates are linked results; those below a threshold are classified as NIL and discarded to prevent irrelevant entities from polluting iterative expansion. Coreference resolution employs a two-level scheme of rules and models: the rule layer is responsible for alias unification and abbreviation expansion (e.g., "this company" refers to the most recently appearing company name in the context, "the above product" refers to the current jump entity), and the model layer uses spanBERT to fine-tune OntoNotes coreference data parsing for references such as "it" and "this company"; coreference resolution is performed before entity linking, so that different representations of the same entity are mapped to the same standard entity identifier, ensuring the consistency of deduplication keys in S25.
[0030] S2 is the retrieval scheduling step, where the retrieval scheduling strategy is determined based on the complexity level. The iterative retrieval scheduler maintains four data structures: the current set of jump entities. Visited entity set Evidence set With track list First-hop entity set The entity recognition and linking module extracts and links entities from the question. If no entities can be extracted from the question, the first-hop query is constructed using the question keywords. For single-entity direct-answer types, S21 and S22 are executed only once; for multi-entity association types, S21 to S28 are executed iteratively; for types requiring reasoning and calculation, S29 and S210 are executed additionally outside the loop. S21 is the query construction sub-step: concatenating the original question with the current jump entity set to form the current jump query. Assemble them one by one. This indicates string concatenation; in the case of a single-entity direct answer, the jump query is the original question text. S22 is the retrieval and evidence collection sub-step: the jump query calls the multi-way retrieval engine, and the jump knowledge fragment set is obtained through the dual-way recall and deduplication in step S4. ,Will Incorporated into the evidence set and with (skip sequence number) Entity set , The trajectory is recorded in the form of ) S23 is the candidate entity extraction sub-step: for Each segment runs a named entity recognition model to obtain a set of candidate entities. The extraction is limited to six entity types: product model, organization / company, person name, location, time, and terms. Numerical entities are excluded because they have no linking value. S24 outlines the entity linking and disambiguation steps: starting with the question... Evidence related to this jump For the context, as mentioned above Formula pair Entity-by-entity scoring links, below the threshold Candidates deemed NIL are discarded, and coreference resolution is performed before linking. S25 is the deduplication sub-step: removing visited entity sets from the candidate set after linking. The entities in the set are used to obtain a new set of entities. To prevent loops in the iteration process, S26 is the width and novelty filtering sub-step: first, by relevance: ; Sort the new entity set in descending order, retaining no more than the preset expansion width. The entities are then evaluated; the novelty of the remaining entities is then calculated: ; That is, subtract the highest semantic similarity between the entity and the most similar fragment in the evidence set from 1, and eliminate those with a novelty lower than the preset novelty threshold. The significance of this filtering is that if an entity, although not appearing in the visited set, has its semantics sufficiently covered by the collected evidence, and the information gain from continuing to search around it is limited, it should be stopped early to save computing power. S27 is the advancement sub-step: adding the remaining filtered entities to the visited entity set. and as the set of entities for the next hop. Return to S21 to begin the next hop. S28 is the termination condition sub-step: after each hop, terminate the iteration if any of the following three conditions are met: the new entity set obtained from the current hop is empty; the search depth reaches the preset maximum depth. The novelty of all entities in the new entity set is lower than that of the others. The three conditions correspond to three scenarios: "no new entities to search," "guaranteed depth limit," and "insufficient information gain," respectively, which together ensure that the iteration does not exceed the worst-case scenario. The upper bound of computing power is predictable. S29 is the sub-step for solving reasoning-computation type sub-problems: When the complexity is reasoning-computation type, the problem intent is detected using a rule template. If comparison words are hit, comparison sub-problems are generated; if aggregation words are hit, aggregation sub-problems are generated; if time words and numerical values are hit, time operation sub-problems are generated. The sub-problems are solved through structured query and numerical calculation. For example, the sub-problem "Repair rate of A300 washing machine from 2023 to 2025" is transformed into an SQL query SELECT year, AVG (repair_rate) FROM monthly _repair WHERE product='A300' AND year BETWEEN 2023 AND 2025 GROUPBY year. The comparison sub-problem "Difference between repair rates in 2025 and 2024" is subtracted by the calculator, and the result is returned in the form of numerical facts. S210 is the sub-step for incorporating synthetic evidence: The sub-problem results of S29 are written into the retrieval trajectory as synthetic evidence entries. The source identifier is recorded as calc or sql, and the version number is recorded as the current version. Together with the retrieved evidence, they enter the generation stage of step S6.
[0031] S3 is the incremental update step for the knowledge base. In response to knowledge base change events, it performs partial re-embedding only on the changed knowledge blocks, writes the re-embedding results to the incremental index, updates the version number of the corresponding entity in the entity registry, and writes tombstone records carrying incremental version numbers for deletion-type changes. Change events can originate from one or a combination of three sources: proactive push from the source system's Webhook (lowest latency), database CDC monitoring of change logs (suitable for structured data sources), and a polling scan every 5 minutes as a fallback (to prevent missed pushes). Events enter the processing pipeline via a Kafka message queue. Each entity record in the entity registry contains at least the following: entity_id (primary key, generated according to the rule "knowledge base prefix + resource identifier + first 8 digits of name hash"), standard_name (standard name), aliases (alias list), type (entity type), version (current version number), status (active or tombstone), source_id (source document identifier), source_priority (source priority, official documents are higher than internal standards, which are higher than secondary reposts), updated_at (update time), and chunk_count (number of associated chunks). The registry also stores a (entity_id, chunk_id) chunk list mapping for each entity, serving as the basis for S4 grouping and change cleanup. The content hash is calculated when each knowledge chunk is written. ; If the hashes are the same, the content is considered unchanged and no re-embedding is performed; only if the hashes are different will embedding be performed. The version number generation rule is version = 13-bit millisecond timestamp + 3-bit incremental sequence number, ensuring global monotonically increasing. Modifying a class change increments the version number on the original basis; deleting a class change does not physically delete the data, but instead writes a tombstone record with a larger version number and a status of "tombstone". The index segment manifest records the list of currently effective index segments. Each existing segment is marked as immutable (read-only), and incremental segments are marked as incremental (writable). The retrieval service only locates the currently effective index segment through the manifest. The manifest switches to atomic operations (atomic file renaming or pointer comparison and swapping), which is the basis for "hot-plugging" of indexes, i.e., taking effect without downtime. S31 is the parsing and segmentation sub-step: The modified document is processed according to the parsing and segmentation strategy completely consistent with the existing index to obtain each block. An example strategy is to first segment by title level, and then by paragraph and sentence boundaries. The target length is 512 characters, and adjacent blocks overlap by 50 characters. Each block retains its title path (such as "Product Manual / A300 Series / Maintenance Chapter") as metadata. Strategy consistency is a prerequisite for the correctness of subsequent hash comparison and merging. S32 is the hash comparison and local re-embedding sub-step: SHA-256 is calculated for each block and compared with the existing blocks. Only blocks whose content has changed are called to generate new vectors using the embedding model. Unchanged blocks retain the old vectors and metadata, and only their version association is updated. For example, if an 8000-word document modifies a paragraph of about 200 characters, according to the 512-character segmentation strategy, only 1 to 2 blocks will have hash changes. The number of embedding calls is reduced by about 80% to 90% compared to full document re-embedding. S33 is the idempotent write sub-step: Using (entity_id, version) as the idempotent key, the vector of the new block and the inverted record are written to the incremental index with upsert semantics. At the same time, the version and updated_at of the entity registry are updated. Repeated delivery will not generate duplicate records. S34 is the version verification sub-step: Before writing, the version number carried by the event is verified to be no less than the current version number in the registry. Otherwise, it is considered that the event is out of order or expired, and it is directly discarded and written to the audit log. Ordered consumption of the same document event by doc_id partition can further reduce out-of-order from the source. S35 is the retry and dead letter sub-step: For change events that fail to write, exponential backoff retry is performed, with intervals of 2 seconds, 4 seconds, and 8 seconds respectively, up to 3 times. If it still fails, it is entered into the dead letter queue and an alarm is set. It is then repaired manually or by a compensation task and re-delivered. S36 is the merge trigger judgment sub-step: Determine if the number of incremental index entries exceeds the preset threshold of 50,000, the proportion of tombstone records exceeds 10%, or more than 24 hours have passed since the last merge. If any of these conditions are met, the merge is triggered.S37 is the off-peak merging sub-step: During the preset off-peak period (02:00 to 05:00), the incremental index is merged and sorted with the most recent existing index segment to generate a new read-only index segment. During the merging process, tombstone records and old entries replaced by higher versions are physically cleared. At the same time, multiple versions are converged by (entity_id, chunk_id) to restore the index size and retrieval efficiency to a steady state. S38 is the atomic switching sub-step: After the merge is completed, the integrity of the new segment is verified (number of entries, sampling hash, retrieval sampling), and the manifest is atomically switched to the new segment. All subsequent retrievals immediately point to the new segment, and no downtime is required throughout the process. S39 is the old segment retention and rollback sub-step: The replaced old segment and old manifest are physically reclaimed after a preset retention period of 7 days for quick rollback in case of switch failure. Before each merge, a snapshot of the current manifest and registry is taken. If the merge fails, it automatically rolls back to the previous manifest and issues an alert.
[0032] S4 is a dual-path recall and deduplication step, executed in each hop of the retrieval process. The dense vector retrieval path uses the open-source bge-large-zh-v1.5 embedding model from the Beijing Academy of Artificial Intelligence (bge-m3 can be used for multilingual scenarios), with a vector dimension of 1024. The output is L2 normalized. Index writing and querying must use the same version of the same embedding model. The model name and version number are written as mandatory metadata to each index entry. Inconsistencies during query verification result in rejection or alerts / downgrades. Retrieval is completed using an HNSW approximate nearest neighbor index (example construction parameters M=16, efConstruction=200). The sparse inverted index retrieval path uses BM25 scoring, considering term frequency, document length normalization, and inverse document frequency. , The specific process of each hop retrieval is as follows: Dense searches are performed on both the incremental and existing indexes using the embedding vector of the current hop query, retrieving the top 200 from each; BM25 searches are performed on the inverted tables of both indexes using the word segmentation of the current hop query, retrieving the top 200 from each; the four recall sets enter the merger; first, the results from the four paths are merged according to their reciprocal ranking, and then uniformly sorted. The merging formula is: ; For document In the Rankings along the way The ranking merging process relies solely on rank, not score, to avoid the incomparability of dimensions between dense and sparse scores. Then, groups are created using the grouping key (entity_id, chunk_id). Within each group, the activity entry with the highest version number is retained. If a group contains a tombstone record with a version number higher than any other activity entry, the entire group is removed. Cross-source conflicts involving the same entity (the same standard name linked as the same entity from different sources) are resolved in the order of "source priority > version number > version number > timestamp". Finally, the top-ranked entries are truncated in descending order of their merged scores. (This embodiment) As the output of this knowledge fragment set, each fragment carries metadata (entity_id, chunk_id, version, ts, source_id). For example: A policy document's same chunk has an active entry for version 5 in the existing index, and active entries for versions 7 and 9 in the incremental index. The merger retains the version 9 entry. If the document is subsequently deleted, and the incremental index writes a tombstone record for version 10, then this group is completely removed during retrieval, and the answer no longer references obsolete content.
[0033] S5 is the trajectory recording step, which records the entities, knowledge fragments, and version information of each knowledge fragment hit in each hop, forming a retrieval trajectory. Sub-step S51 organizes the trajectory by hop, and each trajectory entry carries at least the hop number, the set of entities used in that hop, the original text of the hit knowledge fragment, the source identifier, the version number, and the timestamp. An example structure of a two-hop trajectory is as follows: Question: "Which company manufactures the motor used in the A300 washing machine?", complexity: "Multi-entity association type"; the first hop entity set is "A300 washing machine", evidence item 1 text: "The A300 washing machine uses an XH-900 direct-drive inverter motor.", source: "Product Manual A300.pdf", block identifier c88421, version 7, timestamp 2026-08-12T10:00:00Z; the second hop entity set is "XH-900 direct-drive inverter motor", evidence item 2 text: "The XH-900 direct-drive inverter motor is manufactured by a certain electromechanical company.", source: "Parts list.pdf", block identifier c11209, version 3, timestamp 2026-05-30T08:12:00Z. The trajectory serves two purposes: first, as an evidence chain for generating the answer; second, as a carrier of version information for use in S6 conflict resolution.
[0034] Step S6 is the answer generation step. It involves inputting the user's question, complexity level, and retrieval trajectory into a prompt template, which is then input into a large language model to generate an answer with evidence citations. Sub-step S61 injects citation constraints into the prompt template, requiring each factual assertion in the answer to be labeled with the evidence number it is based on. A sample full template is as follows: The first part is the original question and complexity level; the second part is the retrieval trajectory, listing entities and evidence by jump, with each piece of evidence including its number, source, version, and timestamp; the third part is the reasoning requirements, including reasoning hop-by-hop based on the trajectory without using facts outside the trajectory, labeling each factual assertion in the answer with an evidence number, prioritizing evidence with a larger version number or more recent timestamp when multiple pieces of evidence conflict about the same fact, and explicitly stating "cannot be determined based on existing knowledge" when evidence is insufficient, rather than fabricating evidence. Sub-step S62 uses both prompt constraints and post-processing to resolve conflicts: the generation stage is constrained by the template, and the post-processing stage performs consistency checks on the evidence entries cited in the answer. If an answer is found to have adopted an older version of evidence, it prompts for regeneration or labels the answer as "this information may have been updated." Example of generated result: "The A300 washing machine uses an XH-900 direct drive inverter motor [1], which is produced by a certain electromechanical company [2]." The [1][2] in the answer correspond one-to-one with the trajectory evidence number, realizing the traceability of assertions.
[0035] The following are three end-to-end implementation examples. Implementation example one is a single entity direct answer type, the question is "How long is the warranty period for the whole machine of the A300 washing machine?". The output probability distribution of the S1 classifier is approximately {0.91, 0.07, 0.02}, the maximum probability is greater than the threshold of 0.6, and it is judged as a single entity direct answer type. The entity extraction obtains the first hop entity set "A300 washing machine". S2 Since the level is a single entity direct answer type, only S21 and S22 are executed once. The question is combined with known entities as the query. The top-3 evidence is obtained through S4 dual-path recall and version filtering. Among them, the service policy document block (version 6) clearly states "the whole machine is under warranty for three years and the main components are under warranty for ten years". The other two are descriptions of appearance and performance. S5 records the 1-hop trajectory, and S6 generates the answer "The whole machine of the A300 washing machine is under warranty for three years and the main components are under warranty for ten years" [1]. Except for the classifier overhead of about 18 milliseconds, the entire link is comparable to the fixed single-hop retrieval enhancement generation delay, which shows that the delay of simple questions does not increase.
[0036] Example 2 is a multi-entity association problem, with the question "Which company manufactures the motor used in the A300 washing machine?". S1: The classifier outputs approximately {0.05, 0.88, 0.07}, classifying it as a multi-entity association problem. S2: The first hop constructs a query using the entity "A300 washing machine". S4: Recalls evidence 1 "The A300 washing machine uses an XH-900 direct-drive inverter motor". S23: Extracts the candidate entity "XH-900 direct-drive inverter motor" from this evidence. S24: The link is successful. S25: This entity is not in the visited set. S26: The relevance (0.81) and novelty (0.92) both exceed the threshold. S27: Obtain the next-hop entity set. S28: The judgment is not terminated. The second hop constructs a query with the entity "XH-900 direct drive inverter motor". S4 recalls evidence 2 "XH-900 direct drive inverter motor is produced by a certain electromechanical company". S23 extracts the candidate entity "a certain electromechanical company". S25 removes the visited entities and the new entity set is empty. S28 the termination condition is met and the iteration ends. S6 generates the answer "A300 washing machine uses XH-900 direct drive inverter motor [1], which is produced by a certain electromechanical company [2]". If only a single hop search is performed, the first hop evidence only contains the motor model and not the manufacturer, so the answer cannot be completed. This embodiment reflects the source of the improved answer completeness rate of complex questions.
[0037] Example 3 is a reasoning calculation type, the question is "From 2023 to 2025, how many percentage points did the annual repair rate of the A300 washing machine decrease?". S1 hits the comparison word and time word, and is judged to be a reasoning calculation type. S2 iterates and retrieves the repair rate reports for each year from 2023 to 2025 as evidence 1, 2 and 3, each with version and timestamp; S29 decomposes the aggregate sub-problem and time operation sub-problem, and uses SQL to retrieve data and calculator to calculate the difference; S210 takes "2023 repair rate 4.2%, 2025 repair rate 2.1%, a decrease of 2.1 percentage points" as synthetic evidence 4 and incorporates it into the trajectory. S6 generates the answer "The annual repair rate of the A300 washing machine decreased from 4.2% in 2023[1] to 2.1% in 2025[3], a decrease of 2.1 percentage points[4]", the calculation process is interpretable and the value is traceable.
[0038] The complete process of building the system from scratch to stable operation is as follows: The first step is data preparation: Collect knowledge base documents, parse various formats using PDF, Word, and HTML parsers respectively, remove headers, footers, page numbers, and watermarks, unify full-width and half-width characters and encoding, then segment and record title path metadata according to the segmentation strategy described in S31, and simultaneously build the entity registry skeleton and alias table. The second step is classifier cold start: Complete the seed question set construction, automatic large model annotation, manual sampling, and distillation training according to S12, so that the complexity classification module meets the online standard. The third step is full index initialization: Segment according to S31, generate vectors block by block using the bge-large-zh-v1.5 embedding model, build existing read-only segments and write them to the manifest, and simultaneously build an inverted index and batch register entities; this stage is the only full construction. The fourth step is gray-scale deployment: Use the index alias mechanism to switch retrieval traffic to the new system proportionally. For example, first use 10% traffic to verify the recall hit rate and answer satisfaction, then gradually increase the volume, with rollback possible throughout. The fifth step is steady-state operation: incremental pipelines are continuously written according to S31 to S35, low-peak merging is routinely executed according to S36 to S39, classifier sample backflow is periodically retrained according to S12, and closed-loop operation and maintenance are achieved by monitoring indicators such as knowledge update effective latency (UTL), recall hit rate, tombstone ratio, dead letter queue backlog, and P95 end-to-end latency. UTL is defined as the difference between the time when a change event enters the pipeline and the time when the change content is first retrieved and recalled. It is measured by injecting 10 probe changes per hour and querying their recallability.
[0039] To verify the technical effectiveness of this invention, exemplary evaluations were conducted on publicly available datasets. The data provided are results obtained under the parameter configurations of this embodiment, used to illustrate the magnitude of the effect and not to limit the effectiveness of this invention. The evaluation datasets and sources are: HotpotQA validation set (multi-hop bridging and comparison, 7405 entries), MuSiQue development set (combined multi-hop), 2WikiMultiHopQA (cross-document multi-hop), and NaturalQuestions open set (single-hop, 3610 entries). The metrics are defined as follows: Exact Match (EM) is the percentage of samples where the predicted answer is completely identical to the standard answer; Answer F1 is calculated based on term overlap, i.e., the product of twice the precision and recall divided by the sum of the two; Evidence Coverage (EC) is the size of the intersection of the answer's cited evidence set and the reference answer's evidence set divided by the size of the reference answer's evidence set; P95 latency is the 95th percentile end-to-end response time; UTL is as defined above. The comparison baselines include fixed 1-hop retrieval enhancement generation, fixed 2-hop retrieval enhancement generation, and a retrieval system that replicates the complexity-adaptive retrieval concept. Exemplary results are as follows: On multi-hop datasets, the F1 score of the present invention is improved by 9 to 14 percentage points compared to the fixed 1-hop baseline and by 3 to 5 percentage points compared to the fixed 2-hop baseline, with an evidence coverage improvement of approximately 12 percentage points; On the NaturalQuestions single-hop dataset, the P95 latency increases by only 3.2% compared to the fixed 1-hop baseline, the classifier overhead is approximately 18 milliseconds, and the latency for verifying simple questions does not increase; In terms of update timeliness, the UTL is reduced from approximately 8.2 hours for the full reconstruction scheme to approximately 3 minutes, with incremental writes taking effect within minutes; Ablation experiments show that after enabling the novelty termination condition, the average number of retrieval hops decreases from 2.3 hops to 1.7 hops, the end-to-end latency decreases by 22%, and the multi-hop F1 score drops by only 0.8 percentage points, demonstrating that the verification termination condition achieves significant computational gains at a minimal quality cost; After enabling peak merging, the number of index entries running for 30 days converges by approximately 60% compared to the non-merging case, and the retrieval throughput remains stable.
[0040] Those skilled in the art can make various modifications within the scope of this invention, all of which fall within the protection scope of this invention: The complexity classifier can be replaced by a general large language model that directly prompts classification, but the classification latency is high, making it only suitable for low-traffic or offline scenarios; the vector library can be replaced by pgvector, Qdrant, or self-managed FAISS indexes. When the selected vector library does not provide segment merging capabilities, the application layer implements segment management and list switching from S37 to S39, with the method and effect remaining unchanged; the incremental index can use a small memory index plus WAL logs to achieve immediate effect upon writing, and reconstruction from the logs after a crash, further reducing write latency; in distributed deployment, the iterative retrieval scheduler can be horizontally scaled with stateless multi-replicas, with entity registry master-slave deployment, merging tasks executed in a single instance and mutually exclusive with distributed locks; in multi-language scenarios, it can be replaced by multi-language embedding models such as bge-m3; in multimodal scenarios, chart-type documents are first converted to text via OCR or layout parsing before entering the same pipeline; during domain migration, the S14 vocabulary and entity categories are replaced with a domain vocabulary, such as agency names, document numbers, and policy clauses in the government domain, or drug names, disease names, and dosages in the medical domain, with the rest of the process remaining unchanged.
[0041] The above embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention. All modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A question-answering method and system for a large language model based on multi-source knowledge retrieval, characterized in that, Includes the following steps: S1: Receive user questions, input the user questions into a pre-trained lightweight complexity classifier, and obtain the complexity level and corresponding confidence level. The complexity level includes at least single entity direct answer type, multi-entity association type and reasoning calculation type. S2: Determine the retrieval scheduling strategy based on the complexity level; when the complexity level is single-entity direct answer type, execute single-hop retrieval; when the complexity level is multi-entity association type or requires reasoning calculation type, execute entity-driven iterative multi-hop retrieval: construct a query with the entity set of the current hop and the user question, and retrieve the knowledge fragment set of the current hop; perform named entity recognition on the knowledge fragment set of the current hop to obtain a candidate entity set; after performing entity linking and coreference resolution on the candidate entity set, remove the entities in the visited entity set, and use the remaining entities as the entity set of the next hop, until the preset termination condition is met; S3: In response to knowledge base change events, only perform local re-embedding on the knowledge blocks that have been changed, write the re-embedding results to the incremental index, and update the version number of the corresponding entity in the entity registry; delete class changes are written with tombstone records carrying incremental version numbers. S4: In each hop search, dense vector search and sparse inverted index search are performed on the incremental index and the stock index respectively. The recall results of each index and each method are grouped according to entity identifier and block identifier. The entry with the largest version number is retained in the same group. When there is a tombstone record with a higher version number in the same group, the group is removed. After merging and deduplication, the knowledge fragment set of this hop is obtained. S5: Record the entities, knowledge fragments, and version information of each knowledge fragment hit in each hop to form a retrieval trajectory; S6: Combine the user question, the complexity level, and the retrieval trajectory into the prompt template, input it into the large language model, and generate an answer carrying evidence citations.
2. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, Step S1 includes the following sub-steps: S11: After encoding the user question, the lightweight complexity classifier outputs the probability distribution of each complexity level through the lightweight classification network, and takes the level with the highest probability as the complexity level. S12: The lightweight complexity classifier is trained by distillation of training samples, which are automatically labeled by a large language model and manually checked and corrected. S13: When the maximum probability in the probability distribution is lower than the preset confidence threshold, the complexity level is conservatively upgraded and subsequent steps are executed, wherein the single-entity direct answer type is upgraded to the multi-entity association type, and the multi-entity association type is upgraded to the reasoning calculation type.
3. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, Step S1 includes the following sub-steps: S14: Perform entity recognition on the user question, extract the number of entities, and detect the hit status of comparison words, aggregation words, ordinal words and conjunction words; S15: Determine the complexity level based on the number of entities and the hit rate: direct-answer questions containing only a single entity and without comparison and aggregation features are determined as single-entity direct-answer questions; associative questions that require bridging across two or more entities are determined as multi-entity associative questions; questions that require aggregation, comparison, or numerical and time operations are determined as reasoning-computation-required questions. In step S2, the retrieval scheduling strategy further includes: when the complexity level is single entity direct response type, the retrieval depth is 1 hop and the number of recalled knowledge fragments does not exceed a first preset value; When the complexity level is multi-entity association type or requires reasoning calculation type, the upper limit of the retrieval depth is the preset maximum depth.
4. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, In step S2, each hop of the entity-driven iterative multi-hop retrieval includes the following sub-steps: S21: Construct a query for this jump using the set of entities in the current jump and the user question; S22: Perform a retrieval based on the jump query to obtain a jump knowledge fragment set, and incorporate the jump knowledge fragment set into the evidence set and record it in the retrieval trajectory; S23: Perform named entity recognition on the knowledge fragment set of this jump to obtain a candidate entity set; S24: Using the user question and the knowledge fragment set of this jump as context, perform entity linking and coreference resolution on the candidate entity set and map it to standard entities; S25: Remove the entities from the visited entity set from the linked candidate entity set to obtain a new entity set; S26: Sort the new entity set according to its relevance to the user's question, retain entities whose number does not exceed a preset expansion width, and remove entities whose novelty relative to the collected evidence set is lower than a preset novelty threshold; S27: Add the remaining entities to the visited entity set as the entity set for the next hop.
5. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, In step S2, the preset termination condition is determined in sub-step S28, which includes: The system determines whether the newly obtained entity set is empty, whether the search depth has reached the preset maximum depth, and whether the novelty of all entities in the new entity set relative to the collected evidence set is lower than the preset novelty threshold. If any of these determinations are true, the iterative multi-hop search is terminated. The novelty is defined as the difference between 1 and the maximum semantic similarity between the entity and the most similar knowledge fragment in the collected evidence set.
6. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, In step S2, when the complexity level is of the type requiring reasoning computation, the following sub-steps are also included: S29: Decompose the user problem into subproblems to obtain aggregation subproblems, comparison subproblems, or numerical and time operation subproblems, and perform structured queries and numerical calculations on the subproblems to obtain the subproblem results; S210: The results of the sub-question are incorporated as synthetic evidence into the retrieval trajectory and used to generate the final answer.
7. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, Step S5 includes sub-step S51: Organize the retrieval trajectory by jump, with each trajectory entry carrying at least the jump number, the entity set used by the jump, the original text of the hit knowledge fragment, the source identifier, the version number, and the timestamp; Step S6 includes the following sub-steps: S61: Inject constraints into the prompt template such that each factual assertion in the answer references the evidence number of the corresponding knowledge fragment in the retrieval trajectory; S62: When multiple knowledge fragments provide conflicting information about the same fact, the knowledge fragment with the larger version number or the more recent timestamp shall be given priority.
8. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, In step S3, the processing of the change event includes the following sub-steps: S31: Receive change events, process the changed documents according to the parsing and splitting strategy consistent with the existing index, and obtain each block; S32: Calculate the content hash of each block and compare it with the existing blocks. Only perform embedding calculations on blocks whose content has changed. S33: Using the combination of entity identifier and version number as an idempotent key, write the re-embedding result into the incremental index and the entity registry; S34: Verify that the version number carried by the change event is not less than the current version number in the entity registry; otherwise, discard the change event and record the audit log. S35: Retry the write failure change event. After the preset number of retries is exceeded, transfer it to the dead letter queue.
9. The question-answering method and system based on a large language model with multi-source knowledge retrieval according to claim 1, characterized in that, Step S3 also includes the maintenance of the incremental index, which includes the following sub-steps: S36: Determine whether the merge triggering conditions are met. The merge triggering conditions include any one of the following: the number of incremental index entries exceeds a preset threshold, the proportion of tombstone records exceeds a preset ratio, or the time elapsed since the last merge exceeds a preset duration. S37: When the merge triggering condition is met, the incremental index is merged with the most recent existing index segment during a preset off-peak period to generate a new read-only index segment. During the merging process, tombstone records and old entries replaced by higher versions are physically cleared. S38: After the merging is completed, the index segment list is atomically updated, and subsequent searches are directed to the new read-only index segment; S39: The old index segment that has been replaced is retained for a preset time and then recycled for rollback in case of failure.
10. A large language model question-answering system based on multi-source knowledge retrieval, used to implement the method of any one of claims 1-9, characterized in that, include: The complexity classification module is used to: receive user questions and output complexity level and confidence level, wherein the complexity level includes at least single entity direct answer type, multi-entity association type and reasoning calculation type; The entity recognition and linking module is used to extract question entities from the user questions, and to perform named entity recognition, entity linking and coreference resolution on the recalled knowledge fragments; The versioned knowledge index module is used to perform the following actions: responding to knowledge base change events, partially re-embedding and writing incremental indexes only for the changed chunks, updating the version number of entities in the entity registry, and deleting class changes to write tombstone records carrying incremental version numbers. When the merge trigger condition is met, the incremental index is merged with the most recent existing index segment during off-peak hours to generate a new read-only index segment and atomically update the index segment list; the versioned knowledge index module includes existing index segments, incremental index segments, and an index segment list; A multi-way retrieval engine is used to perform: dense vector retrieval and sparse inverted index retrieval on the incremental index and the existing index respectively, and to perform structured query; and to group the recall results according to entity identifier and block identifier, retain the entry with the largest version number in the same group, and remove the group when there is a tombstone record with a higher version number in the same group, and merge and deduplicate to obtain the knowledge fragment set of this jump; An iterative retrieval scheduler is used to execute: determine a retrieval scheduling strategy based on the complexity level; when the complexity level is a single-entity direct response type, execute a single-hop retrieval; when the complexity level is a multi-entity association type or a reasoning calculation type, execute an entity-driven iterative multi-hop retrieval, and feed back new entities appearing in the knowledge fragment set of this hop into the next hop query after processing by the entity recognition and linking module, until a preset termination condition is met. The trajectory assembly and generation module is used to: record the entities, knowledge fragments and version information hit in each hop to form a retrieval trajectory, and combine the user question, the complexity level and the retrieval trajectory into a prompt template, input it into a large language model, and generate an answer with evidence citations.
Citation Information
Patent Citations
Complex question multi-hop intelligent question answering method based on knowledge graph representation learning
CN115757715A
Retrieval generation method and system based on large model and knowledge graph fusion
CN121388189A
Adaptive question answering method based on large language model and intelligent large language model
CN121960739A
