Knowledge retrieval filtering method and system for assisting large model questions and answers
By constructing a retrieval master graph of entity relationships and combining a random walk algorithm and a lightweight filter, the problems of user differences and noisy documents in the question-and-answer system in the financial and tax field are solved, realizing efficient and personalized knowledge retrieval and question-and-answer services.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-03-13
AI Technical Summary
Existing Q&A and retrieval systems in the finance and tax field cannot effectively capture the diverse needs of users. Traditional RAG frameworks are difficult to accurately capture complex multi-hop information and abstract relationships in finance and tax scenarios, and are prone to introducing noisy documents that can lead to illusions.
A master graph of entity relationships is constructed for retrieval. Subgraph retrieval is performed using the random walk algorithm (PPR) and the re-ranking model (Rerank). Lightweight filters are used to filter irrelevant documents, forming an efficient knowledge retrieval and filtering mechanism.
It improves the accuracy and reliability of large-scale models in the field of finance and taxation, reduces illusions, and provides personalized and efficient intelligent finance and tax services.
Smart Images
Figure CN121658639A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large model-assisted question answering, and specifically to a knowledge retrieval and filtering method and system for assisting large model question answering. Background Technology
[0002] In the era of big data, data is growing at an unprecedented speed and scale, exhibiting characteristics of massive volume, diverse sources, complex modalities, and significant fragmentation. Especially in the field of finance and taxation, with the advancement of informatization and digitalization, massive amounts of invoices, policies and regulations, business cases, and announcement interpretations are constantly accumulating, forming a vast and complex knowledge asset. This data itself contains rich value, but due to its diverse forms, inconsistent organization, and frequent updates, traditional financial and tax services have failed to fully utilize this data resource. Existing service models are mostly limited to simple queries based on fixed databases, unable to achieve deep knowledge utilization across documents and relationships. This model is functionally limited, lacks flexibility, and is even less able to meet users' needs for intelligent, personalized, and interpretive financial and tax services.
[0003] Meanwhile, there are significant differences among tax and finance users. Different users have different knowledge backgrounds, business scenarios, and key needs. For example, corporate finance personnel focus more on policy interpretation and practical business operations, tax officials emphasize regulations and compliance, while researchers tend to focus on historical data and trend analysis. However, traditional tax and finance Q&A and retrieval systems typically provide uniform search results, lacking the ability to perceive user differences. This means that the system cannot effectively capture users' subjective mental characteristics, struggles to understand their true needs, and often produces a "one-size-fits-all" approach, lacking personalization and targeting. Therefore, given the ever-expanding scale of tax and finance data, how to fully leverage the capabilities of large-scale models, combined with effective knowledge retrieval and filtering mechanisms, to build an intelligent agent capable of perceiving user needs and dynamically adjusting strategies has become a core issue in promoting the upgrade of a smart tax and finance service system.
[0004] In recent years, the rapid development of artificial intelligence, especially Large Language Models (LLMs), has provided new opportunities for the intelligent transformation of the finance and taxation field. LLMs have demonstrated powerful capabilities in natural language understanding, knowledge reasoning, and dialogue interaction, and can, to some extent, simulate the human process of knowledge acquisition and utilization. However, the knowledge source of LLMs mainly relies on the corpus used in the training phase, which is often static and limited, and is clearly insufficient for the constantly updated and highly specialized field of finance and taxation. On the other hand, continuously injecting new data into the model is both impractical and incurs huge costs. Therefore, how to enable LLMs to efficiently utilize external knowledge bases has become a key research focus.
[0005] Retrieval-Augmented Generation (RAG) is currently the mainstream method for enhancing the knowledge utilization capabilities of LLM (Limited Language Management). Its basic idea is to first retrieve relevant documents from an external knowledge base when answering user questions, and then input these documents along with the query into the LLM to generate a more accurate answer. The RAG method has greatly alleviated the dependence of LLM on training corpora, making its application in professional fields possible. However, traditional RAG frameworks face two prominent problems in financial and tax scenarios: First, external knowledge bases are often structurally complex, containing not only legal provisions and invoice data, but also case interpretations and policy announcements. Implicit cross-document and cross-path relationships exist between the data, making it difficult for traditional RAG to accurately capture these complex multi-hop information and abstract relationships. Second, RAG typically retrieves a fixed number of m documents for each query, but in practical applications, the number of relevant documents for some queries is far less than m, inevitably introducing a large number of noisy documents. As the number of irrelevant documents increases, the probability of hallucination when the LLM generates an answer also increases, leading to results that deviate from user needs. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention provides a knowledge retrieval and filtering method for assisting large-scale model question answering, comprising:
[0007] Based on the knowledge base, construct a retrieval master graph of entity relationships;
[0008] Receive the user's input search question, and divide the master search graph into subgraphs related to the search question;
[0009] The random walk algorithm PPR is used in the subgraph to obtain a first candidate document set by calculating the score of each node in the subgraph; the first candidate document set is ranked by the reranking model Rerank according to the relevance between the user input retrieval question and the first candidate document set.
[0010] Using a pre-trained lightweight filter, the first candidate document set with an input tag of 0 is filtered according to the relevance between the user's input retrieval question and the first candidate document set to obtain a second candidate document set;
[0011] The second candidate document set and the user-input search question are used as prompt words and input into the large model to assist the large model in answering the user-input search question.
[0012] Furthermore, based on the knowledge base, a master graph for entity relationship retrieval is constructed, including:
[0013] Extract entities from knowledge base documents using a large model;
[0014] Based on the entity, extract the triplet relation containing the entity's corresponding entity;
[0015] The entity is treated as a single node, and the relationship edge is used as the edge connecting two entities. The document and the entities it contains are connected by a containing edge.
[0016] The KNN algorithm is used to calculate the score and number of nodes for each node, and to identify similar nodes for each node.
[0017] Each node is connected to similar nodes through similar edges to construct a master graph for entity relationship retrieval.
[0018] Furthermore, the specific steps for obtaining similar edges are as follows:
[0019] The vectors between the nodes are obtained by embedding the nodes;
[0020] Use the coherent similarity to calculate the vector similarity between nodes and obtain the top K similar nodes corresponding to each node;
[0021] The vector from each node to the first K similar nodes is the similar edge between each node and the similar nodes.
[0022] Furthermore, the system receives a search question input by the user and divides the master search graph into sub-graphs related to the search question, including:
[0023] Based on the user's input search question, extract the corresponding entity and triple relationship;
[0024] Obtain the vector embedding of the triplet as the query vector;
[0025] Calculate the similarity between the query vector and the triples in the knowledge base, and take the entity nodes and text block nodes whose similarity vector scores exceed the threshold from the top K most relevant triples as the initial nodes of the subgraph PPR algorithm.
[0026] In the retrieval parent graph, find the subgraph that is adjacent to (k=1) the initial node k of the subgraph PPR algorithm as the subgraph related to the retrieval question.
[0027] Furthermore, the Random Walk (PPR) algorithm is used in the subgraph to obtain a first candidate document set by calculating the score of each node in the subgraph, including:
[0028] The random walk algorithm PPR is used to walk through the subgraph and calculate the score of each node;
[0029] After the walk ends, the scores of each node are sorted to obtain the first candidate document set.
[0030] Furthermore, based on the relevance of the user-input search question to the first candidate document set, the first candidate document set is ranked using the Rerank model, including:
[0031] The user-input search question is concatenated with candidate documents and used as input to a bidirectional encoder structure.
[0032] The bidirectional encoder structure calculates the semantic interaction information of the input through multiple layers of Transformer encoding and outputs the corresponding relevance score;
[0033] The first candidate document set is sorted according to the relevance score, and the sorting result is used as the re-sorting result.
[0034] Furthermore, the training steps for the lightweight filter are as follows:
[0035] Collect the retrieved data as training data;
[0036] The teacher model is used to query the training data and determine its relevance to the first candidate document, generating 0 / 1 labels;
[0037] The student model is trained using hard distillation with (Q, D, Label) to learn the relevance judgment ability of the teacher model.
[0038] The trained lightweight student model is used to differentiate the relevance of the search results and filter out irrelevant documents.
[0039] Furthermore, the specific steps for generating tags are as follows:
[0040] The relevance of the query and candidate documents to the input teacher model is determined.
[0041] The teacher's large model outputs binary labels, where a label of 0 indicates no relevance and a label of 1 indicates relevance, resulting in a training dataset consisting of (Q, D, Label).
[0042] Furthermore, the student model is hard-distilled using (Q, D, Label), including:
[0043] The (Q, D, Label) pairs are fed into the lightweight student model as input.
[0044] The lightweight student model learns the mapping relationship of the teacher model on the difference in relevance.
[0045] The student model is fitted to the discriminative ability of the teacher model through hard distillation.
[0046] This invention also provides a knowledge retrieval and filtering system for assisting large-scale question answering models, comprising:
[0047] The retrieval master graph construction module, based on the knowledge base, constructs a retrieval master graph of entity relationships;
[0048] The subgraph partitioning module is used to receive the search question input by the user and partition the search master graph into subgraphs related to the search question.
[0049] The first candidate document set acquisition module is used to obtain the first candidate document set by using the random walk algorithm PPR in the subgraph and calculating the score of each node in the subgraph; and to sort the first candidate document set by the reordering model Rerank according to the relevance between the user input retrieval question and the first candidate document set.
[0050] The second candidate document set acquisition module is used to use a pre-trained lightweight filter to filter the first candidate document set with an output label of 0 based on the relevance between the user-input retrieval question and the first candidate document set, thereby obtaining the second candidate document set.
[0051] The question-answering module is used to input the second candidate document set and the user-inputted search question as prompt words into the large model to assist the large model in completing the question-answering of the user-inputted search question.
[0052] This invention provides a knowledge retrieval filtering method and system for assisting large model question answering. Based on graph-based retrieval enhancement generation, it uses query-related subgraphs for intensive retrieval and filters the retrieval results using proprietary models. This significantly enhances the ability of large models to answer questions in a specific domain and reduces phenomena such as large model illusion. Attached Figure Description
[0053] Figure 1 This is a flowchart illustrating a knowledge retrieval and filtering method for assisting large-scale question answering provided in an embodiment of the present invention;
[0054] Figure 2 This is a schematic diagram of the subgraph construction method involved in an embodiment of the present invention;
[0055] Figure 3 This is a schematic diagram of the training and usage method of the lightweight filter involved in the embodiments of the present invention;
[0056] Figure 4 This is a schematic diagram of a knowledge retrieval and filtering framework for assisting large-scale question answering, according to an embodiment of the present invention.
[0057] Figure 5 This is a schematic diagram of the structure of a knowledge retrieval and filtering system for assisting large-scale question answering, provided in an embodiment of the present invention. Detailed Implementation
[0058] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0059] To address the aforementioned technical problems, this invention provides a knowledge retrieval filtering method and system for assisting large-scale model question answering, solving the problems of multi-hop reasoning difficulties, insufficient retrieval accuracy, and high illusion rate in existing RAG systems for complex knowledge retrieval. This method introduces structured subgraph reasoning and lightweight discrimination mechanisms into large-scale model question answering tasks, thereby improving the relevance and controllability of retrieval results and providing a more reliable intelligent service experience for fields with high precision requirements such as finance and taxation. Figure 1 As shown, the present invention provides a knowledge retrieval and filtering method for assisting large-scale question answering models, comprising the following steps:
[0060] Step S101: Based on the knowledge base, construct a retrieval master graph of entity relationships.
[0061] The system extracts informational entities from knowledge base documents using a large model; based on the extracted entities, it uses the large model to extract the triplet relations corresponding to the entities; the entity is treated as a single node, and the relation edge is used as the edge connecting two entities, with the document and its contained entities connected by an inclusion edge; the KNN algorithm is used to calculate the score and number of each node, and the similar nodes of each node are determined; each node is connected to similar nodes by similarity edges to construct a retrieval master graph of entity relations.
[0062] Furthermore, the specific steps for obtaining similar edges are as follows:
[0063] The vectors between the nodes are obtained by embedding the nodes;
[0064] The covariance similarity is used to calculate the vector similarity between nodes, and the top K similar nodes corresponding to each node are obtained, that is, all similar edges starting from that point.
[0065] Specifically, the construction of the retrieval master graph involves extracting key information from knowledge base documents using a large model to construct a graph containing entity nodes, relational edges, containment edges, and similar edges.
[0066] The retrieval master graph is a multi-relational graph structure, where each node corresponds to an entity concept in the knowledge base, and entities possess specific semantic meanings and attribute characteristics. Edges between nodes describe various relationships between entities, including semantic relationships, inclusion relationships, and similarity relationships. In the retrieval master graph, the importance of each node depends both on its own semantic information and its connectivity within the graph structure; therefore, it is necessary to construct a graph network that can fully express these complex relationships.
[0067] Suppose the knowledge base contains N documents, and M entity nodes are extracted from the large model M. The specific steps for constructing the retrieval master graph using a graph structure are as follows:
[0068] Entity extraction: First, a large language model M is used to perform deep semantic analysis on all documents in the knowledge base to identify and extract information-rich entities. These entities include, but are not limited to, named entities, concept entities, and event entities, each carrying rich semantic information.
[0069] Relation extraction: Based on the extracted entities, the semantic relationships between entities are further analyzed using a large model M, and triple relations containing the corresponding entities are extracted. These relations constitute direct connection edges between entities, forming the core structure of the knowledge graph.
[0070] Graph structure construction: Entities are represented as nodes in the graph, and relational edges are used to connect two entity nodes. Additionally, documents are connected to their contained entities using "containment" edges, representing the ownership relationship between the document and the entity. Furthermore, the semantic similarity between entity nodes is calculated using the KNN algorithm, finding the K most similar nodes for each entity node and connecting them with "similar" edges.
[0071] Similar edge calculation: Generate entity embedding representations and use similarity calculation methods to establish similar connections between nodes.
[0072] Generate entity embedding representation: First, use the embedding model to vectorize the extracted entity nodes, generating the embedding vector for each entity, resulting in a set E = {e1, e2, ..., e...} n These embedding vectors capture the semantic features of entities, and the semantic similarity between entities can be reflected by the distance in the vector space.
[0073] Cosine similarity is used to calculate similar nodes: For each entity node, the cosine similarity between its embedding vector and the embedding vectors of all other entity nodes is calculated. Cosine similarity assesses the degree of similarity between two vectors by measuring the cosine of the angle between them, and the formula is as follows:
[0074]
[0075] Among them, e i ·e j Denotes the dot product of vectors, ||e i ||and||e j || represent the magnitude of the vector, respectively.
[0076] Obtain the top K similar nodes: For each entity node, sort them from high to low according to their cosine similarity scores, and select the top K most similar nodes. These similar nodes are connected to the current node through "similar" edges, forming a semantic similarity network.
[0077] Establishment of similar edges: For node i, the set of its first K similar nodes is S_i={s1,s2,...,s_K}. For each similar node s_j∈S_i, establish a "similar" edge from node i to s_j. The weight of the edge is the corresponding cosine similarity score.
[0078] By constructing this retrieval master graph containing multiple types of edges, not only are direct semantic relationships between entities preserved, but indirect associations based on semantic similarity are also established, while maintaining the hierarchical structure between documents and entities. This multi-level graph structure provides a rich semantic information foundation for subsequent knowledge retrieval and filtering.
[0079] Step S102: Receive the search question input by the user, and divide the search master graph into subgraphs related to the search question.
[0080] In this invention, the overall retrieval steps are as follows:
[0081] Extract the portion of the parent graph that is most relevant to the current query, i.e., the subgraph;
[0082] Knowledge retrieval based on PPR algorithm: The PPR algorithm is used to retrieve knowledge from the extracted subgraphs;
[0083] Perform a reranking step on the PPR search results to improve search accuracy.
[0084] Specifically, based on the user-input search question, the corresponding entity and triple relationship is extracted; the vector embedding of the triple is obtained as the query vector; the similarity between the query vector and the vectors of triples in the knowledge base is calculated, and the entity nodes and text block nodes whose similarity vector scores exceed a threshold in the top K most relevant triples are selected as the initial nodes of the subgraph PPR algorithm, such as... Figure 2 As shown; in the retrieval mother graph, find the subgraph that is adjacent to the initial node k (k=1) of the subgraph PPR algorithm as the subgraph related to the retrieval question.
[0085] Specifically, the subgraph extraction process involves parsing the query semantics using a large model and mapping it to a knowledge graph structure, forming an initial set of nodes. This process first leverages the semantic understanding capabilities of the large model to identify the core entities and relationships in the query, transforming them into structured representations. Then, vector similarity calculations are used to locate relevant nodes in the knowledge base, ensuring that the subgraph covers the core semantic scope of the query. Finally, the graph structure is expanded to obtain a fully connected subgraph, providing a foundation for subsequent retrieval.
[0086] Step S103: In the subgraph, the random walk algorithm PPR is used to obtain a first candidate document set by calculating the score of each node in the subgraph; according to the relevance between the user-input retrieval question and the first candidate document set, the first candidate document set is sorted by the Rerank model.
[0087] The random walk algorithm (PPR) is used to walk through the subgraph and calculate the score of each node. After the walk terminates, the scores of each node are sorted to obtain the first candidate document set, which is the preliminary search result.
[0088] Specifically, the PPR (Personalized PageRank) algorithm:
[0089]
[0090] in Let A be the initial node weights, A be the adjacency matrix, D be the degree diagonal matrix, D_{uu} = deg(u), α control the restart probability in the random walk, and M = D -1 A represents the transition probability matrix.
[0091] A random walk process is performed on the subgraph, iteratively calculating node importance scores. The algorithm uses query-relevant nodes as the initial set and propagates weights across the subgraph structure using a transition probability matrix. During the walk, the connection strength and semantic relevance between nodes are considered, eventually converging to obtain a stable probability distribution for each node. Text block nodes are sorted according to their final scores to form preliminary search results, ensuring coverage of multi-faceted semantic information from the query.
[0092] Furthermore, the specific steps for sorting the first candidate document set are as follows:
[0093] The user-input search question is concatenated with candidate documents and used as input to a bidirectional encoder structure.
[0094] The bidirectional encoder structure calculates the semantic interaction information of the input through multiple layers of Transformer encoding and outputs the corresponding relevance score;
[0095] The first candidate document set is sorted according to the relevance score, and the sorting result is used as the re-sorting result.
[0096] Specifically, the re-ranking module employs a deep semantic matching model to refine the initial retrieval results. The model captures cross-attention features between the query and the document through a Transformer encoder, learning fine-grained semantic interaction patterns. The output layer generates relevance scores through linear transformations and activation functions, reflecting the semantic matching degree between candidate documents and the query. The final ranking results comprehensively consider semantic relevance and structural importance, improving the accuracy and readability of the retrieval results.
[0097] Overall, the retrieval process narrows the search space through subgraph extraction, captures structural information using graph algorithms, and finally optimizes the quality of results through a reordering model, forming an efficient multi-level retrieval pipeline.
[0098] Step S104: Using a pre-trained lightweight filter, the first candidate document set with an output tag of 0 is filtered according to the relevance between the user-input retrieval question and the first candidate document to obtain the second candidate document set.
[0099] Furthermore, the training steps for the lightweight filter are as follows:
[0100] A certain amount of data retrieved by this model SBR-RAG is collected as training data. Due to the existence of the reranker, the data at the head is either correct data or bad case.
[0101] The teacher model is used to query the training data and determine its relevance to the first candidate document, generating 0 / 1 labels;
[0102] The student model is trained using hard distillation with (Q, D, Label) to learn the relevance judgment ability of the teacher model.
[0103] Using a trained lightweight student model to differentiate the relevance of search results, irrelevant documents are filtered out, reducing the illusion of a large model and improving the F1 score.
[0104] Furthermore, the specific steps for generating tags are as follows:
[0105] The relevance of the input teacher model is determined by comparing the query and candidate documents;
[0106] The teacher's large model outputs binary labels, where a label of 0 indicates no relevance and a label of 1 indicates relevance, resulting in a training dataset consisting of (Q, D, Label).
[0107] Specifically, the teacher model annotation process leverages the powerful semantic understanding capabilities of large language models to perform refined relevance evaluation on query-document pairs. This process first concatenates the query and candidate documents into a specific format of prompts, which are then input into the teacher model. Based on its deep understanding of semantic relevance, the teacher model outputs a binary classification result. This large-model-based annotation method captures fine-grained semantic associations, providing high-quality supervision signals for subsequent student model training, such as... Figure 3 As shown.
[0108] Furthermore, the student model is hard-distilled using (Q, D, Label), including:
[0109] The (Q, D, Label) pairs are fed into the lightweight student model as input.
[0110] The lightweight student model learns the mapping relationship of the teacher model on the difference in relevance.
[0111] The student model is fitted to the discriminative ability of the teacher model through hard distillation.
[0112] Furthermore, the distillation training process employs a hard-label learning strategy, enabling the student model to directly learn the binary classification decision boundary of the teacher model. During training, the student model gradually approximates the discriminative performance of the teacher model by minimizing the cross-entropy loss between the predicted results and the teacher labels. This distillation method ensures that the student model maintains high discriminative accuracy while possessing inference efficiency far exceeding that of the teacher model, making it suitable for practical deployment in large-scale retrieval scenarios.
[0113] Furthermore, the specific steps for applying lightweight filters are as follows:
[0114] The retrieved candidate documents are input one by one into the student model to determine their relevance to the query;
[0115] The student model outputs relevance labels and filters out documents that are determined to be irrelevant.
[0116] Ultimately, only the relevant documents identified are retained as external knowledge input for subsequent LLM generation.
[0117] Specifically, in the lightweight filter application stage, the trained student model is integrated into the retrieval pipeline to perform secondary filtering on the initial retrieval results. Each candidate document and query constitute an input pair, and the student model outputs a binary classification result (relevant / irrelevant). Based on the classification result, the system filters out low-relevance documents, retaining only high-confidence relevant documents as knowledge input for the larger model. This filtering mechanism effectively reduces the interference of noisy knowledge on the generation of the larger model, significantly reduces the illusion phenomenon, and improves question-answering accuracy.
[0118] This lightweight filter training framework utilizes a teacher-student distillation paradigm to transfer relevance discrimination capabilities from a powerful but cumbersome teacher model to a lightweight and efficient student model. The resulting filter significantly improves the quality of retrieval results with almost no increase in system latency, providing cleaner and more relevant knowledge input for larger models, thereby comprehensively enhancing the performance and reliability of the question-answering system.
[0119] Step S105: The second candidate document set and the user-input search question are used as prompt words and input into the large model to assist the large model in answering the user-input search question.
[0120] Furthermore, it also includes an evaluation of the effectiveness of the knowledge retrieval filtering framework. The evaluation specifically involves assessing the effectiveness of retrieval and large model responses on multiple datasets using two basic evaluation metrics.
[0121] Based on the same inventive concept, this invention also provides a knowledge retrieval and filtering framework for assisting large-scale question answering models, such as... Figure 4 As shown, the entire framework consists of three stages: mother graph construction, subgraph retrieval, and retrieval filtering.
[0122] The mother graph construction process includes three stages: entity extraction, triple extraction, and graph construction. It primarily involves using a large model to extract knowledge from the current documents in the knowledge base, then leveraging the similarity relationships between points to form edges, thus constructing a more closely related graph, which serves as the foundation for subsequent retrieval steps.
[0123] The subgraph retrieval process utilizes the similarity between the query and each relation to select the corresponding subgraph with the closest relation in the parent graph. The personalized RageRank (PPR) algorithm is used to perform retrieval in this subgraph, and the retrieval results are obtained according to the score. The results are then further reordered to obtain the final retrieval results.
[0124] The retrieval filtering process first uses a portion of the retrieval results to learn how the lightweight model makes filtering decisions from the larger model. Based on this, the trained lightweight model is used to filter the retrieval results, minimizing the illusion of a large model.
[0125] In a specific embodiment, the specific content of the feasibility verification of the solution is as follows:
[0126] The Qwen3-8B library was used for entity extraction, triple extraction, and the final question-answering LLM. bge-large-en was used as the embedding model, bge-reranker-v2-m3 as the reranking model, and bert-base-uncased was fine-tuned as the filter. In the experimental phase, after re-ranking 100 documents, the top 5 documents were placed into the filter. Finally, the filtered results were merged with the query to form a proposal, which was then input into the LLM.
[0127] Table 1 shows the retrieval results compared to other models.
[0128] Retrive Musique 2WikiMultihopQA HotpotQA Avg Recall@2 Recall@5 Recall@2 Recall@5 Recall@2 Recall@5 Recall@2 Recall@5 BM25 0.2028 0.2743 0.2895 0.4070 0.4530 0.6290 0.3151 0.4368 BGE-M3 0.4038 0.5267 0.6410 0.7180 0.7150 0.8640 0.5866 0.7029 HippoRAG2 0.3323 0.4802 0.4815 0.5965 0.6750 0.8520 0.4963 0.6429 Ours 0.4405 0.5658 0.6420 <![CDATA[ 0.7065 ]]> 0.8370 0.9220 0.6398 0.7314
[0129] Table 1 shows the retrieval recall of the knowledge retrieval filtering framework compared with other baseline models. The results show that using subgraphs for PPR algorithm retrieval combined with the Reranker mechanism greatly improves the retrieval recall. By limiting the retrieval scope of the graph and performing two-stage reranking, the knowledge retrieval filtering framework can more accurately find the relevant text corresponding to the query.
[0130] Table 2 shows the response performance of the large model compared with other models.
[0131] QA Musique 2WikiMultihopQA HotpotQA Avg BM25 0.2482 0.4502 0.5820 0.4268 BGE-M3 0.2636 0.5839 0.7095 0.5190 HippoRAG2 0.3588 0.4211 0.6637 0.4812 Ours 0.4380 0.4971 0.7219 0.5523
[0132] Table 2 shows the LLM response performance of the knowledge retrieval filtering framework compared with other baseline models. The results show that in most cases, the LLM response in the knowledge retrieval filtering framework is closer to the gold_answer, that is, the articles input to the LLM are more accurate and do not introduce too much noise content, demonstrating the superiority of the retrieval strategy and document filtering strategy.
[0133] Table 3 shows the search efficiency compared to HippoRAG2.
[0134] Retrival Efficiency Musique 2WikiMultihopQA HotpotQA Avg HippoRAG2 2.32s / piece 2.43s / piece 1.88s / piece 2.21s / piece Ours 2.00s / piece 1.39s / piece 1.16s / piece 1.52s / piece
[0135] Table 3 shows the comparison results of the retrieval efficiency between the knowledge retrieval filtering framework and HippoRAG2. By simplifying the process of obtaining relevant triples and using subgraphs instead of parent graphs for document retrieval, the knowledge retrieval filtering framework improves the retrieval efficiency by 13% to 42% compared with the classic Graph-RAG method HippoRAG2, which greatly reduces the time required for retrieval.
[0136] After testing with different data, this knowledge retrieval and filtering framework consistently outperformed other mainstream models, validating the feasibility of the aforementioned design.
[0137] Based on the same inventive concept, this invention also provides a knowledge retrieval and filtering system for assisting large-scale question answering models, such as... Figure 5 As shown, it includes:
[0138] The retrieval master graph construction module 510 constructs a retrieval master graph of entity relationships based on the knowledge base;
[0139] The subgraph partitioning module 520 is used to receive the retrieval question input by the user and partition the retrieval master graph into subgraphs related to the retrieval question.
[0140] The first candidate document set acquisition module 530 is used to obtain the first candidate document set by using the random walk algorithm PPR in the subgraph and calculating the score of each node in the subgraph; and to sort the first candidate document set by the reordering model Rerank according to the relevance between the user input retrieval question and the first candidate document set.
[0141] The second candidate document set acquisition module 540 is used to use a pre-trained lightweight filter to filter the first candidate document set with an output label of 0 based on the relevance between the user-input retrieval question and the first candidate document, thereby obtaining the second candidate document set.
[0142] The question-answering module 550 is used to input the second candidate document set and the user-inputted search question as prompt words into the large model to assist the large model in completing the question-answering of the user-inputted search question.
[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0144] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.
[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.
Claims
1. A knowledge retrieval and filtering method for assisting large-scale question answering models, characterized in that, include: Based on the knowledge base, construct a retrieval master graph of entity relationships; Receive the user's input search question, and divide the master search graph into subgraphs related to the search question; The random walk algorithm PPR is used in the subgraph to obtain a first candidate document set by calculating the score of each node in the subgraph; the first candidate document set is ranked by the reranking model Rerank according to the relevance between the user input retrieval question and the first candidate document set. Using a pre-trained lightweight filter, the first candidate document set with an output label of 0 is filtered according to the relevance between the user-input retrieval question and the first candidate document set to obtain the second candidate document set; The second candidate document set and the user-input search question are used as prompt words and input into the large model to assist the large model in answering the user-input search question.
2. The method according to claim 1, characterized in that, Based on the knowledge base, a master graph for entity relationships is constructed, including: Extract entities from knowledge base documents using a large model; Based on the entity, extract the triplet relation containing the entity's corresponding entity; The entity is treated as a single node, and the relationship edge is used as the edge connecting two entities. The document and the entities it contains are connected by a containing edge. The KNN algorithm is used to calculate the score and number of nodes for each node, and to identify similar nodes for each node. Each node is connected to similar nodes through similar edges to construct a master graph for entity relationship retrieval.
3. The method according to claim 2, characterized in that, The specific steps for obtaining similar edges are as follows: The vectors between the nodes are obtained by embedding the nodes; Use the coherent similarity to calculate the vector similarity between nodes and obtain the top K similar nodes corresponding to each node; The vector from each node to the first K similar nodes is the similar edge between each node and the similar nodes.
4. The method according to claim 1, characterized in that, The system receives a search question input by the user and divides the master search graph into sub-graphs related to the search question, including: Based on the user's input search question, extract the corresponding entity and triple relationship; Obtain the vector embedding of the triplet as the query vector; Calculate the similarity between the query vector and the triples in the knowledge base, and take the entity nodes and text block nodes whose similarity vector scores exceed the threshold from the top K most relevant triples as the initial nodes of the subgraph PPR algorithm. In the retrieval parent graph, find the subgraph that is adjacent to (k=1) the initial node k of the subgraph PPR algorithm as the subgraph related to the retrieval question.
5. The method according to claim 1, characterized in that, The Random Walk (PPR) algorithm is used in the subgraph to obtain a first candidate document set by calculating the score of each node in the subgraph, including: The random walk algorithm PPR is used to walk through the subgraph and calculate the score of each node; After the walk ends, the scores of each node are sorted to obtain the first candidate document set.
6. The method according to claim 1, characterized in that, Based on the relevance of the user-input search question to the first candidate document set, the first candidate document set is ranked using the Rerank model, including: The user-input search question is concatenated with candidate documents and used as input to a bidirectional encoder structure. The bidirectional encoder structure calculates the semantic interaction information of the input through multiple layers of Transformer encoding and outputs the corresponding relevance score; The first candidate document set is sorted according to the relevance score, and the sorting result is used as the re-sorting result.
7. The method according to claim 1, characterized in that, The training steps for a lightweight filter are as follows: Collect the retrieved data as training data; The teacher model is used to query the training data and determine its relevance to the first candidate document, generating 0 / 1 labels; The student model is trained using hard distillation with (Q, D, Label) to learn the relevance judgment ability of the teacher model. The trained lightweight student model is used to differentiate the relevance of the search results and filter out irrelevant documents.
8. The method according to claim 7, characterized in that, The specific steps for generating tags are as follows: The relevance of the query and candidate documents to the input teacher model is determined. The teacher's large model outputs binary labels, where a label of 0 indicates no relevance and a label of 1 indicates relevance, resulting in a training dataset consisting of (Q, D, Label).
9. The method according to claim 7, characterized in that, Hard distillation training of the student model using (Q, D, Label) includes: The (Q, D, Label) pairs are fed into the lightweight student model as input. The lightweight student model learns the mapping relationship of the teacher model on the difference in relevance. The student model is fitted to the discriminative ability of the teacher model through hard distillation.
10. A knowledge retrieval and filtering system for assisting large-scale question answering models, characterized in that, include: The retrieval master graph construction module, based on the knowledge base, constructs a retrieval master graph of entity relationships; The subgraph partitioning module is used to receive the search question input by the user and partition the search master graph into subgraphs related to the search question. The first candidate document set acquisition module is used to obtain the first candidate document set by using the random walk algorithm PPR in the subgraph and calculating the score of each node in the subgraph; and to sort the first candidate document set by the reordering model Rerank according to the relevance between the user input retrieval question and the first candidate document set. The second candidate document set acquisition module is used to use a pre-trained lightweight filter to filter the first candidate document set with an output label of 0 based on the relevance between the user-input retrieval question and the first candidate document set, thereby obtaining the second candidate document set. The question-answering module is used to input the second candidate document set and the user-inputted search question as prompt words into the large model to assist the large model in completing the question-answering of the user-inputted search question.