An intelligent archive question-answering system based on hierarchical topic forest and entity enhancement
Patent Information
- Application Number
- CN202610965530.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-30
- Publication Date
- 2026-10-23
AI Technical Summary
[0003]在现有技术中，存在以下问题：1、语义碎片化问题：档案文献（尤其是长篇历史纪事）中主题密度分布极不均匀
1、基于传统单轮密度聚类在处理主题密度分布不均的档案语料时，低密度主题被大量判定为噪声而丢弃，导致语料主题覆盖残缺；标准层次聚类生成的二叉树深度过大、分支单一，难以提供多层级语义导航的技术问题。本发明通过第一轮聚类后，对噪声文档执行第二轮较低阈值的聚类，有效回收稀疏主题；采用Ward层次聚类生成二叉树，基于合并距离的四分位距动态计算熔断阈值，切断异常合并；进一步通过比例阈值折叠语义相近的冗余中间节点，将二叉树压缩为多叉主题森林。
Smart Images
Figure CN122964209A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent retrieval and question-answering technology for digital archives, specifically an intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation. Background Technology
[0002] Currently, archival management is transforming from passive preservation to proactive knowledge service, with users expecting intelligent and integrated semantic retrieval and question answering of unstructured historical information in archival documents. Intelligent question answering for archives has become a research hotspot. In recent years, large-scale language models (LLMs) have been explored for application in archival knowledge services due to their powerful reasoning and text generation capabilities. To further address the illusion problem in practical applications of LLMs, retrieval-enhanced generation has become the mainstream technological paradigm. It improves the reliability of responses by introducing external knowledge bases to constrain the generation process.
[0003] The existing technologies suffer from the following problems: 1. Semantic fragmentation: The topic density distribution in archival documents (especially long historical chronicles) is extremely uneven. Existing methods generally adopt fixed-length text block segmentation strategies, which easily split a complete topic thread into multiple fragments or introduce irrelevant noise into the retrieval context, causing the system to be unable to form a coherent semantic understanding of narratives across chapters and documents. 2. Granularity mismatch in entity-level retrieval: Archival queries usually revolve around specific historical figures, places, policies, or events. The semantics of these fine-grained entities are not significant enough in general dense vector representations. Standard dense retrieval models struggle to achieve "entity-level" recall, easily leading to factual omissions or errors in the final generated answers. 3. Lack of a unified framework for archival scenarios: Although existing enhanced retrieval schemes can improve corpus organization, they fail to simultaneously solve the two coexisting problems of macro-level corpus structure organization (to deal with uneven topic density) and micro-level entity-aware retrieval (to deal with fine-grained question answering). They also lack an adaptive routing mechanism for different user intentions (precise factual question answering vs. thematic data exploration). Summary of the Invention
[0004] To address the problems existing in the prior art, this invention proposes an intelligent question-answering system for archives based on hierarchical topic forest and entity enhancement.
[0005] The technical solution of the present invention is as follows: This invention proposes an intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation, comprising: The hierarchical topic forest construction module preprocesses and vectorizes the original files to generate dense vectors representing text blocks; A two-round clustering method is used to process dense vectors. The first round of clustering generates topic clusters and a set of noisy documents. The second round of clustering recovers sparse but semantically coherent topic clusters from the set of noisy documents. The topic clusters from the two rounds are merged to obtain the final topic set. The c-TF-IDF algorithm is used to construct feature vectors for each topic cluster, and the semantic differences between topics are measured based on cosine distance. Ward hierarchical clustering is performed on all topic clusters to generate a binary tree. Anomaly merging is cut off through a dynamic circuit breaker mechanism to divide the binary tree into several subtrees. Each subtree is folded and compressed to reconstruct the binary tree into a multi-branch topic forest, and a natural language description is generated for each node. The multi-granularity hybrid retrieval module is used to receive user queries, perform dense retrieval, sparse retrieval and entity association retrieval in parallel, and merge and sort all retrieval results; The intent-aware question-answering module is used to identify the type of intent in a user's query and automatically select a question-answering strategy based on the intent type to generate the final answer.
[0006] In a preferred embodiment, the steps of preprocessing and vectorizing the original files specifically include: The original archive is segmented into a set of searchable text blocks; the article title and the first 500 words of the main text of each text block are selected as the embedding input, and the summary fragments are used to replace the full text; each text block is encoded into a dense vector using a pre-trained BGE-m3 embedding model.
[0007] As a preferred embodiment, the two-round clustering method is specifically as follows: First round of clustering: UMAP is used to reduce the original vector embedding from 1024 dimensions to 5 dimensions while preserving the global topological structure; then HDBSCAN density clustering is applied to obtain the first round of topic clusters and noisy document sets; Second round of clustering: HDBSCAN clustering is performed again on the noisy document set to recover sparse but semantically coherent topic clusters.
[0008] As a preferred embodiment, the dynamic circuit breaker mechanism is specifically as follows: The circuit breaker threshold is automatically calculated based on the interquartile range of the merge distance sequence using Ward hierarchical clustering. Abnormal merges exceeding the circuit breaker threshold are cut off, and the binary tree is divided into several independent binary subtrees.
[0009] In a preferred embodiment, the folding and compression step of folding and compressing each subtree specifically involves: Calculate the effective height span of each subtree, set a folding threshold, recursively traverse each internal node, and if the difference between its merge cost and that of its parent node is less than the folding threshold, then fold and absorb the node, and its child nodes are attached to the grandparent node.
[0010] In a preferred embodiment, the dense retrieval, sparse retrieval, and entity association retrieval are specifically as follows: Dense retrieval: Encode user queries into query vectors using the same BGE-m3 embedding model, calculate the cosine similarity between the query vector and each text block vector, and return the top text blocks with the highest similarity and their ranking. Sparse retrieval: The BM25 algorithm is used to calculate the lexical matching score between the query and the text blocks, and the top-scoring text blocks and their rankings are returned. Entity association retrieval is divided into two stages: offline construction and online retrieval. Offline build phase: Named entity recognition is performed on each document, at least organizational, policy, project, and geographic location entities are extracted, and an inverted index from entity to document is built; Online retrieval phase: Identify entities from user queries; for each identified entity, retrieve all documents containing that entity using an inverted index, filtering out generalized entities; add the retrieved documents to the candidate set and assign initial scores based on document frequency to obtain entity retrieval ranking.
[0011] In a preferred embodiment, the step of merging and sorting all search results specifically includes: The three ranking lists generated by dense retrieval, sparse retrieval, and entity association retrieval are merged using the inverse rank fusion formula; the candidate documents are then reordered from high to low according to the fusion score, and the final multi-granularity retrieval result list is output.
[0012] In a preferred embodiment, the intent types in the intent-aware question-answering module include precise factual question-answering and retrieval-based topic question-answering.
[0013] As a preferred embodiment, the question-answering strategy for precise fact-based question answering is specifically as follows: Extract evidence context from the top-ranked documents returned by the multi-granularity retrieval module, assemble the evidence context with the user query into a prompt template, call the large language model to generate a factual answer, and attach the source of the citation at the end of the answer.
[0014] As a preferred embodiment, the question-answering strategy of the retrieval-based topic question answering is specifically as follows: Extract titles, publication dates, abstracts, and key entities from the top-ranked documents returned by the multi-granularity retrieval module, and call the large language model to generate a structured topic summary including a topic overview, event timeline, main entity relationships, and document citation list.
[0015] The present invention has the following beneficial effects: 1. Traditional single-round density clustering suffers from several technical problems when dealing with archival corpora with uneven topic density distribution. Low-density topics are often discarded as noise, resulting in incomplete topic coverage. Standard hierarchical clustering also generates binary trees with excessive depth and limited branches, hindering multi-level semantic navigation. This invention addresses these issues by performing a second round of clustering with a lower threshold on noisy documents after the first round, effectively recovering sparse topics. It uses Ward hierarchical clustering to generate binary trees, dynamically calculating the circuit breaker threshold based on the interquartile range of the merge distance to cut off abnormal merges. Furthermore, it compresses the binary tree into a multi-branched topic forest by folding semantically similar redundant intermediate nodes using a proportional threshold.
[0016] 2. The technical problem of insufficient precise matching capability of a single dense retrieval for specific entities; reports related to the same entity are scattered in the semantic space, making effective aggregation difficult. This invention addresses this issue by parallel execution of dense vector retrieval (BGE-m3) and BM25 sparse retrieval, achieving complementarity through inverse rank fusion; offline construction of an inverted index from entity to document (focusing on four core entity types: institutions, projects, policies, and locations); and precise retrieval of related documents based on the identified entity names during online queries; and differentiated weighting of entity retrieval results: articles that are both matched by semantic retrieval and receive a reward, while articles that only match by entity receive a lower weight as a safety net.
[0017] 3. Based on the fact that user queries are divided into two categories—precise fact-based question answering and retrieval-based topic exploration—uniform processing can lead to redundant answers or unfocused information. This invention introduces a lightweight large language model as a router to automatically identify query intent and route queries accordingly: precise question answering generates concise factual answers with source annotations from a small number of highly relevant documents; retrieval-based question answering uses multi-path fusion recall to generate introductory summaries and article lists. Attached Figure Description
[0018] Figure 1 This is a system structure diagram of the present invention; Figure 2 Flowchart of the steps involved in building a hierarchical theme forest module; Figure 3 This is a flowchart illustrating the working steps of the multi-granularity hybrid retrieval module. Figure 4 This is a flowchart illustrating the workflow of the intent-aware knowledge question-answering module. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.
[0021] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0022] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.
[0023] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.
[0024] Example 1: See Figure 1 An intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation includes: The hierarchical topic forest construction module preprocesses and vectorizes the original files to generate dense vectors representing text blocks; A two-round clustering method is used to process dense vectors. The first round of clustering generates topic clusters and a set of noisy documents. The second round of clustering recovers sparse but semantically coherent topic clusters from the set of noisy documents. The topic clusters from the two rounds are merged to obtain the final topic set. The c-TF-IDF algorithm is used to construct feature vectors for each topic cluster, and the semantic differences between topics are measured based on cosine distance. Ward hierarchical clustering is performed on all topic clusters to generate a binary tree. Anomaly merging is cut off through a dynamic circuit breaker mechanism to divide the binary tree into several subtrees. Each subtree is folded and compressed to reconstruct the binary tree into a multi-branch topic forest, and a natural language description is generated for each node. In this embodiment, the specific working steps of the hierarchical topic forest construction module are as follows: like Figure 2 As shown, this module transforms the original archival corpus into a structured topic forest: Step A1: Document Text Preprocessing and Vectorization First, the archival document is segmented into a set C of searchable text blocks. Considering that news reports adopt an inverted pyramid structure, with core information concentrated in the lead paragraph, and taking into account the applicability of the method to very long texts, this system selects the article title and the first 500 words of the main text as the embedding input, replacing the full text with a summary fragment. Then, a pre-trained BGE-m3 embedding model is used to encode each text block into a dense vector, obtaining a vector representation of the core semantics of the text.
[0025] Step A2: Two rounds of clustering optimization to handle uneven topic density The archival data suffers from uneven distribution of thematic density. High-frequency reporting types (such as weather information, festival activities, and government announcements) are repeated extensively in the corpus, forming dense areas with highly concentrated semantics; while the number of reports on specific topics (such as new energy industry, urban planning, and foreign exchanges) is limited, presenting a localized sparse low-density distribution in the embedding space.
[0026] Traditional density-based clustering algorithms define cluster boundaries based on global density estimation. Dense regions significantly raise the density threshold, causing semantically relevant text points in low-density regions to fail to meet density requirements and be largely identified as noise points and discarded. However, these discarded noise points are not semantically isolated; they contain numerous reports on specific topics with real thematic value. Ignoring these points will result in incomplete thematic coverage of the corpus.
[0027] Therefore, this module employs a two-round clustering method: First round of clustering: UMAP is used to reduce the dimensionality of the original vector embedding from 1024 dimensions to 5 dimensions while preserving the global topological structure; then HDBSCAN density clustering is applied to obtain the first round of topic clusters T. (1) and the collection of noisy documents N (1) .
[0028] Second round of clustering: For the noisy document set N (1) Perform HDBSCAN clustering again (minimum cluster size set to 10) to recover the sparse but semantically coherent topic clusters T. (2) .
[0029] By merging the results of the two rounds, we obtain the final topic set T=T (1) ∪T (2) .
[0030] Step A3: Building a Hierarchical Theme Forest After obtaining the leaf topic clusters, this step organizes them into a semantically hierarchical topic forest.
[0031] First, the c-TF-IDF algorithm is used to construct a feature vector for each topic cluster. All documents within the cluster are concatenated into a "superdocument," and the category-weighted frequency of terms is calculated to obtain a high-dimensional sparse vector representing the semantic fingerprint of the topic. Based on this vector, cosine distance is used to measure the semantic difference between two topics—the smaller the distance, the more likely the topics belong to the same superordinate concept.
[0032] Secondly, using all leaf topics as initial nodes, Ward hierarchical clustering is performed. This method, at the cost of the increased variance within each merged cluster, selects the two clusters with the smallest cost for merging, and records the distance value of each merge to generate a complete binary tree. The merge distance monotonically increases along the sequence, with larger values indicating more significant semantic differences.
[0033] To avoid forcibly merging topics with significantly different semantics, this invention introduces a dynamic circuit breaker mechanism: the circuit breaker threshold δ is automatically calculated based on the interquartile range (IQR) of the merged distance sequence. prune =Q3+1.5×IQR (where Q3 represents the upper quartile of the merged sample sequence, and IQR is the interquartile range, i.e., the upper quartile Q3 minus the lower quartile Q1), cut off abnormal merges exceeding this threshold, and divide the binary tree into several independent binary subtrees.
[0034] Finally, folding compression is performed on each subtree. The merge sequence retained after the circuit breaker covers the minimum merge cost d. min To the circuit breaker threshold δ prune Calculate the effective height span Δ=δ within the distance range. prune d min Set the folding threshold δ flatten =α×Δ (α is 0.1). Recursively traverse each internal node: if the difference between its merge cost and that of its parent node is less than the folding threshold, it is considered to be at the same semantic level, and the node is folded and absorbed, with its child nodes being attached to the grandparent node. After this compression, the binary tree is reconstructed into a multi-branched topic forest, with reduced depth and increased branching, and each node representing a semantically coherent topic unit.
[0035] Step A4: Generating Natural Language Descriptions for Topic Nodes For each leaf node, representative keywords and document titles are sampled from its topic clusters, and the large language model is invoked to generate the node name and a brief description. For internal nodes, the names and descriptions of all its child nodes are recursively used as context, and the large language model is invoked to generate the name and description of that internal node.
[0036] Ultimately, each node stores: node ID, name, description, a list of original text blocks it contains, and a list of child nodes.
[0037] The multi-granularity hybrid retrieval module is used to receive user queries, perform dense retrieval, sparse retrieval and entity association retrieval in parallel, and merge and sort all retrieval results; In this embodiment, the working steps of the multi-granularity hybrid retrieval module are as follows: like Figure 3 As shown, the multi-granularity hybrid retrieval module receives user queries and executes three retrieval paths in parallel.
[0038] Step B1: Dense Search The BM25 algorithm is used to calculate the lexical matching score between the query and the text blocks. To improve time and context matching capabilities, metadata such as the document title and publication date are concatenated into the text block content during the indexing phase. The top k text blocks and their rankings are returned.
[0039] Step B2: Sparse Search The BM25 algorithm is used to calculate the lexical matching score between the query and the text blocks. To improve time and context matching capabilities, metadata such as the document title and publication date are concatenated into the text block content during the indexing phase. The top k text blocks and their rankings are returned.
[0040] Step B3: Entity Extraction and Association Retrieval Offline phase: Named entity recognition is performed on each document to extract entities such as organizations, policies, projects, and geographical locations, and an inverted index from entities to documents is built.
[0041] Online phase: Identify entities from user queries; for each identified entity, retrieve all documents containing that entity using an inverted index, filtering out generalized entities (such as names, dates, and words without distinctiveness); add the retrieved documents to the candidate set and assign an initial score based on document frequency to obtain the entity retrieval ranking.
[0042] Step B4: Multi-path retrieval fusion and ranking The three ranking lists R={dense, sparse, entity} generated in order of dense search, sparse search, and entity association search are merged using the inverse rank fusion formula:
[0043] Where R represents the set of three ranking lists, and r represents the specific search path; Document Rank in the r-th search path; The smoothing constant is set to 60 in this system.
[0044] The intent-aware question-answering module is used to identify the type of intent in a user's query and automatically select a question-answering strategy based on the intent type to generate the final answer.
[0045] In this embodiment, the working steps of the intent-aware question-answering module are as follows: like Figure 4 As shown, this module adaptively selects the question-and-answer strategy based on the type of intent of the user's query.
[0046] Step C1: Query intent route A lightweight large language model is used as a router to analyze user query text and identify query intent. The model outputs two types of query questions: 1. exact fact answer, 2. retrieval-based topic answer.
[0047] Precise fact-based question answering: Queries require specific factual answers, such as "When was a certain policy promulgated?".
[0048] Search-based topic-based Q&A: Queries require a summary of relevant topic information or a reading guide, such as "an introduction to the economic policies of a certain region in 2024".
[0049] Step C2: Precise Fact-Based Questions and Answers From the top N documents returned by the multi-granularity retrieval module, the title, publication date, and first paragraph of each document are extracted as evidence context. This evidence context is then combined with the user query to form a prompt template, which is used to generate a concise factual answer using a large language model. The source citation (document ID, title) is appended to the end of the answer to ensure traceability.
[0050] Step C3: Search-based topic question answering From the top M (M>N) documents returned by the multi-granularity retrieval module, extract the title, publication date, abstract, and key entities of each document. Use a large language model to generate a comprehensive topic summary, including: a topic overview, a timeline of related events, main entities and their relationships, and a list of document citations. The output is presented in structured text, facilitating a quick understanding of the overall picture of the topic within the archival corpus.
[0051] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. An intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation, characterized in that, include: The hierarchical topic forest construction module preprocesses and vectorizes the original files to generate dense vectors representing text blocks; A two-round clustering method is used to process dense vectors. The first round of clustering generates topic clusters and a set of noisy documents. The second round of clustering recovers sparse but semantically coherent topic clusters from the set of noisy documents. The topic clusters from the two rounds are merged to obtain the final topic set. The c-TF-IDF algorithm is used to construct feature vectors for each topic cluster, and the semantic differences between topics are measured based on cosine distance. Ward hierarchical clustering is performed on all topic clusters to generate binary trees. By using a dynamic circuit breaker mechanism to cut off abnormal merging, the binary tree is divided into several subtrees; Each subtree is folded and compressed to reconstruct the binary tree into a multi-branched topic forest, and a natural language description is generated for each node; The multi-granularity hybrid retrieval module is used to receive user queries, perform dense retrieval, sparse retrieval and entity association retrieval in parallel, and merge and sort all retrieval results; The intent-aware question-answering module is used to identify the type of intent in a user's query and automatically select a question-answering strategy based on the intent type to generate the final answer.
2. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The specific steps for preprocessing and vectorizing the original archives are as follows: The original archive is segmented into a set of searchable text blocks; the article title and the first 500 words of the main text of each text block are selected as the embedding input, and the summary fragments are used to replace the full text; each text block is encoded into a dense vector using a pre-trained BGE-m3 embedding model.
3. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The two-round clustering method is specifically as follows: First round of clustering: UMAP is used to reduce the original vector embedding from 1024 dimensions to 5 dimensions while preserving the global topological structure; then HDBSCAN density clustering is applied to obtain the first round of topic clusters and noisy document sets; Second round of clustering: HDBSCAN clustering is performed again on the noisy document set to recover sparse but semantically coherent topic clusters.
4. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The dynamic circuit breaker mechanism is specifically as follows: The circuit breaker threshold is automatically calculated based on the interquartile range of the merge distance sequence using Ward hierarchical clustering. Abnormal merges exceeding the circuit breaker threshold are cut off, and the binary tree is divided into several independent binary subtrees.
5. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The folding and compression step in the process of folding and compressing each subtree specifically involves: Calculate the effective height span of each subtree, set a folding threshold, recursively traverse each internal node, and if the difference between its merge cost and that of its parent node is less than the folding threshold, then fold and absorb the node, and its child nodes are attached to the grandparent node.
6. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The dense search, sparse search, and entity association search are specifically as follows: Dense retrieval: Encode user queries into query vectors using the same BGE-m3 embedding model, calculate the cosine similarity between the query vector and each text block vector, and return the top text blocks with the highest similarity and their ranking. Sparse retrieval: The BM25 algorithm is used to calculate the lexical matching score between the query and the text blocks, and the top-scoring text blocks and their rankings are returned. Entity association retrieval is divided into two stages: offline construction and online retrieval. Offline build phase: Named entity recognition is performed on each document, at least organizational, policy, project, and geographic location entities are extracted, and an inverted index from entity to document is built; Online retrieval phase: Identify entities from user queries; for each identified entity, retrieve all documents containing that entity using an inverted index, filtering out generalized entities; add the retrieved documents to the candidate set and assign initial scores based on document frequency to obtain entity retrieval ranking.
7. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The specific steps for merging and sorting all search results are as follows: The three ranking lists generated by dense retrieval, sparse retrieval, and entity association retrieval are merged using the inverse rank fusion formula; the candidate documents are then reordered from high to low according to the fusion score, and the final multi-granularity retrieval result list is output.
8. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 1, characterized in that, The intent types in the intent-aware question-answering module include exact fact-based question-answering and retrieval-based topic-based question-answering.
9. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 8, characterized in that, The specific question-answering strategy for precise fact-based questions is as follows: Extract evidence context from the top-ranked documents returned by the multi-granularity retrieval module, assemble the evidence context with the user query into a prompt template, call the large language model to generate a factual answer, and attach the source of the citation at the end of the answer.
10. The intelligent question-answering system for archives based on hierarchical topic forest and entity augmentation according to claim 8, characterized in that, The specific question-answering strategy for the retrieval-based topic question answering is as follows: Extract titles, publication dates, abstracts, and key entities from the top-ranked documents returned by the multi-granularity retrieval module, and call the large language model to generate a structured topic summary including a topic overview, event timeline, main entity relationships, and document citation list.