Archives credible question and answer method based on graph context retrieval and knowledge graph enhancement

Abstract

This invention relates to a reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement, belonging to the fields of computer software and artificial intelligence. This invention utilizes a multimodal large-scale language model (MLLM) to extract key entities and relationships between entities from historical archives, and automatically identifies low-confidence fields to generate tags requiring manual review. This reduces the workload of manual review while ensuring the accuracy of information extraction. Then, based on a given knowledge base, it retrieves previous related questions and answers according to the query question to enhance the context. Finally, it combines the information extracted from historical archives with the large-scale language model (MLLM) to generate a reliable answer. This invention significantly improves processing efficiency and knowledge service capabilities while maintaining the value of archival documents.

Description

Technical Field

[0001] This invention belongs to the field of computer software and artificial intelligence, and specifically relates to a reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement. Background Technology

[0002] Intelligentization is an important and cutting-edge issue in the in-depth application of archival technology. Against the backdrop of the digitalization and intelligentization of archives, the evidentiary value of archives remains their core value. This means that the digital question-and-answer system of archives must ensure the accuracy of archival data acquisition and question-and-answer generation. However, the intelligent processing and intelligent question-and-answer of historical archives generally suffer from problems such as multimodal mixing (printed / handwritten / graphical text coexisting), inconsistent image quality due to age leading to inaccurate information extraction, complex entity relationships requiring professional verification, and intelligent question-and-answer systems exhibiting illusions or inaccuracies and incompleteness.

[0003] Traditional processing methods primarily rely on manual cataloging, keyword retrieval, and template-based mechanical question answering, which suffers from low efficiency in digitizing historical archives and difficulties in ensuring the authenticity and accuracy of answers generated by question-and-answer systems. Specifically, existing OCR technology achieves less than 60% accuracy in recognizing mixed-format historical documents (such as vertical text from the Republican era) and faded handwriting; manual verification accounts for over 70% of the entire data processing time; traditional question-and-answer systems lack reasoning capabilities based on the complete context of historical events; and static knowledge graphs are ill-suited to new research findings.

[0004] To address these issues, this invention proposes an innovative solution: It significantly improves the accuracy of handwritten character recognition by employing a hybrid OCR engine of LayoutLMv3 and Tesseract; it utilizes MLLM confidence grading to construct an error marking system, reducing the workload of manual review; the constructed question-answer graph supports cross-temporal and spatial event correlation analysis; and it combines existing and open knowledge bases to enhance context. These methods significantly improve the intelligence level of historical archive processing and enhance the credibility of answers generated by intelligent question-and-answer systems in archives that primarily manage historical archives. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The technical problem this invention aims to solve is how to provide a reliable question-and-answer method for archives based on graph context retrieval and knowledge graph enhancement, in order to address the shortcomings of traditional processing methods, such as low efficiency in digitizing historical archives and difficulty in ensuring the authenticity and accuracy of answers generated by question-and-answer systems.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, this invention proposes a reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement. This method includes the following steps:

[0009] S1. The multimodal preprocessing module preprocesses historical archives, including:

[0010] (1) Enhance the images of historical documents;

[0011] (2) Use the hybrid OCR engine Tesseract+LayoutLMv3 to extract text;

[0012] (3) Intelligent region segmentation, detecting and segmenting handwritten regions, tables and illustrations;

[0013] S2, the trusted information extraction module extracts key entities, including:

[0014] (1) Multimodal Large Language Model (MLLM): Input OCR text and images, extract key entities;

[0015] (2) Error marking system: Automatically identifies low-confidence fields, generates marks that require manual review, and performs manual review;

[0016] S3, the trusted digest generation module retrieves the question and generates the answer, including:

[0017] (1) Graph-driven related question retrieval: Construct a question-question graph and find the most relevant questions in the existing knowledge base based on the query question;

[0018] (2) Knowledge graph-based context enhancement: Generate the context of the query question based on the most relevant question, and use an external knowledge base to expand the triples of the context to form an enhanced context;

[0019] (3) The historical archives processed and extracted by the trusted information extraction module are used as the training corpus of the large language model to generate a dedicated large language model. The query question and the enhanced context are used as input to generate the answer.

[0020] (III) Beneficial Effects

[0021] This invention proposes a reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement. Addressing the technical bottlenecks in the intelligent processing of historical archives, this invention offers an innovative solution. Through the synergistic innovation of three major technical systems—multimodal information processing, dynamic knowledge fusion, and credibility assurance—it achieves a leapfrog development of historical archives from digitization to intelligentization. While maintaining the value of archival evidence, the system significantly improves processing efficiency and knowledge service capabilities, providing a new technical solution reference for the development and utilization of archives.

[0022] 1) Multimodal collaborative analysis reduces labor costs.

[0023] By using a hybrid OCR engine in conjunction with an adaptive image enhancement algorithm, the system improves the recognition accuracy of complex handwriting, vertical text, and severely faded documents in historical archives. The integrated multimodal joint processing engine achieves simultaneous analysis of text, tables, and illustrations through an attention mechanism. Furthermore, the dynamic confidence grading system built on MLLM can automatically identify and label low-confidence content (fields with confidence <85%), greatly reducing the workload of manual verification.

[0024] 2) The question-and-answer generation is made credible, ensuring the value of archival documents.

[0025] It employs a dedicated LLM model with fine-tuned archival corpus, combined with question-question graph association retrieval and knowledge graph context enhancement architecture, and automatically includes source annotation for answers, ensuring that the value of archival evidence is fully preserved. Attached Figure Description

[0026] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0027] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0028] The purpose of this invention is to propose a system and method for generating credible question-and-answer responses for historical archives, combining a multimodal large-scale language model (MLLM), graph-based retrieval, and knowledge graph-enhanced context. This system features low training data requirements, multimodal recognition, high credibility, and low manual verification costs. The method utilizes a multimodal large-scale language model (MLLM) to extract key entities and relationships between entities from historical archives, and automatically identifies low-confidence fields to generate tags requiring manual review. This reduces the workload of manual review while ensuring the accuracy of information extraction. Based on a given knowledge base, it retrieves previous related questions and answers to enhance the context. Finally, it combines the information extracted from the historical archives with the large-scale language model (MLLM) to generate credible answers.

[0029] This invention proposes a reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement, such as... Figure 1 As shown, the method includes the following steps:

[0030] S1. The multimodal preprocessing module preprocesses historical archives:

[0031] (1) Enhance the image of historical documents (denoise removal, rotation correction);

[0032] (2) Use a hybrid OCR engine (Tesseract + LayoutLMv3) to extract text;

[0033] (3) Intelligent region segmentation: detect and segment handwritten areas, tables and illustrations.

[0034] S2, The trusted information extraction module extracts key entities:

[0035] (1) Multimodal Large Language Model (MLLM) (such as GPT-4V): Input OCR text and image, extract key entities (name, date, event);

[0036] (2) Error marking system: Automatically identify low confidence fields (such as fuzzy handwritten text), generate marks that require manual review and perform manual review.

[0037] S3. The trusted digest generation module retrieves the question and generates the answer:

[0038] (1) Graph-driven related question retrieval: Construct a question-question graph and find the most relevant questions in the existing knowledge base based on the query question;

[0039] (2) Knowledge graph-based context enhancement: Generate the context of the query question based on the most relevant question, and use an external knowledge base to expand the triples of the context to form an enhanced context;

[0040] (3) The historical archives processed and extracted by the trusted information extraction module are used as the training corpus of the archive's big language model to generate a dedicated big language model. The query question and the enhanced context are used as input to generate the answer.

[0041] Example 1:

[0042] 1.1 Multimodal Information Preprocessing

[0043] This module includes a three-stage processing flow: (1) In the image enhancement stage, a hybrid denoising algorithm of nonlocal mean and BM3D is used to optimize parameter configuration for yellowed paper documents and achieve automatic rotation correction with an accuracy of ±15° through text line detection; (2) The hybrid OCR engine integrates the dual-path recognition results of Tesseract (92% accuracy in printed text recognition) and LayoutLMv3 (complex layout cross-column reorganization) and uses a confidence weighting strategy for output fusion; (3) In intelligent region segmentation, a lightweight CNN model is deployed to realize handwritten stroke feature detection, an improved TableNet structure is used to complete table recognition, and an adaptive threshold segmentation and contour optimization algorithm are combined to achieve accurate extraction of illustrations. Each sub-module achieves adaptive parameter adjustment through transfer learning, providing multimodal standardized input for subsequent question-and-answer generation.

[0044] 1.2 Trusted Information Extraction Module

[0045] A multimodal large language model (MLLM) is used to construct an intelligent document processing pipeline. In the entity extraction stage, the system first performs structured parsing of the OCR text, preserving the original document's layout features, and simultaneously extracts image features through a visual Transformer model. A cross-modal attention mechanism is employed to achieve the alignment and fusion of text and visual features, and a high-precision recognition of three types of entities—names, dates, and technical terms—is achieved based on an improved BiLSTM-CRF model.

[0046] The error marking system employs a multi-level quality assessment mechanism: Differential confidence thresholds are set at the field level (0.9 for names / 0.85 for dates / 0.8 for technical terms); for handwritten content, readability is assessed by combining stroke continuity analysis and image blur detection; finally, a document-level credibility score is generated through a dynamic weighted algorithm. The system automatically marks low-confidence content (such as blurry handwritten text) and outputs review tags containing specific error reasons and location information according to the XML standard format for manual review, significantly improving the efficiency of manual review.

[0047] 1.3 Credible Question Answer Generation

[0048] 1.3.1 Preliminary Knowledge

[0049] Based on past archival Q&A records or expert-edited Q&A records, a Q&A knowledge base is formed. This knowledge base contains a series of questions and related, generally accepted answers, which are represented as follows: The query was made by... This indicates that the initial problem pool consists of... The process involves finding the most relevant questions and answers, along with enhanced context, from a question pool. Finally, the query question and enhanced context are used as input to generate the final answer. The relevant steps will be described in detail below.

[0050] 1.3.2 Contextual Retrieval

[0051] The purpose of the context retriever is to retrieve data based on the current query question. This module finds related questions in the knowledge base. It consists of two parts: the first is a "question-question graph," represented by a QQ icon; the second is the most relevant questions retrieved.

[0052] (1) Construction of QQ images

[0053] The purpose of building QQ images is to start from the initial question pool. Find the question related to the current query The most relevant question. We will represent QQ images as... , where nodes It's a problem, the side It is calculated based on the cosine similarity between two questions. We only adopt an edge when the similarity score exceeds a certain threshold. The value of QQ graphs. The advantage of QQ graph construction is that it can help identify semantically similar problems based on the structural properties of the graph.

[0054] (2) Derive the most relevant question

[0055] For the given current query problem Based on the constructed QQ image We will investigate the problem. As a node, measurement and The similarity score of all nodes in the query is used to determine the similarity score. If the similarity score exceeds a certain threshold, then the query is considered. and Edges are formed between corresponding nodes. Therefore, we believe that from the problem From the perspective of the diagram Problems with high node center scores can be classified as problems related to... The most relevant questions (nodes). Using the PangRank algorithm, customized rankings can be obtained based on prior information. This method allows us to obtain the graph. All nodes (except for the problem) The PangRank score. Then, for the query node... We choose the most relevant These questions form a set of questions. .

[0056] 1.3.3 Knowledge Graph-Based Context Enhancement

[0057] Based on the method described above, we can ultimately obtain the set of most relevant questions. and answer and treat it as a query question In the context, we use This indicates that, in actual large-scale model usage, we found that even with context provided... Large models often fall short in generating answers to open-ended questions. Therefore, this module will use knowledge graph methods to integrate context. Enhancement is performed. This module consists of two parts: the first part is entity recognition and triple formation; the second part is the implementation of enhanced context.

[0058] (1) Entity recognition and triple formation

[0059] At this stage, we first need to identify the context. All important information (entities) in the database. For the identification of important information, we use LLM and REBEL from... Obtain the initial triplet By sending simple prompts and context. Extract relation triples from the LLM, and then use REBEL to... The pairs are passed as input to their internal functions to obtain triples, ultimately forming a set. Next, we will use external knowledge bases, such as Wikipedia and Baidu Baike, to obtain the collection. For each entity, a hop neighbor and its relationships are then used to form a new triple. and only retain Head and tail entities in the original context The triplet exists in the array. Then the final initial triplet... and the new triplet Together they form a new extended set of triples. .

[0060] (2) Enhanced context generation

[0061] In the new extended set of triples In the middle, construct a collection of sub-contexts. It takes the form of a sequence of sentences, which are composed by placing the head entity, relation, and tail entity in sequence, and finally connecting the beginning and end contexts. and subcontext To build an enhanced context .

[0062] 1.3.4 Answer Generation

[0063] In this module, we use the historical archive data processed by the trusted information extraction module as training data for the large-scale model. Based on this, we train a dedicated large-scale language model for the archive, and then apply it to query questions. and enhanced context As input, obtain the generated answer. Furthermore, the automatic source attribution function for the answers ensures that the value of archival documents is fully preserved.

[0064] This invention addresses the technical bottlenecks in the intelligent processing of historical archives by proposing an innovative solution. Through the synergistic innovation of three major technical systems—multimodal information processing, dynamic knowledge fusion, and credibility assurance—it achieves a leapfrog development of historical archives from digitization to intelligent processing. While maintaining the value of archival evidence, the system significantly improves processing efficiency and knowledge service capabilities, providing a new technical solution reference for the development and utilization of archives.

[0065] 1) Multimodal collaborative analysis reduces labor costs.

[0066] By using a hybrid OCR engine in conjunction with an adaptive image enhancement algorithm, the system improves the recognition accuracy of complex handwriting, vertical text, and severely faded documents in historical archives. The integrated multimodal joint processing engine achieves simultaneous analysis of text, tables, and illustrations through an attention mechanism. Furthermore, the dynamic confidence grading system built on MLLM can automatically identify and label low-confidence content (fields with confidence <85%), greatly reducing the workload of manual verification.

[0067] 2) The question-and-answer generation is made credible, ensuring the value of archival documents.

[0068] It employs a dedicated LLM model with fine-tuned archival corpus, combined with question-question graph association retrieval and knowledge graph context enhancement architecture, and automatically includes source annotation for answers, ensuring that the value of archival evidence is fully preserved.

[0069] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement, characterized in that, The method includes the following steps: S1. The multimodal preprocessing module preprocesses historical archives, including: (1) Enhance the images of historical documents; (2) Use the hybrid OCR engine Tesseract+LayoutLMv3 to extract text; (3) Intelligent region segmentation, detecting and segmenting handwritten regions, tables and illustrations; S2. The trusted information extraction module extracts key entities, including: (1) Multimodal Large Language Model (MLLM): Input OCR text and images, extract key entities; (2) Error marking system: Automatically identifies low-confidence fields, generates marks that require manual review, and performs manual review; S3, the trusted digest generation module retrieves the question and generates the answer, including: (1) Graph-driven related question retrieval: Construct a question-question graph and find the most relevant questions in the existing knowledge base based on the query question; (2) Knowledge graph-based context enhancement: Generate the context of the query question based on the most relevant question, and use an external knowledge base to expand the triples of the context to form an enhanced context; (3) The historical archives processed and extracted by the trusted information extraction module are used as the training corpus of the large language model to generate a dedicated large language model. The query question and the enhanced context are used as input to generate the answer; in, In S3, the set of most relevant questions is obtained. and answer Then, it was used as a query question. The context, using express; (1) Entity recognition and triple formation First, we need to identify the context. All entities in: using LLM and REBEL from Obtain the initial triplet By sending simple prompts and context Extract relation triples from the LLM, and then use REBEL to... The input is passed to an inner function to obtain triples, which are then used to form a set. ; Next, we will use an external knowledge base to obtain the collection. For each entity, a hop neighbor and its relationships are then used to form a new triple. and only retain Head and tail entities in the original context The initial triplet exists in the set; then the final initial triplet is... and the new triplet Together they form a new extended set of triples. ; (2) Enhanced context generation In the new extended set of triples In the middle, construct a collection of sub-contexts. It takes the form of a sequence of sentences, which are composed by placing the head entity, relation, and tail entity in sequence, and finally connecting the beginning and end contexts. and subcontext To build an enhanced context .

2. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 1, characterized in that, In step S1, the image enhancement stage employs a hybrid denoising algorithm combining nonlocal mean and BM3D, optimizes parameter configuration for yellowed paper documents, and achieves automatic rotation correction with an accuracy of ±15° through text line detection.

3. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 1, characterized in that, In S1, the hybrid OCR engine merges the dual-path recognition results of Tesseract and LayoutLMv3, and uses a confidence weighting strategy for output fusion.

4. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 1, characterized in that, In S1, during intelligent region segmentation, a lightweight CNN model is deployed to detect handwritten stroke features, an improved TableNet structure is used to complete table recognition, and an adaptive threshold segmentation and contour optimization algorithm are combined to achieve accurate extraction of illustrations.

5. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 1, characterized in that, In S2, a multimodal large language model (MLLM) is used to construct a document intelligent processing pipeline. In the entity extraction stage, the OCR text is first structured and parsed to retain the original document's layout features. At the same time, image features are extracted through a visual Transformer model. A cross-modal attention mechanism is used to achieve the alignment and fusion of text and visual features. Based on the improved BiLSTM-CRF model, high-precision recognition of three types of entities, namely names, dates, and technical terms, is achieved.

6. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 1, characterized in that, In S2, the error marking system adopts a multi-level quality assessment mechanism: at the field level, differentiated confidence thresholds are set; for handwritten content, readability is assessed by combining stroke continuity analysis and image blur detection; finally, a document-level credibility score is generated through a dynamic weighted algorithm; low-confidence content is automatically marked, and a review mark containing specific error reasons and location information is output according to the XML standard format for manual review.

7. The document-based trusted question-answering method based on graph context retrieval and knowledge graph enhancement as described in any one of claims 1-6, characterized in that, In step S3, a question-and-answer knowledge base is formed based on previous archival question-and-answer records or expert-edited question-and-answer records. This knowledge base contains a series of questions and related generally accepted answers, which are represented as follows: The query question is from This indicates that the initial problem pool consists of... The process involves finding the most relevant questions and answers, along with enhanced context, from a question pool, and then using the queried question and enhanced context as input to generate the final answer.

8. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 7, characterized in that, In S3, based on the current query question Find related questions that exist in the knowledge base; (1) Problem-problem diagram, i.e., the construction of QQ diagram. The purpose of building QQ images is to start from the initial question pool. Find the question related to the current query The most relevant questions; Representing QQ images as , where nodes It's a problem, the side It is calculated based on the cosine similarity between two questions, and edges are only adopted when the similarity score exceeds a certain threshold. The value; (2) Derive the most relevant question For the given current query problem Based on the constructed QQ image , will query the question As a node, measurement and The similarity score of all nodes in the query is used to determine the similarity score. If the similarity score exceeds a certain threshold, then the query is considered. and Edges are formed between corresponding nodes. So, from the question From the perspective of the diagram Problems with high node center scores are judged to be related to the problem. The most relevant question; using the PangRank algorithm, a customized ranking is obtained based on prior information, resulting in a graph. Besides the problem The PangRank scores of all external nodes; then for the query node Choose the most relevant These questions form a set of questions. .

9. The reliable question-answering method for archives based on graph context retrieval and knowledge graph enhancement as described in claim 8, characterized in that, In step S3, the historical archive data processed by the trusted information extraction module is used as training data for the large model. Based on this, a dedicated large language model for the archives is trained, and then the query question is... and enhanced context As input, obtain the generated answer. Furthermore, the automatic source attribution function for the answers ensures that the value of archival documents is fully preserved.

Images