An agent collaborative application method based on a multi-modal historical corpus

Abstract

This application provides a method for intelligent agent collaborative application based on multimodal historical corpora, belonging to the interdisciplinary field of historical corpus processing and intelligent question answering. Addressing the pain points of complex formatting, numerous variant characters, and easily conflicting historical facts in historical corpora, the method employs the ViT-GCN architecture to achieve format-aware parsing while preserving original coordinates; it constructs a dictionary of variant character equivalence classes and a conflict-aware governance mechanism, and uses an improved quantitative scoring formula to select high-quality corpora; it utilizes a thought chain architecture to decompose the temporal logic of complex question-and-answer processes; and it introduces a spatiotemporal gating unit in the question-and-answer generation stage to effectively suppress the model's illusion of crossing dynasties. This invention achieves precise end-to-end tracing through physical anchors and hash pointers, which can improve the professionalism, accuracy, and credibility of historical knowledge services.

Description

Technical Field

[0001] This invention belongs to the field of historical corpus processing and intelligent question answering technology. More specifically, it relates to an intelligent agent collaborative application method based on multimodal historical corpus, and particularly to a technology that uses layout perception parsing, thought chain collaborative scheduling and spatiotemporal gating constraints to achieve accurate question answering and full-link tracing of historical knowledge. Background Technology

[0002] With the rapid development of digital technology, massive amounts of multimodal heterogeneous historical corpora (including text-based primary data and image-based secondary data) have become core resources for cultural heritage preservation. However, in practical applications, existing historical corpus processing and question-answering systems still face the following insurmountable technical challenges:

[0003] 1. Loss of physical information in multimodal parsing: Existing OCR technologies focus primarily on character recognition, lacking spatial logic awareness of complex ancient text formats (such as vertical layout, interlinear annotations, and commentaries). More importantly, existing technologies lose the physical locational relationship between the text and the original image during the parsing process. This results in subsequent question-and-answer results that can only achieve "textual citation," failing to achieve "pixel-level" tracing of the original image's origin, and thus failing to meet the rigorous requirements of historical research.

[0004] 2. Lack of historical logic constraints in corpus governance: Existing corpus cleaning mechanisms mostly adopt general natural language processing methods, lacking specific governance for historical variant character evolution and conflicts with historical logic. Existing quality assessments are mostly qualitative analyses, lacking a quantitative evaluation system that can integrate sentence segmentation confidence, variant character weights, and historical conflict rates, resulting in "dirty data" polluting the knowledge base and inducing model illusions.

[0005] 3. The "spatiotemporal illusion" is severe in complex question answering: Existing historical question answering systems are mostly based on the general retrieval augmented generation (RAG) architecture. Because large models are insensitive to spatiotemporal dimensions, they are prone to the illusion of "spatiotemporal inversion" (such as applying later systems to earlier dynasties) when dealing with cross-dynastic questions. Existing recall algorithms only focus on semantic similarity and lack a correlation constraint mechanism based on spatiotemporal span decay and knowledge graph topological paths.

[0006] 4. Lack of a collaborative scheduling mechanism with reasoning capabilities: The functional modules (parsing, governance, retrieval, and generation) of the existing system are fragmented, mostly employing hard-coded linear pipelines, lacking a central scheduling mechanism with CoT (Co-chain of Reasoning) capabilities. When faced with complex tasks spanning different dynasties or domains, the system cannot dynamically adjust the execution chain according to the task difficulty, resulting in a disconnect between processing efficiency and professional requirements.

[0007] 5. Unreliability of the chain of evidence: Due to the lack of a one-way, irreversible source index, the question-and-answer results generated by existing systems prevent users from quickly and intuitively verifying the specific location of each viewpoint in the original scanned ancient texts. This "broken chain of evidence" problem greatly limits the application of AI technology in serious historical and literary research.

[0008] To address the aforementioned challenges, there is an urgent need for an agent-based collaborative method that integrates deep visual perception, spatiotemporal logical constraints, and physical evidence anchoring to solve core problems such as low utilization of historical corpora, question-and-answer distortion, and lack of traceability. This invention employs an integrated technical approach of "multimodal normalization parsing + formula-driven quality control + agent-based collaborative scheduling + full-link traceability," adapting to the specific characteristics of historical corpora and providing reliable support for efficient utilization of historical corpora and intelligent question answering. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an intelligent agent collaborative application method based on multimodal historical corpora. This method enables the normalization, precise governance, and structured modeling of multimodal historical corpora. Through the collaborative scheduling of professional domain intelligent agents, it completes question-and-answer generation and full-link tracing, constructs a closed-loop historical question-and-answer application system, improves the utilization efficiency of historical corpora and the accuracy and traceability of knowledge questions and answers, solves many pain points of existing technologies, and provides efficient and reliable technical support for historical research, cultural inheritance, and education.

[0010] To achieve the above objectives, embodiments of the present invention provide a method for intelligent agent collaborative application based on multimodal historical corpus, comprising the following steps:

[0011] Step 1: Acquisition of multi-source, multimodal historical corpus and preliminary feature registration.

[0012] In this step, the process of acquiring the first data (text-based) and the second data (image-based) is essentially the collection of multi-source heterogeneous historical resources. Specific acquisition methods include: calling historical corpus database interfaces, user-uploaded data, and authorized integration with third-party historical data platforms.

[0013] In addition, preliminary format recognition and metadata registration are performed on the collected data. Besides recording the source, era, type, and storage path of the corpus, the unique registration process of this invention also includes reserving a globally unique corpus index ID for subsequent "full-link traceability" and classifying and storing the second data according to image features (such as: vertical woodblock prints, handwritten letters, and rubbings of inscriptions on metal and stone), so as to facilitate the subsequent use of the corresponding format computing power allocation strategy by the first model.

[0014] Step 2: Layout-aware parsing and normalization based on ViT-GCN (Vision Transformer-Graph Convolution Network).

[0015] The core of this step is to solve the problem of heterogeneous formats in multimodal corpora.

[0016] (1) Multi-format preprocessing: The first data is encoded uniformly (converted to UTF-8) and formatted; the second data is image preprocessed and the text area is located by using the page segmentation algorithm.

[0017] (2) First Model Processing (Layout-Aware Ancient Book Visual Transformation Model): The preprocessed data is input into the first model. This model adopts a ViT (Vision Transformer) combined with a GCN (Graph Convolutional Network) architecture:

[0018] Physical anchor point generation: While recognizing characters, the first model assigns a globally unique sequence of physical coordinates (x, y, w, h) to each recognized character. This sequence serves as a physical anchor point, synchronously encapsulated with the text content, providing underlying support for pixel-level source tracing in step 8.

[0019] Spatial logic modeling: GCN is used to model the spatial logical relationships between titles, body text, parenthetical notes, and annotations on the page. For the unique vertical layout and two-column format of historical texts, the model automatically adjusts the Attention weight allocation matrix to ensure that the reading order and sentence segmentation logic conform to the rules of ancient texts.

[0020] Confidence triggering mechanism: When the model outputs characters, it includes a confidence score P. If P < 0.8, the built-in "semantic completion sub-model" is automatically triggered. This sub-model combines the historical context of a specific dynasty with commonly used expressions (such as the evolution path of official titles and geographical names) to perform contextual semantic correction, significantly improving the recognition accuracy of incomplete corpora.

[0021] (3) Normalization transformation: The two types of data are uniformly transformed into a standardized original historical text stream (first result), and variant characters and phonetic loan characters are initially uniformly labeled.

[0022] Step 3: High-quality corpus governance based on historical conflict perception and quantitative assessment.

[0023] This step achieves precise corpus cleaning through a dual-drive governance mechanism.

[0024] (1) Basic cleaning of the rule engine: Completes redundancy cleaning, simplified and traditional characters unification and preliminary conversion of variant characters based on the lookup table.

[0025] (2)Deep Optimization of the Second Model (Historical Fact Conflict-Aware Semantic Alignment Model):

[0026] Contrastive learning alignment: Introduce contrastive learning technology to align the features of different literature descriptions of the same historical event (e.g., official history and unofficial history, local chronicles).

[0027] Variant character equivalence class space: Establish a variant character equivalence class dictionary based on the Embedding space. This dictionary maps variant characters from different historical periods to the same clustering center in the vector space (e.g., making "崋" and "华" completely equivalent at the semantic representation level) to avoid semantic retrieval drift.

[0028] Logical consistency check: Calculate the conflict score by extracting the "person - time - place" triple. If the corpus record violates the basic spatio-temporal logic (e.g., a person takes an official position after their death), it is automatically marked as a suspected historical fact error point to assist in quality assessment.

[0029] (3)Quantitative Scoring and Hierarchical Screening: Build an evaluation system based on an exclusive quantitative scoring formula.

[0030] The quantitative scoring formula is as follows:

[0031]

[0032] In the formula, the specific definitions and calculation methods of each parameter are as follows:

[0033] ①Q is the quantitative score of the corpus quality, with a value range of [0, 100]. The higher the Q value, the better the corpus quality;

[0034] ②C is the content integrity index, with a value range of [0, 1]. The calculation method is "the number of effective semantic fragments in the corpus ÷ the total number of semantic fragments × the semantic fragment integrity". An effective semantic fragment refers to a fragment that contains core historical concepts and can convey complete historical information. The total number of semantic fragments refers to all fragments after the corpus is segmented. The semantic fragment integrity refers to the information integrity of the effective semantic fragment (e.g., no missing characters or words, logical coherence);

[0035] ③S is the text normativity index, with a value range of [0, 1]. Its calculation logic consists of "character integrity" and "structural confidence":

[0036]

[0037] Among them, is the number of garbled characters or format anomalies, is the adaptive risk weight based on the character frequency distribution of a specific dynasty; This is a sentence segmentation confidence score based on Conditional Random Field (CRF), used to verify the risk of grammatical breaks and the regularity of typesetting logic in text.

[0038] ④H is the historical accuracy index, output by the second model, and its calculation logic includes a logical conflict rate term. This study assesses the authenticity and reliability of the corpus in terms of its spatiotemporal logic by comparing it with authoritative historical databases.

[0039] ⑤T is a semantic relevance index that measures the temporal fit between semantic units;

[0040] ⑥ Weight setting: For the preset weighting coefficients, and It can be dynamically adjusted according to the application scenario of historical corpus, highlighting the weight of H (historical compliance) and T (semantic relevance), and is usually set to This ensures accurate corpus quality grading.

[0041] Based on the quantitative scoring Q value, corresponding quality level labels are generated: high-quality corpus (Q≥85), qualified corpus (70≤Q<85), and corpus to be optimized (Q<70). A preset scoring threshold (usually 70) is set, and corpus with a score not less than the preset threshold is selected as the second result (high-quality standardized historical corpus). Unqualified corpus with a score lower than the preset threshold is automatically returned to the cleaning stage. The cleaning rules are adjusted based on the error features fed back by the second model to achieve dynamic iterative optimization of the system.

[0042] Step 4: Construct a structured semantic knowledge base based on spatiotemporal constraints and physical anchor binding.

[0043] The core of this step is to transform high-quality standardized corpora into structured knowledge with temporal logic and traceable characteristics, providing underlying support for subsequent accurate retrieval and end-to-end tracing. Specific technical details are as follows:

[0044] (1) Minimum Semantic Unit Segmentation and Physical Anchor Inheritance: The second result is input into the third model. The third model performs text segmentation with independent and complete semantics as the minimum management granularity. Each segmented minimum semantic unit not only contains text information, but also automatically inherits and binds the original image physical coordinate sequence (x,y,w,h) generated in step 2. By establishing a mapping index of "semantic unit-physical coordinates", it is ensured that every piece of information in the knowledge base has pixel-level original evidence pointing to it.

[0045] (2) Historical Entity Extraction and Spatiotemporal Metadata Annotation: The entity extraction sub-model of the third model is used to assist in the identification of entities in semantic units through historical knowledge graphs (KG). In addition to people and place names, the spatiotemporal correlation attributes between entities are extracted (such as: timeline of official changes, evolution path of place names from ancient to modern times). At the same time, the corpus evidence quality score Qual(K) calculated in step 3 and its corresponding dynasty label are annotated to form a multi-dimensional spatiotemporal metadata matrix.

[0046] (3) Vector encoding and dual-database collaborative storage: Through the vector embedding sub-model of the third model, a semantic encoding algorithm with fine-tuning for Classical Chinese is adopted to generate high-dimensional semantic vectors. The system adopts a dual-database collaborative strategy: the vector database stores semantic vectors and physical anchor hash values. The hash values ​​are generated based on "original image ID + physical coordinates + semantic unit text summary". This strategy is used to realize fast retrieval based on the correlation formula R; the relational database stores entity topological relationships, metadata and original document indexes to realize structured management.

[0047] (4) Construction of a hierarchical spatiotemporal knowledge base: Based on the encoded semantic units and entity relationships, a hierarchical historical semantic knowledge base is constructed. This knowledge base is arranged with multi-dimensional indexes according to "domain-dynasty-event". The unique hierarchical structure of this invention supports the logical alignment of entities across dynasties, completes the construction of the retrieval index, and provides core knowledge support for the subsequent question-and-answer generation process based on spatiotemporal gating units.

[0048] Step 5: Decoupling core functions and encapsulating a standardized skill matrix based on an agent architecture.

[0049] The core of this step is to modularize the system's functions, providing standardized execution units for the autonomous collaborative scheduling and thought chain reasoning of intelligent agents. Specific technical details are as follows:

[0050] (1) Deep decoupling of core functions: Independent execution logic for functions such as corpus parsing, conflict awareness and governance, quality assessment, semantic structured modeling, spatiotemporal gating retrieval, question answer generation, and full-link hash tracing. By standardizing the interface protocol, hard-coded dependencies between modules are eliminated, ensuring that each Skill unit can be dynamically called by the fourth model.

[0051] (2) Skill Module Encapsulation and Parameter Definition: Decoupled functional units are encapsulated into independent Skill modules. Each module, in addition to input / output interfaces, must define adaptive operating parameters. These parameters should include at least:

[0052] Corpus parsing skill: Encapsulates layout awareness functionality based on ViT-GCN;

[0053] Cleaning and Evaluation Skill: Encapsulates governance functions including CRF sentence segmentation confidence verification;

[0054] Semantic modeling skill: Encapsulates physical anchor binding and entity extraction functions;

[0055] Spatiotemporal gating retrieval skill: corresponds to the execution of a relevance recall function that introduces a time decay factor;

[0056] Source tracing and verification Skill: This corresponds to the pixel-level reverse tracing function based on hash pointers;

[0057] Image Enhancement Skill: When the Skill matrix detects severe deficiencies in records about a specific dynasty in the corpus, the 'Image Enhancement Skill' is automatically triggered to perform high-resolution secondary analysis on the original secondary data.

[0058] (3) Standardization and State Machine Management Mechanism Formulation: Establish a unified skill registration and result feedback mechanism. This invention introduces a state machine management mechanism to standardize the intermediate state transmission format of each Skill module in the thought chain scheduling process, ensuring the consistency and integrity of the data link when different skill modules are in parallel collaboration or serial backtracking.

[0059] (4) Standardized Skill Matrix Integration and Agent Adaptation: All independent modules are integrated into a standardized skill matrix (fourth result). This matrix has "Agent understandability," supporting the fourth model to flexibly combine skills and plan paths according to the user's complex intentions (such as cross-dynasty comparison needs). The matrix supports dynamic expansion and reuse, significantly improving the system's flexibility and scalability when facing different vertical historical research scenarios.

[0060] Step 6: Construction of an intelligent agent autonomous collaboration hub based on the thought chain (CoT) logic.

[0061] This step is the core of enabling collaborative operation of intelligent agents. It uses the fourth model (an autonomous decision-making model based on a thought chain architecture) to logically decompose complex tasks and precisely schedule skill modules, adapting to complex historical question-and-answer requirements across dynasties and multiple dimensions. Specific technical details are as follows:

[0062] (1) Construction of the Collaborative Hub: Construct an autonomous collaborative hub for intelligent agents, which includes a fourth model, a demand analysis module, a path planning module based on DQN (deep reinforcement learning), and a feedback processing module. This hub is responsible for transforming unstructured user demands into a task execution graph with temporal logic.

[0063] (2) Demand Analysis and Chain of Thought Decomposition: The fourth model is used to identify the intent of external user needs. For demands that span multiple dynasties or have complex relationships (such as "the evolution of handicrafts in Jiangnan from the Tang and Song Dynasties to the Ming and Qing Dynasties"), the fourth model introduces the Chain of Thought (CoT) reasoning mechanism to decompose the total demand into multiple sub-tasks. Each sub-task is mapped to a standardized demand vector, clarifying the query intent, spatiotemporal constraints and historical source preferences at each stage.

[0064] (3) Skill Path Planning and Dynamic Scheduling: Based on the decomposed core elements of the requirements, the corresponding Skill modules are matched from the fourth result (Skill matrix). The fourth model uses a path planning algorithm to determine the optimal execution link. For example, for requirements spanning multiple dynasties, the system automatically arranges the process as "multi-stage parallel corpus parsing → temporal alignment and cleaning → hierarchical semantic modeling → relevance-weighted retrieval". At the same time, dynamic operating parameters for each module are set (such as scoring thresholds based on document density and relevance weight coefficients based on temporal span).

[0065] (4) Cooperative scheduling scheme generation and adaptive optimization: The fourth model completes the serial calling and parallel collaboration of multiple skill modules and arranges the autonomous execution process. If the knowledge density of the subtask feedback is lower than the preset threshold, the collaborative hub will trigger the backtracking mechanism to automatically adjust the preceding skill (such as improving the OCR recognition accuracy or relaxing the cleaning and filtering rules), and finally generate the fifth result (skill collaborative scheduling scheme).

[0066] Step 7: Accurate question and answer generation based on spatiotemporal gating units and correlation calculation formulas.

[0067] This step integrates the fifth model with the improved RAG retrieval enhancement technology, and uses a correlation calculation formula that introduces a time decay factor to recall corpus from the third result, solving the problems of historical inversion and cross-temporal illusion, and generating the sixth result.

[0068] (1) Dual-path recall and knowledge graph alignment: Based on the fifth result, a dual-path recall mechanism of vector semantic retrieval and BM25 keyword retrieval is initiated. During the recall process, a historical knowledge graph (KG) is introduced for entity alignment to ensure that the retrieval scope is strictly locked within the target historical events and their associated topological paths. This ensures both the comprehensiveness of the recall and the accuracy of the logical pointing through graph constraints.

[0069] (2) Fine-tuning of corpus and deep reconstruction of relevance: Using the re-sorting sub-model combined with the improved relevance calculation formula R, the corpus fragments are sorted in multiple dimensions. The re-sorting process not only considers semantic relevance, but also focuses on measuring the temporal fit between the source time of the corpus and the time of the event, and eliminates invalid corpus that is fake or seriously out of time.

[0070] (3) Spatiotemporal Gated Generation of the Fifth Model: The refined corpus is input into the fifth model (large language model). This invention embeds a temporal-spatial gate unit in the Transformer decoder of the fifth model. During the generation process, the gate unit verifies the consistency of the spatiotemporal labels of the generated content with the original corpus in real time through a sparse attention layer. If the generated content deviates from the target dynasty constraint, the gate unit will suppress the weight of that path through an attention mask, forcibly correcting the model illusion and ensuring that the question-and-answer results are not fabricated or out of bounds. The mask is applied to the Logits space before the Softmax layer, and hard generation interception is achieved by setting the probability value of conflicting words to negative infinity.

[0071] (4) Precise screening based on relevance: Based on the results of the improved relevance formula, filter the corpus with relevance R not less than the preset threshold.

[0072] The formula for calculating the correlation degree is:

[0073]

[0074] The specific definitions and calculation methods of each parameter in the formula are as follows:

[0075] ①R represents the relevance, with a value range of [0,1]. The larger the R value, the higher the degree of matching between the corpus fragment and the user's needs and search instructions, and the stronger the relevance.

[0076] ② The semantic vector similarity between the instruction intent and the corpus fragment M;

[0077] ③ This refers to the path similarity in the knowledge graph. It is calculated based on the topological distance of entity E in the historical knowledge graph K; the closer the distance, the stronger the logical connection.

[0078] ④ It is an exponential time decay function. This allows users to query the time offset between a dynasty and the dynasties recorded in the corpus. Let be the attenuation coefficient, when When the threshold (e.g., 100 years) is reached, the decay factor tends to 0, and the corpus is forcibly removed. The larger the offset, the more rapidly the score of this item decreases exponentially, which is used to suppress the problem of "misattribution" of eras from the algorithm's bottom layer;

[0079] ⑤ The preset weighting coefficients, and For specialized research scenarios, increase... and The weights are adjusted to strengthen logical and temporal constraints; for generalized query scenarios, the weights are increased. To enhance semantic coverage.

[0080] Finally, the fifth model was used to semantically integrate and logically organize the selected corpus fragments, generating accurate, compliant, historically accurate, and evidence-supported historical knowledge Q&A content, which served as the sixth result.

[0081] Step 8: Accurate end-to-end traceability verification based on physical anchors and hash pointers.

[0082] The core of this step is to utilize the previously generated coordinate data and encrypted index to construct an irreversible chain of evidence from the question-and-answer result to the original image pixel region, thereby achieving a high degree of credibility for the question-and-answer result. Specific technical details are as follows:

[0083] (1) Deep binding and hashing of source information: The sixth result (question and answer content) is input into the sixth model (source verification model). This model binds a unique semantic unit ID and a one-way irreversible hash pointer (HashPointer) to each set of viewpoints or information fragments in the result. The hash pointer not only contains the index of the corresponding semantic unit in the third result, but also contains the original image path and physical anchor coordinates (x, y, w, h) associated with the unit. The hash pointer is used to form a one-way linked list structure, so that each question and answer conclusion can be backtracked to the original physical coordinates that cannot be tampered with through recursive verification.

[0084] (2) Construction of a full-link pixel-level source tracing index: A full-link source tracing index is constructed through the sixth model to achieve reverse tracing from the sixth result to the third result. The source tracing path presents a progressive feature: question and answer viewpoint → hash pointer → structured semantic unit → historical metadata → original image physical coordinates. Through this path, the system can accurately locate the specific paragraph of the first data (electronic text) and the corresponding specific visual area (pixel-level positioning) in the second data (image-based historical materials), thoroughly clarifying the physical source of the question and answer results.

[0085] (3) Visualization of Evidence Chain and Generation of Closed-Loop System: Based on the source tracing results, a complete knowledge application evidence chain is constructed. This invention supports the "click to trace source" function on the front-end interface: when a user questions the question-and-answer results or needs to verify the evidence, the system highlights the corresponding pixel area in the original ancient book scan based on the hash pointer. Integrating all the aforementioned steps, a closed-loop historical question-and-answer application system of "collection → analysis → governance → modeling → scheduling → generation → source tracing" is formed (seventh result), ensuring that the knowledge source is traceable and verifiable throughout the entire process.

[0086] Step 9: Feedback-driven dynamic closed-loop iteration and algorithm self-evolution mechanism.

[0087] To achieve continuous system evolution and improve question-answering accuracy and spatiotemporal constraints, this method includes a dynamic closed-loop iterative mechanism:

[0088] (1) Accurate collection of multi-dimensional feedback information: During the question-and-answer output process, the autonomous collaborative center of the intelligent agent collects feedback information through user interaction and system self-check. The feedback content is classified as: semantic conflict, spatiotemporal span deviation (spatiotemporal illusion), recognition error and logical weight anomaly.

[0089] (2) Iterative optimization of algorithm parameters and model weights: Feedback information is fed back to each core component for parameter tuning:

[0090] Regarding spatiotemporal deviation feedback: the system automatically increases the time decay coefficient in the correlation calculation formula R. To enhance the interception capability of the spatiotemporal gating unit;

[0091] Regarding quality assessment feedback: dynamically adjust the weighting coefficients in the quantitative scoring formula Q. (Historical validity) or update the CRF sentence segmentation confidence model in the S-index;

[0092] For feedback on recognition errors: trigger fine-tuning of the parameters of the first model and optimize the attention allocation strategy of the page segmentation sub-model for specific types of historical materials (such as damaged rubbings).

[0093] (4) System autonomous iteration and knowledge base hot update: The system can be upgraded autonomously through continuous parameter iteration and content optimization. The system supports "hot update" of the structured semantic knowledge base, that is, without stopping the service, it can correct erroneous entity associations and invalid hash links, further improve the quality and accuracy of the sixth and seventh results, and adapt to the ever-evolving historical research needs and multimodal corpus resources.

[0094] This embodiment utilizes normalized parsing, precise governance, and structured modeling of multimodal historical corpora. It employs professional domain-specific intelligent agents for collaborative scheduling to complete question-and-answer generation and full-link tracing, constructing a closed-loop historical question-and-answer application system. This improves the utilization efficiency of historical corpora and the accuracy and traceability of knowledge-based question-and-answer, providing efficient and reliable technical support for historical research, cultural heritage preservation, and education dissemination. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart of the overall process for an intelligent agent cooperative method based on multimodal historical corpus.

[0097] Figure 2 A detailed logic diagram for corpus governance and structured modeling;

[0098] Figure 3 This is a logic diagram of agent-based collaborative scheduling based on the CoT (CoT) framework.

[0099] Figure 4 A schematic diagram for generating spatiotemporal gating embedded in a Transformer decoder;

[0100] Figure 5 This is a logical pointer diagram for the entire chain of pixel-level traceability evidence. Detailed Implementation

[0101] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, techniques, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Example 1

[0102] Typical historical research scenario (Qing Dynasty imperial examination system)

[0103] This example demonstrates in detail how the model solves the challenge of ancient text analysis through spatial feature fusion and attention masking:

[0104] 1. Image parsing stage (first model mechanism description):

[0105] The system inputs a scanned image of the "Examination Regulations" into the first model. The underlying layer of this model uses ViT (VisionTransformer) to encode 16×16 image patches and extract global visual features.

[0106] The GCN (Graph Convolutional Network) in the model treats each identified text unit as a node and builds an adjacency matrix based on spatial distance. When the model detects that the text is densely arranged vertically, the GCN automatically adjusts the edge weights between nodes, guiding the Attention mechanism to aggregate features along the y-axis.

[0107] Set the confidence threshold P = 0.82. For the damaged character "闱", the original probability output by the model is 0.65, triggering the semantic completion sub-model. This sub-model performs BeamSearch (beam search) based on the previous characters "进入顺天" and retrieves the highest co-occurrence probability of the character "闱" in the historical word vector space, thus correcting the final recognition confidence to 0.88.

[0108] 2. Corpus Governance Phase (Description of the Second Model Mechanism):

[0109] The second model loads a conditional random field (CRF) layer. When calculating the normalization index S, the state transition probability matrix of the CRF layer scans the text stream.

[0110] If the OCR recognition result is "录取名单如下张三", the CRF layer detects that there is no pause marker between "如下" and "张三", and the state transition probability drops sharply. , and the system determines that there is a typesetting misalignment at this place, automatically records the physical anchor coordinates [x, y] in the relational database, and marks it as the "to be optimized" state.

[0111] 3. Structured Modeling (Third Model)

[0112] The structured semantic units constructed by the third model have the following JSON data structure:

[0113] { "unit ID": "SU_20241024_001", "text content": "康熙三十六年,圣祖下令停止乡试中额划分...", "physical anchor": {"image ID": "IMG_KQ_045", "coordinates": [120, 450, 30, 100]}, "spatiotemporal metadata": {"dynasty": "Qing", "specific year": 1697, "region": "Shuntianfu"}, "entity labels": ["Kangxi", "provincial examination", "imperial examination reform"], "quality score Q": 88.5, "hash pointer": "0x7f8a9d1b2c3e4f5a" }

[0114] Through the above standardized data structure, the fourth model can accurately read the spatiotemporal attributes of each semantic unit, providing deterministic data support for subsequent gated interception and hash traceability.

[0115] 4. Intelligent Agent Collaborative Scheduling Phase (Fourth Model Agent Decision Logic):

[0116] After receiving a user's inquiry about "measures to prevent cheating in the imperial examinations during the Kangxi era," the fourth model initiates a path planning mechanism based on Chain of Reasoning (CoT) reasoning:

[0117] Task decomposition and intent alignment: The fourth model, through intent recognition, discovered that the requirement contained dual attributes of "policy retrieval (preventive measures)" and "event verification (cheating case)". The model automatically generated an execution graph, decomposing the total task into: ① retrieving laws and regulations from the Kangxi era regarding examination discipline; ② locating the specific handling results of the "Jiangnan Examination Case"; ③ verifying the spatiotemporal consistency of the corpus.

[0118] Skill Dynamic Path Planning: Based on the task graph, the fourth model accurately matches the execution link from the Skill matrix. Decision Logic Judgment: The "Spatiotemporal Awareness Retrieval Skill" must be called first to ensure that the time domain is locked in the early Qing Dynasty. Then, the "Logical Conflict Verification Skill" is connected to eliminate interference from unofficial historical records. Finally, the processed knowledge flow is pushed to the "Gated Generation Skill".

[0119] Execution monitoring and adaptive adjustment: During the scheduling process, if the fourth model detects that the quality score Q of the "laws and decrees" corpus is lower than the preset weight, the backtracking mechanism is triggered, and the automatic instruction retrieval skill is expanded to expand the search radius to the "court memorials" category to ensure that the context finally converged on the generating model has the highest evidentiary value.

[0120] 5. Question-answer generation stage (fifth model spatiotemporal gating logic):

[0121] The fifth model receives the corpus fragments transmitted by the Skill matrix.

[0122] During the decoding process of the Transformer, the system invokes the spatiotemporal gating unit. This unit inserts a binary mask based on dynasty labels after the Cross-Attention layer.

[0123] When the model attempts to generate information about "Qianlong" to answer the question about "Kangxi," the gating unit calculates the time decay factor. When the probability approaches zero, a masking mechanism is triggered, directly blocking the generation probability of related words in the Softmax layer, forcing the model to revert to the corpus path of the Kangxi era, and generating accurate answers.

[0124] The specific interactive data flow is as follows:

[0125] A user entered the query: 'What preventative measures were taken against cheating in the imperial examinations during the Kangxi era?'

[0126] The recalled fragment retrieved by the system (fragment A): '...Anyone who writes on behalf of others, smuggles, or passes documents within the examination hall will be severely punished. In the fiftieth year of the Kangxi Emperor's reign, the Jiangnan Examination Hall Scandal occurred...'.

[0127] The conflicting fragment intercepted by the spacetime gating unit (fragment B, from the Qianlong era): '...Emperor Qianlong emphasized that the examination hall search must be more rigorous...' (The calculated time offset caused attenuation and was intercepted by the gating).

[0128] The fifth model ultimately generated the sixth result: 'During the Kangxi era, strict preventative and punitive measures were implemented to address cheating in the imperial examinations. On the one hand, the procedures for searching examination halls were strictly regulated, and the use of smuggled items and proxy writing were strictly prohibited; on the other hand, severe penalties were imposed for cheating cases. For example, in the Jiangnan imperial examination scandal that broke out in the fiftieth year of Kangxi's reign, the Emperor severely punished the officials involved in the cheating, thereby rectifying officialdom and preventing cheating.'

[0129] In this embodiment, the generated sixth result (question and answer content) includes dynamically generated source tracing anchor links. Taking the 'Jiangnan Imperial Examination Case' as an example, the underlying layer of this link encapsulates the physical coordinate index (x:200, y:800, w:150, h:50) pointing to the original historical image and the corresponding evidence consistency hash value.

[0130] When a user triggers the link, the sixth model parses the addressing protocol in the link and instantly retrieves the original scanned image stored in the relational database on the front-end interface (page 45 of the Draft History of the Qing Dynasty). Based on the coordinate parameters, it automatically generates a red rectangle highlighting box in the corresponding pixel area of ​​the image, realizing a 'one-click, pixel-level' reverse closed-loop verification from semantic viewpoint to original physical evidence. Example 2

[0131] A cross-dynastic historical and literary research context (economic connections in Jiangnan during the Tang, Song, Ming, and Qing dynasties).

[0132] This example demonstrates in detail how the model handles massive heterogeneous corpora and solves deep logical reasoning problems through knowledge graph topology paths and contrastive learning:

[0133] 1. Heterogeneous Layout Perception and Resolution Stage (First Model Mechanism Description):

[0134] For the "Tang Dynasty mural inscriptions" and "Qing Dynasty handicraft rubbings" collected in this embodiment, the first model performs a specific analysis:

[0135] Image enhancement and feature extraction: To address the characteristics of blurred and mottled text in murals, the model incorporates a spatial frequency enhancement operator before ViT encoding to improve the sharpness of character edges.

[0136] Non-standard layout processing: Since rubbings and murals do not have a fixed layout, GCN (Graph Convolutional Layer) dynamically constructs undirected graph weights between characters. When text wrapping or irregular arrangement is detected, the model automatically identifies the reading order and records the physical coordinate anchor point (x, y, w, h) of each character by calculating the geodesic distance between nodes.

[0137] 2. Cross-dynastic pre-alignment stage (second model mechanism description):

[0138] Comparative learning logic: The second model employs Triplet Loss. The text vectors of "Tang Dynasty polder fields", "Song Dynasty Champa rice", and "Ming and Qing Dynasty cotton textiles" are projected onto the same high-dimensional semantic space.

[0139] Technical effect: By narrowing the cross-dynastic vector distance with the attribute of "productivity accumulation", the model can identify the internal logical chain of economic evolution across hundreds of years (e.g., agricultural surplus is transformed into raw materials for handicrafts).

[0140] 3. Cross-spatial-temporal structured modeling stage (third model mechanism description):

[0141] The third model decomposes the recognized text into its smallest semantic units and generates JSON data with "spatiotemporal alignment" capabilities.

[0142] Entity logical alignment: The model automatically identifies "Huating County in the Tang Dynasty" and "Songjiang Prefecture in the Ming Dynasty" as different spatiotemporal states of the same spatial entity and assigns them a unified geographic entity code.

[0143] Metadata encapsulation: In addition to the text vector, each unit also encapsulates the evidence confidence score Qual(K). For example, for the printed text of "Qimin Yaoshu", the system assigns it a high weight level of 0.92 to ensure that it has priority in subsequent fine sorting.

[0144] 4. Intelligent Scheduling Phase (Fourth Model Agent Decision Logic):

[0145] Thought chain execution: After receiving the instruction, the fourth model (Agent) triggers the ReAct (reasoning-action) loop and evaluates the execution path through DQN (deep reinforcement learning).

[0146] Decision-making: The decision requires first using the "Place Name Evolution Mapping Skill" for spatial alignment, and then using the "Cross-Dynasty Association Skill" to calculate the multi-hop path similarity in the knowledge graph. .

[0147] 5. Relationship Ranking and Spatiotemporal Gating Generation (Fifth Model Mechanism Description):

[0148] Fine-grained ranking logic: Set β=0.5 (knowledge graph weight). If corpus fragment A records "a bumper rice harvest supported weaving laborers", it is connected to "progress in handicrafts" in the graph through the node "surplus materials → hired labor", with a path distance of only 2 hops. The calculated... It is 0.85.

[0149] Spatiotemporal gating function: When generating the final report, the fifth model embeds a spatiotemporal gating unit, which enables attention bias gating for the four dynasties of "Tang, Song, Ming, and Qing". When the model discusses the Tang Dynasty, the gating unit automatically filters out irrelevant noise from the "Qing Dynasty contract", ensuring that the generated report maintains logical coherence while accurately corresponding to the specific historical facts of each dynasty.

[0150] 6. End-to-end cross-dynastic tracing (sixth model mechanism description):

[0151] Hash chain tracing: The sixth model establishes a hash chain pointer for each conclusion in the report (such as "commercialization of agriculture promotes the refinement of the division of labor").

[0152] Physical evidence loop: When researchers questioned the "Song Dynasty water conservancy technology" mentioned in the report, the system used hash pointers to cross the dynasty index and directly retrieved rubbings of the Tang Dynasty's *Qimin Yaoshu* and scanned copies of Ming Dynasty local gazetteers from the front end, precisely highlighting the original recorded areas on the images. This proves that the conclusion is not based on probabilistic guesses from a large model, but on a pixel-level empirical evidence chain spanning thousands of years.

[0153] The above are merely specific embodiments of this application. Parameters and module configurations can be adjusted according to actual application scenarios, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal history corpus based agent coordination application method, characterized in that, Includes the following steps: Step 1: Obtain the first data in text category and the second data in image category. Perform format recognition and metadata registration on the first and second data. Classify and store the second data according to image features to reserve a globally unique corpus index ID for subsequent full-link tracing. Step 2 (Layout Awareness Analysis): Input the first data and the second data into the first model. The first model is a layout awareness OCR model based on ViT-GCN (Visual Transformer-Graph Convolutional Network). While recognizing characters, it extracts the original image physical coordinate sequence corresponding to each character as physical anchor points, and performs standardization transformation according to the preset variant character equivalence class dictionary to generate a standardized original historical text stream with physical anchor points. Step 3 (Conflict-Aware Governance): Integrate the rule engine with the second model, which is a conflict-aware governance model based on contrastive learning. It uses the equivalence class space of variant character embedding for semantic alignment and adopts a quantitative scoring formula Q that includes a conflict rate penalty term to perform multi-dimensional quantitative scoring of the text and select high-quality corpus. Step 4 (Semantic Structured Modeling): The smallest semantic unit is decomposed through the third model, which is a semantic structured modeling model. It inherits the physical anchor point and combines the historical knowledge graph to extract entities and their spatiotemporal metadata to construct a hierarchical historical semantic knowledge base. Step 5 (Collaborative Scheduling Preparation): Decouple and encapsulate the functions of corpus parsing, intelligent cleaning, quality assessment, semantic modeling, retrieval and recall, question answer generation and source tracing verification into a standardized Skill matrix; Step Six (Intelligent Collaborative Scheduling): Intent recognition and task decomposition are performed through the fourth model, which is an intelligent agent collaborative scheduling model based on the CoT (CoT) architecture. It generates a skill collaborative scheduling scheme that includes dynamic skill routing according to user needs. Step 7 (Spatiotemporal Gated Question-Answer Generation): Integrating the fifth model with RAG (Retrieval Enhanced Generation) technology, the fifth model is a knowledge enhancement generation model embedded with spatiotemporal gating units. It uses the relevance calculation formula R based on the spatiotemporal decay factor to recall the corpus, and in the Transformer decoding stage, it uses the binary mask generated by the spatiotemporal gating unit to mask words that conflict with the target dynasty to generate question-answer content. Step 8 (Full-link physical tracing): Establish a full-link physical tracing path through the sixth model, which is a tracing verification model that uses hash pointers to establish a one-way irreversible chain of evidence from the question and answer viewpoint to the physical anchor point.

2. The method according to claim 1, characterized in that, The quantitative scoring formula Q is: ， Wherein, Q is the corpus quality quantitative score, C is the content integrity index, H is the historical rationality index including the logical conflict rate item, S is the text normativity index and includes the word segmentation and sentence segmentation confidence score based on conditional random field (CRF), and T is the semantic relevance index. Preset weighting coefficients; The correlation degree calculation formula R is: in, For semantic vector similarity, For knowledge graph path similarity An exponential time decay function, used to determine the dynasty from which the query is performed. With the corpus records of dynasties The time offset suppresses the illusion of spacetime; α, β, and γ are preset weighting coefficients.

3. The method of claim 1, wherein, Step one specifically includes: The first data is encoded and formatted uniformly, while the second data is preprocessed based on image denoising and blur correction. The data is input into the first model, and the ViT module is used to perform image block encoding to extract global visual features. The GCN module is used to establish an adjacency matrix based on the spatial distance of the text and perform vertical feature aggregation to locate the text region of the second data. The spatial logic model of the layout elements is completed through the first model, the layout features are extracted and the character-level physical coordinate sequence (x,y,w,h) is recorded; The first model completes character recognition, and when the recognition confidence is lower than a preset threshold, the semantic completion sub-model is triggered to perform contextual error correction, ultimately achieving the splicing and fusion of layout features and semantic features.

4. The method of claim 1, wherein, Step two specifically includes: By introducing contrastive learning techniques through a second model, feature alignment is performed on different documentary descriptions of the same historical event. A dictionary of variant characters equivalence classes is established in the Embedding space, mapping variant characters from different historical periods to the same cluster center in the vector space; Logical consistency is checked by calculating the conflict score of the "person-time-place" triple, and suspected errors in historical facts are marked. High-quality corpora are selected based on the quantitative scoring formula Q, and automatic backflow cleaning and secondary optimization of model parameters are performed on corpora with scores below the threshold.

5. The method of claim 1, wherein, Step three specifically includes: The corpus is divided into the smallest independent and complete semantic units, and each unit inherits the corresponding physical coordinate sequence. Historical knowledge graphs are used to assist in identifying entities and their spatiotemporal attributes, and to label them with dynasty tags and evidence confidence metadata. Semantic units are vector-encoded and synchronously stored in a vector database and a relational database to establish a geospatial evolution mapping index for entities across dynasties.

6. The method as described in claim 1, characterized in that, Step four specifically includes: The functions are decoupled into independent Skill modules for corpus parsing, conflict-aware governance, spatiotemporal gating retrieval, and end-to-end hash tracing, and a state machine-based management mechanism is established. The Skill matrix supports adaptive parameter adjustment based on the intermediate state transmission format during the thought chain scheduling process, and supports dynamic routing and backtracking of the Skill module.

7. The method as described in claim 1, characterized in that, The skill collaborative scheduling scheme generated in step four specifically includes: The fourth model uses a thought chain reasoning mechanism to break down user needs into subtask flow graphs with temporal logic; The temporal constraints are decomposed based on the dynastic span corresponding to the sub-tasks, and the optimal execution link is matched using a path planning algorithm. When the knowledge density reported by a subtask is lower than a preset threshold, the scheduler automatically triggers a backtracking instruction, which is then fed back to the OCR module for a high-precision secondary scan.

8. The method as described in claim 1, characterized in that, Step five specifically includes: Initiate a dual-path recall mechanism that combines vector semantic retrieval with knowledge graph-based path verification; The refined corpus fragments are input into the fifth model. In the Transformer decoding stage, the attention weight allocation matrix is ​​biased by the binary mask generated by the spatiotemporal gating unit to block words that conflict with the target dynasty. By introducing an illusion penalty factor and incorporating the correlation formula R as a component of the loss function, the generation process is constrained, thereby suppressing cross-temporal and spatial logical fabrication.

9. The method as described in claim 1, characterized in that, Step six specifically includes: The sixth model binds one-way irreversible hash pointers to the viewpoints in the question-and-answer results, and the hash pointers are mapped to the corresponding physical coordinate sequences. Construct a progressive source-tracing index that goes from question-and-answer viewpoints to structured semantic units, and then to pixel regions of original historical materials; It supports instantly retrieving the original image and generating a highlighted display box in the corresponding physical coordinate area when a user triggers a source tracing request.

10. The method as described in claim 1, characterized in that, The dynamic closed-loop iteration mechanism specifically includes: The system collects feedback information including spatiotemporal span deviations and logical weight anomalies, and then adjusts the time decay coefficient in the correlation calculation formula R accordingly. ; Based on feedback, fine-tune the attention allocation strategy of the first model and the quantization scoring weight coefficients of the second model. ; Achieve hot content updates and algorithm self-evolution for a structured semantic knowledge base.

Images