A method and system for processing long documents

Abstract

The application relates to the technical field of natural language processing, and discloses a long document processing method and system, which comprises the following steps: constructing a unified document object, extracting global semantics, and generating a navigation type summary as a global constraint; matching an expert library based on the summary, and reusing or deriving an expert when a hit is not obtained; slicing the document structure by the expert, generating evidence cards containing conclusions, original text excerpts and traceability information, and establishing traceability mapping; verifying the evidence cards and question and answer pairs in combination, and removing the question and answer pairs when all the evidence cards relied on by the question and answer pairs do not pass the verification; extracting experience increments, merging and compressing the experience increments under the constraint of memory capacity to update the expert memory; and persisting the processing process data and generating a state audit card. The application solves the problem of untrusted output through global semantic anchoring, dynamic expert routing, traceable evidence and a closed loop of joint verification, and realizes long document processing with high credibility, traceability and auditability.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a method and system for processing long documents. Background Technology

[0002] As large language models are increasingly applied to document understanding, knowledge extraction, and question-answering generation tasks, utilizing intelligent agents to process long professional documents such as papers, technical reports, patent documents, industry manuals, and regulations has become an important direction. Existing solutions typically fall into the following categories: First, the full text or a large amount of text is directly input into the model, which then directly generates summaries, question answers, or conclusions; second, long documents are first segmented into fixed lengths, and then local segments are retrieved based on a retrieval-enhanced generation architecture to generate results; third, fixed prompt word templates and fixed roles are used for unified processing of multiple document types.

[0003] However, the above-mentioned solutions generally have the following shortcomings when dealing with documents that are structurally complex, lengthy, and highly specialized: due to the lack of a systematic result constraint mechanism, most of the existing solutions directly output the initial generated results, which cannot conduct closed-loop verification of the authenticity, self-containment, and consistency between the question-and-answer pairs and the underlying evidence. They cannot rule out the possibility that the model generates content without basis, and they cannot establish a reliable traceability chain from the output conclusion to the physical location of the original text. Ultimately, the credibility of the long document processing results is difficult to guarantee. Therefore, there is an urgent need to provide a long document processing method and system to solve the above problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a long document processing method and system that organically combines global document profiling, dynamic expert routing, traceable evidence generation, and joint verification closed loop to achieve high reliability and traceability of the output results.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A long document processing method includes:

[0007] The process involves acquiring long document data to be processed, extracting document structure information, and constructing a unified document object. Before chunking, the process extracts global semantic features of the document based on the unified document object and generates a navigation-style document state summary as a global constraint for subsequent processing.

[0008] A route identifier is generated based on the navigation document state summary, and the route identifier is used to match the historical expert database. If no match is found, the matching degree of the candidate experts is evaluated based on the navigation document state summary. Based on the evaluation result, the historical experts are reused or new experts are derived to establish the currently active expert. The active expert is an intelligent agent object with domain document processing capabilities and the ability to accumulate processing experience.

[0009] The activation expert, based on the unified document object, divides the full text into chapters according to the document structure and generates chapter slices; for valid slices, a structured evidence card containing a conclusion statement, an excerpt from the original text, and source location information, and a corresponding verification question-and-answer pair are generated, and a source mapping relationship between the evidence card and the original text location is established;

[0010] The structured evidence cards and their corresponding verification question-and-answer pairs are jointly verified; when all the evidence cards on which the verification question-and-answer pair depends fail verification, the verification question-and-answer pair is removed.

[0011] As a preferred embodiment of the present invention, the extraction of document structure information includes extracting the document's title hierarchy, full text, and page layout metadata;

[0012] The global semantic features include language, domain, topic, core issues, processing methods, keyword set, main research findings / core conclusions, global semantic summary, and chapter summary;

[0013] When extracting global semantic features based on the unified document object, if the external model call fails, the first N high-frequency keywords, metadata, and title texts at all levels of the document are automatically extracted and a baseline state summary is generated by splicing them together using rule templates to ensure processing continuity; where N is 5–10.

[0014] As a preferred embodiment of the present invention, the routing identifier is a routing fingerprint; the routing identifier is obtained by semantic embedding multiple core features of the navigation document state summary to obtain a summary feature vector, and the vector is reduced in dimension by a perceptual hash algorithm to generate a fixed-length bit feature string;

[0015] The matching degree evaluation is based on a multi-dimensional coverage score of vector similarity, topic overlap, task overlap, problem overlap, method overlap, domain overlap and keyword overlap, and the sum of the weight coefficients of topic, task, problem and method overlap is greater than or equal to 0.6.

[0016] As a preferred embodiment of the present invention, the historical expert is reused only when the score of the best candidate expert reaches a preset total score threshold, the margin of the expert leading the second-best candidate reaches a preset threshold, and the maximum value among the overlap of topic, task, problem, and method reaches a preset core overlap threshold; otherwise, a new expert object is derived; the derived new expert object is initially marked as a temporary expert object, and is promoted to an active expert object after being reused or selected a preset number of times.

[0017] In a preferred embodiment of the present invention, the chapter slices adaptively configure the slice length budget and overlap length according to the document language type; the target length of Chinese documents is 5200 characters, the maximum length is 6000 characters, and the overlap length is 600 characters; the target length of English documents is 9000 characters, the maximum length is 11000 characters, and the overlap length is 900 characters; the target length of mixed language documents is 7000 characters, the maximum length is 8500 characters, and the overlap length is 800 characters; the activation expert outputs slice-level control signals for each slice, and the slice-level control signals include accepting slice, rejecting slice, and terminating subsequent slices.

[0018] As a preferred embodiment of the present invention, the data model of the structured evidence card includes a unique card identifier, a content verification field, and a structured source tracing field; the content verification field includes a conclusion statement field, a text excerpt field, and a self-contained check bit; the structured source tracing field includes a document identifier, a multi-level path list of chapters, and the character start index and end index of the slice in the original text.

[0019] As a preferred embodiment of the present invention, the joint verification adopts a two-stage verification method: the first stage verifies the authenticity and self-containment of the evidence card, and the second stage verifies the consistency between the question and answer pair and the evidence card; for items that fail the joint verification, verification opinions are generated and fed back to the activating expert for secondary generation, with a maximum of two rounds per item; the reasons for rejection are recorded for the rejected verification question and answer pair.

[0020] As a preferred embodiment of the present invention, it further includes: extracting expert experience increments based on the results of the current verification round; merging, deduplicating, and compressing the experience increments with historical expert memories under a preset memory capacity constraint; updating the memories of the activated experts; the deduplication is performed based on the Jaccard similarity of the token set for near-duplicate filtering, and pattern merging is triggered when the similarity is greater than a preset threshold; the compression uses a dynamic priority scoring function based on decay weight to evaluate and prune memory entries, and the scoring mechanism comprehensively considers the memory type weight and time decay factor to determine the value of each memory, wherein the weight of failure mode memories is greater than the weight of success mode memories.

[0021] As a preferred embodiment of the present invention, it further includes: persistently storing the document status summary, routing decision trajectory, slice control signal, evidence card, verification question-and-answer pair, expert memory update and model call record during the processing, and generating a structured status audit card.

[0022] A long document processing system includes a document parsing unit, a content processing unit, and a result output unit, and also includes a global constraint generation module, an expert routing adaptation module, a traceable content generation module, and a joint verification closed-loop module;

[0023] The global constraint generation module is used to perform standardized parsing on the long document to be processed and generate a unified document object. Before performing local decomposition and block processing on the unified document object, global semantic and structural modeling is performed on the unified document object to generate a navigation-style global state summary covering the core semantic and structural features of the document. The global state summary is used as the global semantic anchor point and boundary constraint for a single processing step in the entire process.

[0024] The expert routing adaptation module is communicatively connected to the global constraint generation module. It is used to generate routing identifiers representing the core processing requirements of the document based on the global state summary, and to perform matching in a pre-built historical expert database using the routing identifiers. If a match is found, the corresponding historical expert is reused as the current active expert. If a match is not found, the matching degree of the candidate experts is evaluated based on the global state summary. Based on the evaluation result, the best candidate expert is reused or a new expert adapted to the current document is derived to establish the current active expert. The active expert is an intelligent agent object with document processing capabilities in the corresponding domain.

[0025] The traceable content generation module is communicatively connected to the global constraint generation module and the expert routing adaptation module, respectively. It is used to call the activated expert, and based on the global state summary, the unified document object is hierarchically decomposed according to the document structure to generate chapter slices with positioning identifiers. For the effective chapter slices, the core information is extracted to generate structured evidence cards and corresponding verification question-and-answer pairs. The structured evidence cards include conclusion statements, supporting content of the original text and traceability positioning information, and a traceable mapping relationship is established between the evidence cards and the corresponding content of the original document.

[0026] The joint verification closed-loop module is communicatively connected to the traceable content generation module and the result output unit, respectively. It is used to perform multi-dimensional joint verification on the structured evidence cards and their corresponding verification question-and-answer pairs. The content is deemed valid only when all verification dimensions meet the standards. Closed-loop correction is performed on the content that fails the verification. When all evidence cards on which a verification question-and-answer pair depends fail the verification, the verification question-and-answer pair is automatically removed. Finally, the verified structured evidence cards and verification question-and-answer pairs are sent to the result output unit to complete the single long document processing.

[0027] The beneficial technical effects of this invention are:

[0028] Through a joint verification closed-loop and linkage rejection mechanism, the authenticity, self-containment, and supporting consistency of structured evidence cards and verification question-and-answer pairs are systematically verified. When all evidence cards on which the verification question-and-answer pair depends fail verification, they are automatically removed, ensuring that every conclusion in the final output has valid underlying evidence support. This fundamentally eliminates the possibility of the model generating content without basis and directly overcomes the core defect of existing solutions that lack a result constraint mechanism.

[0029] By generating structured evidence cards that include conclusion statements, excerpts from the original text, and source location information, a three-level source mapping is established from evidence cards to chapter slices and then to the physical location of the original text. This allows each output conclusion to be traced back to the specific location in the original text, providing locationable underlying evidence support for joint verification.

[0030] By constructing a navigational document state summary before segmentation, global semantic features such as language, domain, topic, core issues, and processing methods are extracted, providing unified global semantic boundary constraints for subsequent slicing and evidence generation, thus avoiding topic drift and factual bias. At the same time, through route identifier matching and multi-dimensional coverage scoring based on the navigational state summary, expert objects adapted to the current document are dynamically reused or derived, ensuring that professional documents are processed by experts in the corresponding domain, reducing source distortion and laying a preliminary foundation for the credibility of the results.

[0031] By extracting incremental expert experience based on the results of this round of verification, and merging, deduplicating, and compressing the data under preset memory capacity constraints, the experts' memories are updated and activated. This allows experts to continuously accumulate experience in avoiding and correcting errors when processing similar long documents, achieving self-evolution of cross-document capabilities. At the same time, document status summaries, routing decision trajectories, evidence cards, verification records, expert memory updates, and model call records are persistently stored to generate structured status audit cards. This transforms implicit internal reasoning logic into explicit structured records, enabling traceability, review, and auditability of the entire processing process, and meeting business compliance requirements. Attached Figure Description

[0032] Figure 1 This is a flowchart of the processing method of the present invention.

[0033] Figure 2 This is the dynamic expert routing decision logic diagram in this invention.

[0034] Figure 3 This is a flowchart of the traceable evidence card generation and joint verification process in this invention.

[0035] Figure 4 This is a flowchart of expert memory compilation and constraint self-evolution in this invention.

[0036] Figure 5 This is the overall architecture diagram of the long document processing system in this invention. Detailed Implementation

[0037] In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0038] Combination Figures 1-5 The present invention provides the following embodiments:

[0039] Example 1:

[0040] A long document processing method includes:

[0041] The process involves acquiring long document data to be processed, extracting document structure information, and constructing a unified document object. Before chunking, the process extracts global semantic features of the document based on the unified document object and generates a navigation-style document state summary as a global constraint for subsequent processing.

[0042] A route identifier is generated based on the navigation document state summary, and the route identifier is used to match the historical expert database. If no match is found, the matching degree of the candidate experts is evaluated based on the navigation document state summary. Based on the evaluation result, the historical experts are reused or new experts are derived to establish the currently active expert. The active expert is an intelligent agent object with domain document processing capabilities and the ability to accumulate processing experience.

[0043] The activation expert, based on the unified document object, divides the full text into chapters according to the document structure and generates chapter slices; for valid slices, a structured evidence card containing a conclusion statement, an excerpt from the original text, and source location information, and a corresponding verification question-and-answer pair are generated, and a source mapping relationship between the evidence card and the original text location is established;

[0044] The structured evidence cards and their corresponding verification question-and-answer pairs are jointly verified; when all the evidence cards on which the verification question-and-answer pair depends fail verification, the verification question-and-answer pair is removed.

[0045] Furthermore, the extraction of document structure information includes extracting the document's title hierarchy, full text, and page layout metadata;

[0046] The global semantic features include language, domain, topic, core issues, processing methods, keyword set, main research findings / core conclusions, global semantic summary, and chapter summary;

[0047] When extracting global semantic features based on the unified document object, if the external model call fails, the first N high-frequency keywords, metadata, and title texts at all levels of the document are automatically extracted and a baseline state summary is generated by splicing them together using rule templates to ensure processing continuity; where N is 5–10.

[0048] By encapsulating heading levels, full-text content, and layout metadata into standardized document objects, a consistent format boundary is provided for downstream segmentation, source tracing, and evidence generation. The multi-dimensional slot design of these global semantic features collectively constitutes a complete semantic profile of the document, ensuring that subsequent expert routing and evidence extraction are always constrained by the global semantic boundary, avoiding high-dimensional semantic drift caused by localized processing. A heuristic fallback strategy, using a rule template of "high-frequency keywords + metadata + headings," can still generate a baseline state summary containing core features even when the external large model is unavailable, ensuring that the long document processing pipeline is not interrupted by a single point of failure.

[0049] Furthermore, the route identifier is a route fingerprint; the route identifier is obtained by semantic embedding multiple core features of the navigation document state summary to obtain a summary feature vector, and the vector is reduced in dimension using a perceptual hash algorithm to generate a fixed-length bit feature string;

[0050] The matching degree evaluation is based on a multi-dimensional coverage score of vector similarity, topic overlap, task overlap, problem overlap, method overlap, domain overlap and keyword overlap, and the sum of the weight coefficients of topic, task, problem and method overlap is greater than or equal to 0.6.

[0051] First, discrete semantic features are mapped into dense vectors through semantic embedding. Then, perceptual hashing is used to reduce dimensionality and generate fixed-length bit strings. This allows routing identifiers to retain the core semantics of the document while possessing uniqueness and stability, avoiding interference from irrelevant features in matching accuracy. In the multidimensional coverage scoring, the sum of the weights of the four items—topic, task, problem, and method—is limited to no less than 0.6 because these four items directly represent the expert's core domain capabilities. Increasing their weight ratio can prevent expert mismatch caused by vector similarity or accidental keyword overlap, ensuring that routing decisions are dominated by essential domain features.

[0052] Furthermore, the historical expert is reused only when the score of the best candidate expert reaches a preset total score threshold, the margin of the lead over the second-best candidate reaches a preset threshold, and the maximum value among the overlap of topic, task, problem, and method reaches a preset core overlap threshold; otherwise, a new expert object is derived; the derived new expert object is initially marked as a temporary expert object, and is promoted to an active expert object after being reused or after accumulating a preset number of selections.

[0053] The triple-judgment mechanism prevents the forced reuse of mismatched historical experts due to slight scoring advantages or single-dimensional coupling, thus avoiding a decline in generation quality caused by expert mismatch. The marking and upgrading mechanism for temporary expert objects ensures that newly derived experts must be verified through actual reuse before they can become active experts, achieving a gradual accumulation of domain processing capabilities and preventing the expert database from becoming bloated and disordered due to the generation of a large number of unverified temporary objects at once.

[0054] Furthermore, the chapter slices adaptively configure the slice length budget and overlap length according to the document language type; the target length for Chinese documents is 5200 characters, the maximum length is 6000 characters, and the overlap length is 600 characters; the target length for English documents is 9000 characters, the maximum length is 11000 characters, and the overlap length is 900 characters; the target length for mixed language documents is 7000 characters, the maximum length is 8500 characters, and the overlap length is 800 characters. The activation expert outputs slice-level control signals for each slice, including accepting the slice, rejecting the slice, and terminating subsequent slices.

[0055] Significant differences exist in information density and character semantic efficiency across languages, necessitating differentiated slice budget configurations to ensure that slices from documents in each language fall within the optimal processing window of the large language model, minimizing information truncation or redundancy caused by inappropriate length. Three control signals activated by expert output form a real-time feedback loop for slice quality: the accept signal ensures high-information-density slices are included in evidence generation; the reject signal skips low-value paragraphs; and the termination signal preemptively truncates subsequent reasoning upon detecting topic deviation, avoiding unnecessary computational waste and topic drift.

[0056] Furthermore, the data model of the structured evidence card includes a unique card identifier, a content verification domain, and a structured source tracing domain; the content verification domain includes a conclusion statement field, a text excerpt field, and a self-contained check bit; the structured source tracing domain includes a document identifier, a multi-level path list of chapters, and the character start and end indexes of the slice in the original text.

[0057] The content verification domain, through the ternary constraint of conclusion statement, original text excerpt, and self-contained verification, ensures that the evidence cards can still be understood independently after being removed from the context, and that their conclusions strictly correspond to the original text excerpts, preventing the model from generating unfounded abstract generalizations. The structured source tracing domain, through multi-level paths of chapters and character-level start and end indexes, establishes a three-level mapping of evidence cards, chapter slices, and the physical location of the original text, so that the conclusion of any card can be located back to the original text in O(1) complexity through the index interval, providing a locationable and verifiable underlying support for subsequent joint verification and manual review.

[0058] Furthermore, the joint verification adopts a two-stage verification method: the first stage verifies the authenticity and self-containment of the evidence card, and the second stage verifies the consistency between the question and answer pair and the evidence card; for items that fail the joint verification, verification opinions are generated and fed back to the activated expert for secondary generation, with a maximum of two rounds per item; the reasons for rejection are recorded for the rejected verification question and answer pair.

[0059] The two-stage verification follows a logical sequence of first verifying the evidence and then verifying the questions and answers: the first stage ensures that the evidence card itself is consistent with the original text and can be independently understood, eliminating source distortion; the second stage verifies whether the answers to the questions and answers can be completely derived from the corresponding evidence, eliminating logical jumps. Only when both stages are met can the final output be obtained, forming a systematic result constraint. For items that do not meet the standards, a secondary generation is triggered and limited to a maximum of two rounds, which can correct errors while avoiding system blockage caused by infinite loops; recording the reasons for rejection provides a structured data foundation for end-to-end auditing and subsequent evolution of expert experience.

[0060] Furthermore, it also includes: extracting expert experience increments based on the results of this round of verification; merging, deduplicating, and compressing the experience increments with historical expert memories under a preset memory capacity constraint; updating the memories of the activated experts; the deduplication is based on the Jaccard similarity of the token set for near-duplicate filtering, and triggering pattern merging when the similarity is greater than a preset threshold; the compression uses a dynamic priority scoring function based on decay weight to evaluate and prune memory entries; the scoring mechanism comprehensively considers memory type weights and time decay factors to determine the value of each memory, wherein the weight of failure mode memories is greater than the weight of success mode memories.

[0061] The incremental increase in expert experience stems from the real feedback of this round of processing. Accumulating this feedback in expert memory enables subsequent processing of similar documents to gain error avoidance and correction capabilities, achieving cross-round capability evolution. Using Jaccard similarity of token sets for deduplication can identify nearly repeated experiences at the semantic level, avoiding memory redundancy. A time decay factor is introduced in the compression stage, naturally devaluing outdated experiences; simultaneously, higher weight is given to failure modes because error avoidance experience is scarcer in real-world scenarios and makes a greater marginal contribution to improving output credibility, thus prioritizing the retention of high-value correction knowledge within limited memory capacity.

[0062] Furthermore, it also includes: persistently storing document status summaries, routing decision trajectories, slice control signals, evidence cards, verification question-and-answer pairs, expert memory updates, and model call records during the processing, and generating structured status audit cards.

[0063] Long document processing involves multi-round reasoning and decision-making within a large language model, and its internal logic is inherently black-box. By persisting state summaries, routing trajectories, slice signals, evidence cards, verification results, memory updates, and original model call records, the implicit reasoning process can be transformed into explicit, structured audit data. This allows any output conclusion to be traced back to its generation path, expert selection criteria, underlying evidence fragments, and model call context, thereby achieving traceability, replayability, and auditability of the entire processing flow, meeting compliance requirements in high-reliability business scenarios.

[0064] Example 2:

[0065] A long document processing system includes a document parsing unit, a content processing unit, and a result output unit, and also includes a global constraint generation module, an expert routing adaptation module, a traceable content generation module, and a joint verification closed-loop module;

[0066] The global constraint generation module is used to perform standardized parsing on the long document to be processed and generate a unified document object. Before performing local decomposition and block processing on the unified document object, global semantic and structural modeling is performed on the unified document object to generate a navigation-style global state summary covering the core semantic and structural features of the document. The global state summary is used as the global semantic anchor point and boundary constraint for a single processing step in the entire process.

[0067] The expert routing adaptation module is communicatively connected to the global constraint generation module. It is used to generate routing identifiers representing the core processing requirements of the document based on the global state summary, and to perform matching in a pre-built historical expert database using the routing identifiers. If a match is found, the corresponding historical expert is reused as the current active expert. If a match is not found, the matching degree of the candidate experts is evaluated based on the global state summary. Based on the evaluation result, the best candidate expert is reused or a new expert adapted to the current document is derived to establish the current active expert. The active expert is an intelligent agent object with document processing capabilities in the corresponding domain.

[0068] The traceable content generation module is communicatively connected to the global constraint generation module and the expert routing adaptation module, respectively. It is used to call the activated expert, and based on the global state summary, the unified document object is hierarchically decomposed according to the document structure to generate chapter slices with positioning identifiers. For the effective chapter slices, the core information is extracted to generate structured evidence cards and corresponding verification question-and-answer pairs. The structured evidence cards include conclusion statements, supporting content of the original text and traceability positioning information, and a traceable mapping relationship is established between the evidence cards and the corresponding content of the original document.

[0069] The joint verification closed-loop module is communicatively connected to the traceable content generation module and the result output unit, respectively. It is used to perform multi-dimensional joint verification on the structured evidence cards and their corresponding verification question-and-answer pairs. The content is deemed valid only when all verification dimensions meet the standards. Closed-loop correction is performed on the content that fails the verification. When all evidence cards on which a verification question-and-answer pair depends fail the verification, the verification question-and-answer pair is automatically removed. Finally, the verified structured evidence cards and verification question-and-answer pairs are sent to the result output unit to complete the single long document processing.

[0070] Application example:

[0071] The long document processing method includes the following steps:

[0072] S1. Constructing a unified document object and a navigational document state summary

[0073] First, the long document data to be processed is obtained. The source document is then subjected to structured parsing by a format parsing engine. The document's heading level, full text, paragraph boundaries, page block information, and page metadata are identified and extracted. These are then encapsulated into a unified document object containing a unique document identifier, heading path, full text content, page block set, and source file path information, providing a standardized data carrier for subsequent global processing.

[0074] Before performing chunking processing on the document to be processed, the large language model is first invoked to perform global semantic analysis on the unified document object to extract the language of the document. ),field( ),theme( ), core issues ( ), processing methods ( Keyword set ( ), Key research findings / core conclusions ( Global Semantic Summary ( ) and chapter summary ( Generate a navigational document status summary :

[0075]

[0076] The status summary As a global navigation map, it provides unified global semantic constraints for subsequent expert routing, slice control, and evidence extraction.

[0077] When an external large language model service fails to call, times out, or encounters a parsing exception, a heuristic fallback strategy is automatically initiated: before extracting the document. High-frequency keywords ( Take 5-10) headings and metadata at each level, and concatenate them according to the template of "document title + high-frequency keywords + metadata" to generate a baseline status summary, ensuring that the long document processing pipeline is continuous and uninterrupted.

[0078] S2, Dynamic Expert Router Fingerprint Matching and Derivation

[0079] Obtain the navigation document status summary generated in step S1 Subsequently, to enable the system to accurately allocate the most suitable agent to perform subsequent decomposition and extraction tasks based on the global features of the document, this step introduces a dynamic expert routing mechanism. In this mechanism, the agents performing the tasks need to be formally modeled in advance to quantify their domain processing capabilities and to incorporate their continuously evolving experience.

[0080] Specifically, in one implementation, the expert object entity is first defined. Its structure is represented by triples. ,in This is the expert capability feature vector, used for subsequent multi-dimensional matching calculations with document features; This is a historical processing decision trajectory matrix, used to record processing behavior for end-to-end auditing; This serves as a dynamic memory snapshot pool, used to store error avoidance and correction experiences accumulated across documents. This structured definition transforms abstract intelligent agents into computable, matchable, and updatable entity objects, laying the computational foundation for subsequent route fingerprint matching and incremental state evolution.

[0081] Based on the navigational document state summary obtained from S1 Construct an expert route description object. The description object includes semantic features such as language, domain, topic, task, problem, method, keywords, global summary, and chapter outline.

[0082] Based on the above 7 core semantic features ( , , , , , , Semantic embedding is performed to obtain the summary feature vector. Subsequently, the perceptual hash algorithm was used to process the vector. Dimensionality reduction mapping is performed to generate a 128-bit fixed-length bit feature string, which serves as the unique routing fingerprint of the document. This ensures that each document corresponds to a unique routing fingerprint, and the fingerprint generation process incorporates the core semantics of the document to avoid irrelevant features interfering with matching accuracy.

[0083] Prioritize precise matching of route fingerprints: If an active expert object with a completely identical fingerprint exists in the historical expert database and is in an available state, then directly reuse that historical expert as the active expert and load its expert DNA and memory snapshot.

[0084] If an exact match is not achieved, a set of candidate experts is recalled from the historical expert database and the seed expert database, and a comprehensive coverage score is calculated. The calculation formula is as follows:

[0085]

[0086] in, Represents vector similarity, Indicates the degree of overlap in topics. Indicates the degree of task overlap. Indicates the degree of overlap in the questions. Representation of method overlap Indicates the degree of overlap in domains. Indicates keyword overlap; all overlap coefficients The sum is 1, and the degree of overlap in themes is 1. Task overlap Problem overlap Method overlap The sum of the weights should not be less than 0.6, ensuring that core domain capabilities dominate decision-making.

[0087] For example, the optimal weight configuration for each dimension is as follows, balancing matching accuracy and generalization:

[0088] .

[0089] The scoring results are reused only when the best candidate expert meets all three of the following conditions: 1. The comprehensive coverage score reaches the preset total score threshold; 2. The score difference relative to the second best candidate reaches the preset leading margin threshold; 3. The maximum value among the theme overlap, task overlap, problem overlap and method overlap reaches the preset core overlap threshold.

[0090] Otherwise, a new expert object is derived from the highest-rated seed expert, generating unique prompts and capability constraints to establish the currently active expert. The derived new expert object is initially marked as a temporary expert object; when it is subsequently reused, or when its cumulative selection count reaches a preset number, it is promoted to an active expert object, realizing the gradual accumulation of domain capabilities.

[0091] S3. Generation of Chapter Witness Slices and Traceable Evidence Cards

[0092] After activating the expert takeover of the unified document object, the document is divided into chapters according to the heading level, and the heading path, starting character position and ending character position of each chapter are recorded.

[0093] Adaptively configure slice budget parameters based on document language type:

[0094] For Chinese documents: target length is 5200 characters, maximum length is 6000 characters, and overlap length is 600 characters;

[0095] For English documents: target length is 9000 characters, maximum length is 11000 characters, and overlap length is 900 characters;

[0096] For mixed-language documents: target length is 7000 characters, maximum length is 8500 characters, and overlap length is 800 characters.

[0097] For chapters exceeding the maximum length, an overlapping sliding window algorithm is used to generate chapter witness slices:

[0098]

[0099] In the formula, For the overlap length, each slice records the document identifier, title path, slice number, and character start and end positions for subsequent tracing. For the first The original text starting character index of each slice, For the first The index of the original text end character of each slice.

[0100] Activation experts target each slice Output slice-level control signals, the slice-level control signals including:

[0101] Accept slice: If the current slice has sufficient information density, proceed to the evidence card generation process;

[0102] Reject slice: If the current slice has insufficient information density, skip it and process the next slice;

[0103] Termination of Subsequent Actions: If the current slice and subsequent content are determined to have deviated from the topic, the remaining reasoning in the current document is terminated in advance, and the full-link audit is initiated directly.

[0104] For each accepted slice, a structured evidence card and corresponding verification question-and-answer pairs are generated. The data model of the structured evidence card is strictly limited to include: a unique card identifier (ID), a content verification field, and a structured source tracing field. The content verification field contains a claim field, an excerpt field, and a self-contained checksum; the structured source tracing field contains a document identifier, a multi-level path list of chapters, and the character starting index of the slice in the original text. and end index .

[0105] By using the index intervals in the structured tracing domain, a system is established that traces evidence cards, chapter slices, and the physical location of the original text. A three-level complexity tracing mapping mechanism ensures that every conclusion is verifiable and locatable.

[0106] Verify that the question-and-answer pairs are generated from the same source as the evidence cards, record the questions, answers, and associated evidence identifiers, and provide a basis for subsequent verification.

[0107] S4. Joint verification closed loop and linked rejection based on evaluation matrix

[0108] Constructing a three-dimensional verification and evaluation matrix The authenticity of the evidence cards was quantitatively evaluated using large language models. Self-containment And verify the consistency between the questions and answers and the evidence cards. .

[0109] The evidence cards and verification questions are jointly verified in two stages, without directly outputting the initial generated results.

[0110] Phase 1 Verification: Verifying the authenticity of the evidence cards. That is, consistency with the original text; and self-containment. It can be understood independently without relying on other slices;

[0111] Second-stage verification: Verifying the consistency between the question-and-answer pair and the supporting evidence card. The correct answer to the question can be completely derived from the corresponding evidence card.

[0112] Finally, the decision is made if and only if the matrix... The verification is considered successful when all evaluation values ​​are greater than the set confidence threshold.

[0113] For entries that are determined to be revised or rejected, a verification opinion containing the distribution of revised features is generated and fed back to the activating expert. After a second generation is performed, the verification is re-verified. A single entry can undergo a maximum of two rounds of verification to avoid infinite loops.

[0114] After final verification, a linked rejection constraint is executed: if a verification question-and-answer pair passes its own verification, but all its associated evidence cards fail in the final verification result, the verification question-and-answer pair is automatically marked as rejected, the reason for rejection is recorded and included in the audit, ensuring that the final output results are supported by valid underlying evidence. The final output consists of all verified structured evidence cards and verification question-and-answer pairs, completing the full closed loop from input to highly reliable result output for a single long document.

[0115] S5, Expert State Increment Extraction and Memory Snapshot Evolution

[0116] Extract expert state increments based on final verification results Extract success patterns from verified evidence cards and verification question-and-answer pairs; extract failure patterns and corrective experiences from revision and rejection records; and generate a set of boundary rules by combining the processing boundaries.

[0117] Increment the state Snapshots of memories from historical experts The merge process begins with precise deduplication, followed by near-duplicate filtering based on the Jaccard similarity of the token sets. The similarity calculation formula is as follows:

[0118]

[0119] Given two memory entries, the token set is defined as follows: when the similarity is... When the threshold is exceeded, the trigger patterns are merged, and only one valid high-value pattern is retained to avoid memory redundancy.

[0120] In the preset token budget Under constraints, dynamic prioritization is performed on the merged memories. The dynamic value score for each memory is calculated in real time, using the following formula:

[0121]

[0122] In the formula, For memory type weights, failure mode weights are preset based on experience scarcity. Greater than the weight of the success mode ; The time difference from when this memory snapshot was generated to the present. The time decay constant, and This is the adjustment coefficient.

[0123] When the total memory volume after merging exceeds the preset token budget At that time, according to Perform the removal process from low to high until the total volume is reached. The pruning order is failure mode > success mode > revision rules > boundary rules, prioritizing the retention of high-value error avoidance and correction experience.

[0124] For example, failure mode baseline weights Successful mode benchmark weight Time decay constant Preset Token Budget The preferred value is 8192 Tokens.

[0125] After cropping, an updated expert memory snapshot is generated. :

[0126]

[0127] The memory snapshot includes success mode, failure mode, boundary rules, revision rules and source identifier, and is written back to the corresponding expert object for subsequent processing of similar documents, forming a dynamic expert constraint self-evolution closed loop of verification-driven, incremental extraction, compression update and experience reuse.

[0128] S6, End-to-End Auditing and Status Card Export

[0129] Audit the entire execution process, record all events from document access, summary generation, routing decision, chapter slicing, evidence generation, verification and revision, linkage rejection, incremental extraction and memory update, and store model call requests, response content, execution time and exception logs.

[0130] After processing, the document status summary, routing object, activated expert, slice data, evidence card, question-and-answer pair, verification record, memory snapshot and running event are persistently stored, and a structured status audit card is exported. The status audit card includes running details, profile context, expert routing decision trajectory, slice control signal record, verification flow history, revision opinions, linkage rejection record, model call original transcription and usage statistics, so as to realize the traceability, replayability and auditability of the entire processing process.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing long documents, characterized in that, include: Acquire long document data to be processed, extract document structure information, and construct a unified document object; Before the block processing, the global semantic features of the document are extracted based on the unified document object to generate a navigation-style document state summary, which serves as a global constraint for subsequent processing. A route identifier is generated based on the navigation document state summary, and the historical expert database is matched based on the route identifier; if no match is found, the matching degree of the candidate experts is evaluated based on the navigation document state summary, and the historical experts are reused or new experts are derived based on the evaluation results to establish the currently active expert. The activation expert is an intelligent agent object that has domain document processing capabilities and can accumulate processing experience; The activation expert, based on the unified document object, divides the full text into chapters according to the document structure and generates chapter slices; for valid slices, a structured evidence card containing a conclusion statement, an excerpt from the original text, and source location information, and a corresponding verification question-and-answer pair are generated, and a source mapping relationship between the evidence card and the original text location is established; The structured evidence cards and their corresponding verification question-and-answer pairs are jointly verified; when all the evidence cards on which the verification question-and-answer pair depends fail verification, the verification question-and-answer pair is removed. The extraction of document structure information includes extracting the document's title hierarchy, full text, and page layout metadata; The global semantic features include language, domain, topic, core issues, processing methods, keyword set, main research findings / core conclusions, global semantic summary, and chapter summary; When extracting global semantic features based on the unified document object, if the external model call fails, the first N high-frequency keywords, metadata, and title texts at all levels of the document are automatically extracted and a baseline state summary is generated by concatenating rule templates to ensure processing continuity; where N is 5–10. The route identifier is a route fingerprint; the route identifier is obtained by semantic embedding multiple core features of the navigation document state summary to obtain a summary feature vector, and the vector is reduced in dimension by a perceptual hash algorithm to generate a fixed-length bit feature string. The matching degree evaluation is based on a multi-dimensional coverage score of vector similarity, topic overlap, task overlap, problem overlap, method overlap, domain overlap and keyword overlap, and the sum of the weight coefficients of topic, task, problem and method overlap is greater than or equal to 0.

6. The historical expert is reused only when the score of the best candidate expert reaches the preset total score threshold, the margin of the lead over the second best candidate reaches the preset threshold, and the maximum value of the overlap of topic, task, problem, and method reaches the preset core overlap threshold; otherwise, a new expert object is generated; the generated new expert object is initially marked as a temporary expert object, and is promoted to an active expert object after being reused or after accumulating a preset number of selections. The long document processing method further includes: extracting expert experience increments based on the current verification results; merging, deduplicating, and compressing the experience increments with historical expert memories under a preset memory capacity constraint; updating the memories of the activated experts; the deduplication is based on the Jaccard similarity of the token set for near-duplicate filtering, and triggering pattern merging when the similarity is greater than a preset threshold; the compression uses a dynamic priority scoring function based on decay weight to evaluate and prune memory entries, and the scoring mechanism comprehensively considers memory type weights and time decay factors to determine the value of each memory, wherein the weight of failure mode memories is greater than the weight of success mode memories; The long document processing method further includes: persistently storing document state summaries, routing decision trajectories, slice control signals, evidence cards, verification question-and-answer pairs, expert memory updates, and model call records during the processing, and generating structured state audit cards.

2. The long document processing method according to claim 1, characterized in that, The chapter slices are adaptively configured with a slice length budget and overlap length based on the document language type; the target length for Chinese documents is 5200 characters, the maximum length is 6000 characters, and the overlap length is 600 characters; the target length for English documents is 9000 characters, the maximum length is 11000 characters, and the overlap length is 900 characters; and the target length for mixed-language documents is 7000 characters, the maximum length is 8500 characters, and the overlap length is 800 characters. The activation expert outputs slice-level control signals for each slice, including accepting the slice, rejecting the slice, and terminating subsequent slices.

3. The long document processing method according to claim 1, characterized in that, The data model of the structured evidence card includes a unique card identifier, a content verification field, and a structured source tracing field; the content verification field includes a conclusion statement field, a text excerpt field, and a self-contained check bit; the structured source tracing field includes a document identifier, a multi-level path list of chapters, and the character start and end indexes of the slice in the original text.

4. The long document processing method according to claim 1, characterized in that, The joint verification adopts a two-stage verification method: the first stage verifies the authenticity and self-containment of the evidence card, and the second stage verifies the consistency between the question and answer pair and the evidence card; for items that fail the joint verification, verification opinions are generated and fed back to the activated expert for secondary generation, with a maximum of two rounds for a single item; The reason for rejection of the removed verification questions and answers is recorded.

5. A long document processing system, used to execute the long document processing method according to any one of claims 1-4, comprising a document parsing unit, a content processing unit, and a result output unit, characterized in that, It also includes a global constraint generation module, an expert route adaptation module, a traceable content generation module, and a joint verification closed-loop module; The global constraint generation module is used to perform standardized parsing on the long document to be processed and generate a unified document object. Before performing local decomposition and block processing on the unified document object, global semantic and structural modeling is performed on the unified document object to generate a navigation-style global state summary covering the core semantic and structural features of the document. The global state summary is used as the global semantic anchor point and boundary constraint for a single processing step in the entire process. The expert routing adaptation module is communicatively connected to the global constraint generation module. It is used to generate a routing identifier that represents the core processing requirements of the document based on the global state summary, and to perform matching in the pre-built historical expert database using the routing identifier. If a match is found, the corresponding historical expert is reused as the currently active expert. If no match is found, the matching degree of candidate experts is evaluated based on the global state summary. The best candidate expert is reused or a new expert adapted to the current document is derived according to the evaluation result to establish the currently active expert. The activation expert is an intelligent agent object with document processing capabilities in the corresponding domain; The traceable content generation module is communicatively connected to the global constraint generation module and the expert routing adaptation module, respectively. It is used to call the activated expert, and based on the global state summary, the unified document object is hierarchically decomposed according to the document structure to generate chapter slices with positioning identifiers. For the effective chapter slices, the core information is extracted to generate structured evidence cards and corresponding verification question-and-answer pairs. The structured evidence cards include conclusion statements, supporting content of the original text and traceability positioning information, and a traceable mapping relationship is established between the evidence cards and the corresponding content of the original document. The joint verification closed-loop module is communicatively connected to the traceable content generation module and the result output unit, respectively. It is used to perform multi-dimensional joint verification on the structured evidence cards and their corresponding verification question-and-answer pairs. The content is judged as valid only when all verification dimensions meet the standards. The module performs closed-loop correction on the content that fails the verification. When all the evidence cards on which a verification question-and-answer pair depends fail the verification, the verification question-and-answer pair is automatically removed. Finally, the verified structured evidence cards and verification question-and-answer pairs are sent to the result output unit to complete the long document processing.

Images