Generative document analysis system evaluation method and device

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a generative document analysis system evaluation method and device. The device comprises a content fidelity evaluation module, a word level diagnosis module, a structure and semantic evaluation module and a report generation module. The content fidelity evaluation module adopts a semantic alignment editing distance to quantify semantic consistency; the word level diagnosis module distinguishes content omission and illusion through a token discovery rate and a token addition rate; the structure and semantic evaluation module specially evaluates the table structure and document level understanding ability; and finally generating a comprehensive evaluation report. According to the method, the problem of misjudgment of structural diversity in a traditional evaluation method is solved, 12%-25% of average scoring errors are corrected in a test, and multi-dimensional and interpretable system performance evaluation is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an evaluation method and apparatus for a generative document parsing system. Background Technology

[0002] With the rapid development of multimodal generative document parsing systems, traditional evaluation methods are no longer sufficient to accurately assess the output quality of such systems. Existing document parsing evaluation methods mainly rely on the following four categories: 1. Layout analysis and evaluation: Computer vision metrics such as intersection-union ratio and average precision are used (Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.), but the semantic correctness of the content cannot be evaluated; 2. Text extraction evaluation: Using edit distance-based metrics such as character error rate and word error rate (JunichiKanai, Thomas A Nartker, S Rice, and George Nagy. Performance metrics for document understanding systems. In Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR'93), pages 424–427. IEEE, 1993.), but assuming deterministic output and unable to handle semantically equivalent structural changes; 3. Table structure evaluation: The tree edit distance similarity index was adopted (Xu Zhong, Elaheh Shafiei Bavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564–580. Springer, 2020.), but a single correct interpretation was still assumed; 4. Document structure evaluation: Based on Levenshtein distance and pairwise ordering accuracy (Christian Clausner, Stefan Pletschacher, and Apostolos Antonacopoulos. The significance of reading order in document recognition and its evaluation. In 2013 12th International Conference on Document Analysis and Recognition, pages 688–692. IEEE, 2013.), but lacks tolerance for multiple effective reading paths in complex documents.

[0003] These traditional methods have four fundamental limitations: (1) deterministic assumptions that ignore the multiple valid interpretations in document parsing; (2) format rigidity that makes it impossible to make meaningful comparisons across different representation formats; (3) semantic blind spots that fail to identify representational changes that preserve semantics; and (4) spatial inflexibility that lacks tolerance for minor spatial changes that preserve semantics.

[0004] Therefore, there is an urgent need in this field for an evaluation framework that can accommodate interpretive diversity while maintaining semantic rigor, so as to fairly compare the performance of modern generative document parsing systems. Summary of the Invention

[0005] This invention aims to solve the problem of evaluation distortion in existing document parsing evaluation methods when processing the output of multimodal generation systems. In particular, when the system produces semantically correct but structurally different outputs, traditional methods will incorrectly classify valid interpretations as errors, leading to distorted system rankings and misjudgments of performance. This invention provides a generative document parsing system evaluation method and apparatus.

[0006] To achieve the above objectives, the present invention provides an evaluation method for a generative document parsing system, comprising the following steps: (1) Obtain the output and true values ​​of the generative document parsing system to be evaluated, and calculate the semantic alignment edit distance between the output and the true values ​​to determine the semantic consistency of the content; (2) When the semantic alignment edit distance score is lower than the set threshold, fine-grained token analysis is initiated, and the problem is classified by calculating the token discovery rate and the token addition rate; (3) For each problem type, evaluate the generative document parsing system’s ability to understand table structure and organize document hierarchy; integrate the evaluation results and generate a comprehensive evaluation report.

[0007] Further, step (1) includes: The output of the generative document parsing system to be evaluated is preprocessed with the actual values, including text cleaning, tokenization, and sequence alignment; Calculate the standard normalized edit distance between the preprocessed output and the true value; Identify document elements and categorize them as tables, paragraphs, or figure captions; Calculate the weighted similarity of elements in each class; Based on the standard normalized edit distance and the adjusted score, the semantic alignment edit distance between the output and the true value is calculated. If the semantic alignment edit distance is not lower than the set threshold, a comprehensive evaluation report will be generated directly; otherwise, word-level diagnosis will be performed.

[0008] Furthermore, the adjusted score , where each type of element , Weighted similarity of elements The sum of weights; The semantic alignment edit distance ,in Indicates the output of the system being evaluated True value Standardized edit distance.

[0009] Furthermore, the semantic alignment edit distance introduces a structural alignment compensation mechanism; for table elements, a word-weighted fuzzy matching strategy is adopted, allowing semantically equivalent but structurally different representations; for paragraph text, standard normalized edit distance calculation is retained; for caption elements, a hybrid evaluation combining standard normalized edit distance and semantic similarity is used.

[0010] Furthermore, in step (2), the token discovery rate is Token addition rate ;in, Output multiple sets for the system being evaluated The frequency of occurrence of each token t in the array. For real values ​​of multiple sets The frequency of occurrence of each token t in the array; If the token discovery rate is <0.7 and the token addition rate is ≤0.1, the problem is classified as content omission-dominated; if the token discovery rate is ≥0.7 and the token addition rate is >0.3, the problem is classified as illusion-dominated; otherwise, the problem is classified as mixed.

[0011] Furthermore, if the problem type is content omission-dominant or mixed, the generative document parsing system's ability to understand table structure is evaluated first, with a focus on checking the completeness of table content extraction; if the token addition rate is >0.3, the generative document parsing system's ability to understand table structure and document hierarchy organization is evaluated, and it is checked whether the illusion is caused by structural misunderstanding.

[0012] Further, in step (3), the ability of the generative document parsing system to understand table structures is evaluated, including: All formats of the raw output of the system being evaluated will be mapped to a unified semantic representation; By calculating the content similarity between all the output tables of the evaluated systems and the true value tables, the precision and recall rates are calculated based on the content similarity, and then the detection score is calculated as an evaluation indicator of the table detection capability. Calculate content accuracy and index accuracy; the content accuracy is used to measure whether the evaluated system correctly extracts the text content in each cell; the index accuracy is used to measure whether the evaluated system correctly understands the spatial logical relationship between cells; The hierarchical integrity of a table is assessed using tree edit distance similarity.

[0013] Further, in step (3), the ability of the generative document parsing system to organize document hierarchy is evaluated, including: Based on the original tag set of the evaluated system, a semantic dictionary is used to map heterogeneous tags to standard functional categories, resulting in a unified tag sequence; Calculate the overall similarity to establish the output of the system being evaluated. True value The correspondence; Construct a confusion matrix and calculate a consistency score based on the confusion matrix; If the consistency score is low but the token discovery rate is high, it indicates that the evaluated system extracts complete content but the organizational logic is chaotic.

[0014] To achieve the above objectives, the present invention also provides an evaluation apparatus for a generative document parsing system that implements the above method, comprising: The content fidelity assessment module is used to obtain the output and true values ​​of the generative document parsing system to be evaluated, and to calculate the semantic alignment edit distance between the output and the true values ​​to determine the semantic consistency of the content. The word-level diagnostic module is used to initiate fine-grained token analysis when the semantic alignment edit distance score is lower than a set threshold. It classifies the problem by calculating the token discovery rate and token addition rate. The structure and semantic evaluation module is used to evaluate the generative document parsing system’s ability to understand table structures and organize document hierarchies for different question types. The report generation module is used to integrate the various assessment results and generate a comprehensive assessment report.

[0015] Furthermore, the structure and semantic evaluation module includes a table evaluation submodule and a structure hierarchy understanding submodule; The table evaluation submodule is used to evaluate the table structure parsing capability of the generative document parsing system, separating the accuracy of content recognition and spatial relationship understanding; The structural hierarchy understanding submodule is used to evaluate the overall logical organization capability of the generative document parsing system and detect the consistency of element tag assignment.

[0016] This multi-dimensional, hierarchical design enables the present invention to comprehensively, fairly, and interpretably evaluate the true capabilities of modern generative document parsing systems.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) Improved assessment accuracy: In the test of 1743 pages of documents, the present invention successfully identified 2% to 5% of the pages with effective explanatory diversity, correcting the average scoring error of 12% to 25% of the traditional method; (2) Multidimensional diagnosis: Provides separate indicators such as content accuracy, index accuracy, and hallucination rate; (3) Achieve efficiency optimization: Traditional evaluation indicators can be reproduced without the need for traditional target detection processes. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a schematic diagram of the overall processing flow of the present invention; Figure 2 This is a schematic diagram of the content fidelity assessment process in this invention; Figure 3 This is a flowchart illustrating the word-level diagnosis process in this invention. Figure 4 This is a schematic diagram of the structural and semantic evaluation process in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] The overall architecture and process of this invention are as follows: Figure 1 As shown, this process employs a multi-stage progressive analysis strategy, from basic content fidelity assessment to fine-grained diagnosis, ultimately generating a comprehensive assessment report. This invention is applicable to document intelligent processing, information extraction, and retrieval enhancement systems. The execution process of each step is described in detail below: Step 1: Content Fidelity Assessment. The content fidelity assessment module is the first line of defense for the entire device, and its function is to quickly locate the problematic page. It answers a basic question: "Does the semantic content output by the system match the actual value?" If the score is very low, it indicates that there is obvious content missing, error, or redundancy, but it does not specifically point out the root cause of the problem.

[0023] This stage, serving as the foundational evaluation layer, aims to quantify the semantic consistency between the output of the generative document parsing system and the true values. Specifically, the Semantically-Aligned Edit Distance (SAED) algorithm proposed in this invention is used as the quantification method. This algorithm introduces a structural alignment compensation mechanism based on traditional edit distance: for table elements, a word-weighted fuzzy matching strategy is adopted, allowing semantically equivalent but structurally different representations (such as changes in row and column order); for paragraph text, standard normalized edit distance calculation is retained; for specific elements such as figure captions, a hybrid evaluation combining standard normalized edit distance and semantic similarity is used.

[0024] After semantic consistency quantification is completed, the completeness and accuracy of the content are judged based on the SAED score: if the SAED score is higher than the preset threshold (typical value is 0.85), it is judged as "passed" and directly enters the report generation stage; if a significant difference is detected (score is lower than the threshold), it enters the subsequent diagnostic process.

[0025] The second step is word-level diagnosis. When the content fidelity assessment module detects anomalies, the word-level diagnosis module acts like a "microscope," delving deep into the text to precisely analyze the root cause of the content problem and accurately distinguish whether it stems from "omission" or "illusion." This step further categorizes the "problems" discovered by the content fidelity assessment into "what type of problem," providing a clear direction for subsequent analysis.

[0026] When discrepancies are detected in the content fidelity assessment, fine-grained token analysis is initiated at this stage to differentiate the essential type of the problem. Based on the "token discovery rate" and "token addition rate" indicators proposed in this invention, the diagnostic process employs a two-way evaluation strategy: Token discovery rate: Measures the proportion of genuine tokens successfully retained in the output of the evaluated system, independent of the token order. It directly reflects the system's coverage of the source document content and is used to identify content omissions. The closer the value is to 1, the more completely the evaluated system retains the genuine content; the lower the value, the more severe the content loss.

[0027] Token Addition Rate: Measures the proportion of extra tokens in the output of the evaluated system that exceed the true value, quantifying the level of illusion. The closer the value is to 0, the more faithful the evaluated system is to the source document, and the less illusion there is; the higher the value, the more unsupported or fictitious content the system introduces.

[0028] The word-level diagnostic module outputs specific problem location results, such as "content omission-dominated", "hallucination-dominated", or "hybrid", providing guidance for subsequent special assessments.

[0029] The third step is structural and semantic assessment. The structural and semantic assessment module is responsible for diagnosing high-level understanding capabilities. It is a deeper "capability audit" that specifically assesses the system's understanding of document logic and organizational structure.

[0030] This phase assesses the ability to understand deeper structural issues identified in previous steps, and includes two parallel sub-modules: The table evaluation submodule possesses specialized diagnostic capabilities for understanding table structures. If a document contains tables, this submodule diagnoses whether the system understands the table's internal logic (row and column structure, cell content), and not merely extracts text. It first normalizes heterogeneous outputs (such as HTML and JSON) into a unified semantic representation, then calculates content accuracy and index accuracy using a cell alignment algorithm, and integrates Tree Edit Distance Similarity (TEDS) to assess hierarchical structure preservation capabilities. This submodule specifically introduces a spatial tolerance mechanism, allowing cell positions to offset within a preset range (typically ±2 cells).

[0031] The structural hierarchy understanding submodule assesses the overall document logical organization capability, specifically whether the system understands the macro-level organizational structure of the entire document, such as the hierarchical relationships between headings, paragraphs, and lists. It normalizes system-specific tags (such as "heading" and "subheading") into standard categories through functional category mapping, and then calculates element consistency scores based on a confusion matrix, reflecting the assessed system's ability to maintain semantic coherence in complex layouts.

[0032] Step 4: Generate a comprehensive evaluation report.

[0033] If the evaluated system performs well in the word-level diagnostic module but scores poorly in the structure evaluation, it means that the system can extract all the text but fails to understand the logical relationships between them (e.g., misreading tables as plain text or confusing heading levels). This explains why sometimes the content fidelity is acceptable, but the output is structurally unusable.

[0034] This stage integrates the results from all evaluation dimensions to generate a structured evaluation report. The report includes: (1) Multi-dimensional scoring: content fidelity score, token discovery rate, token addition rate, table content / index accuracy, element consistency and other quantitative indicators; (2) Problem identification analysis: clearly point out the specific strengths and weaknesses of the evaluated system in terms of content integrity, illusion control, and structural understanding; (3) Improvement suggestions: Provide targeted optimization directions based on the diagnostic results (such as enhancing the perception of table structure, reducing specific types of hallucinations, etc.). Through the aforementioned hierarchical and progressive evaluation strategy, a comprehensive diagnosis can be achieved, ranging from surface content consistency to deep structural semantic understanding, ensuring the accuracy and operability of the evaluation results. This multi-dimensional and hierarchical design enables the present invention to comprehensively, fairly, and interpretably evaluate the true capabilities of modern generative document parsing systems. Those skilled in the art can implement this evaluation operation based on the described process and in conjunction with a specific programming environment.

[0035] 1. Content fidelity assessment.

[0036] Content fidelity assessment, as the core initial stage of this invention, aims to quantify the semantic consistency between the output of the evaluated system and the true value, while tolerating reasonable structural variations. It employs the semantic alignment edit distance algorithm proposed in this invention, which addresses the problem of excessive penalty for legitimate interpretation diversity in traditional edit distance methods by combining surface sequence similarity and structure-aware semantic alignment. See also Figure 2 The detailed steps for content fidelity assessment are as follows: (1.1) Input preprocessing.

[0037] First, receive the output of the system being evaluated. and the true value Both can be text sequences or structured data (such as HTML or JSON). Then, the input is normalized, specifically including: text cleaning (consistent capitalization, removal of extra spaces and special characters), tokenization (using BPE tokenizer, with options for WordPiece or SentencePiece, and a minimum token length of 2 to ensure appropriate granularity), and sequence alignment (flattening the structured data into a token sequence while preserving element boundary information).

[0038] (1.2) Calculation of standard normalized edit distance.

[0039] The standard normalized edit distance between the preprocessed output of the evaluated system and the ground truth value is calculated as a baseline similarity measure. The calculation formula is as follows:

[0040] in, The Levenshtein distance is calculated based on the character or token sequence. and These are the lengths of the output and the actual value of the system being evaluated, respectively (in tokens).

[0041] (1.3) Document element identification and classification.

[0042] The system automatically identifies the types of functional elements in a document, categorizing them into three types: table elements (data with a row and column structure, including cell content and table headers), paragraph elements (continuous blocks of text, such as body text and list items), and figure caption elements (images, table titles, or descriptive text). The classification is based on a rule engine or a pre-trained layout model (such as a YOLO-based detector) to ensure accurate element boundaries.

[0043] (1.4) Calculation of element-specific similarity.

[0044] For each type of element (in ={tables, paragraphs, figure captions}), calculate weighted similarity. :

[0045] in, This is a weighting factor, set according to the importance of different elements. For example, set the weight of a table cell to 1.2, a heading to 1.1, and a paragraph to 1.0. The weighting factor is adjustable to suit different document types (e.g., the weight of tables can be increased in financial documents). For the similarity function, for table elements, word-weighted fuzzy matching is used (the table content is normalized into a set of tokens, and Jaccard similarity or cosine similarity is calculated based on the intersection of the tokens, with a similarity threshold of 0.8, i.e., similarity ≥ 0.8 is considered semantically equivalent); for paragraph elements, standard normalized edit distance is used to calculate the text similarity within the paragraph, ignoring minor format differences; for caption elements, edit distance (weight 0.4) and semantic similarity (cosine similarity based on Sentence-BERT embedding, weight 0.6) are used to capture text semantics and surface form; It is a set representing all items classified as number 1. Instances of elements of a class, such as (Table) = {Table1, Table2, Table3}. It is an index variable used for iteration. All element instances in, such as If there are 3 elements, then ∈{1,2,3}. for The Middle Elements, for example If (table) = {table1, table2, table3}, then Representing the A table.

[0046] (1.5) Adjust the score calculation.

[0047] Based on this, the adjustment score is calculated. Semantic fidelity after structural tolerance of the reaction:

[0048] in, It is the sum of all weights, used for normalization. It is an index variable used to iterate through all variables classified as number 1. Instances of elements of a class This represents the weight of each element in the document. For example, if a document has 2 tables (each with a weight of 1.2), 1 heading (with a weight of 1.1), and 5 paragraphs (each with a weight of 1.0), then... =1.2+1.2+1.1+1.0×5=9.5. This score emphasizes the degree of content retention, rather than strict sequence matching.

[0049] (1.6) Semantic alignment edit distance calculation.

[0050] The maximum value of the standard normalized edit distance and the adjustment score is taken as the semantic alignment edit distance. :

[0051] If structural reorganization results in a low standard normalized edit distance but semantic consistency (high adjustment score), the final SAED score will still reflect high-quality output, avoiding false penalties.

[0052] (1.7) Threshold judgment and output.

[0053] Based on historical data or application requirements, a semantic alignment edit distance threshold is set (typically 0.85, but can be set to 0.9 for high-precision scenarios). If SAED ≥ 0.85, the content is deemed "complete and accurate," and the evaluation process jumps directly to the report generation stage. Otherwise, a significant difference is deemed, and the process proceeds to the subsequent word-level diagnostic stage. The output of this stage includes the semantic alignment edit distance score, element-level similarity decomposition, and a pass / fail status flag.

[0054] 2. Word-level diagnosis.

[0055] Word-level diagnostics, as a fine-grained extension of content fidelity assessment, aims to accurately distinguish between content omissions and illusions through token-level frequency analysis. This stage is automatically activated when the content fidelity assessment detects a significant difference (SAED < 0.85), performing multi-set comparisons between the system's output and the actual values ​​to quantify token retention and addition behavior, thereby pinpointing the root cause. See also Figure 3 The detailed execution process for word-level diagnosis is as follows: (2.1) Input reception and initialization.

[0056] Receive the output of the evaluated system marked as "Difference Exists" from the content fidelity assessment phase. ) and the true value ( Both have been preprocessed into token sequences. The tokenization standard uses BPE segmentation, with a minimum token length of 2.

[0057] Converting token sequences into a multiset of frequency statistics: a multiset of real values (Record the frequency of occurrence of each token t) The evaluated system output multiple sets (Record the frequency of occurrence of each token t) When counting token frequencies, case differences are ignored, but the original form of numbers and symbols is preserved (e.g., “$100K” and “$100k” are considered different tokens).

[0058] (2.2) Token discovery rate calculation.

[0059] Token discovery rate quantifies the degree to which the evaluated system retains true content, identifying content omissions. Its calculation formula is: Token discovery rate =

[0060] A token discovery rate ≥ 0.9 indicates that the content is complete and the omissions are minor; a token discovery rate ∈ [0.7, 0.9) indicates that some content is missing and requires attention to medium-risk omissions; a token discovery rate < 0.7 indicates that the content is seriously missing and is marked as a high-risk issue.

[0061] (2.3) Token addition rate calculation.

[0062] The token addition rate quantifies the proportion of illusory content in the output of the evaluated system, detecting unsupported or fictitious tokens. Its calculation formula is as follows: Token addition rate =

[0063] A token addition rate ≤ 0.1 indicates that the illusion control of the output content of the evaluated system is good and the addition problem is minor; a token addition rate ∈ (0.1, 0.3] indicates that there is a moderate illusion and the generation strategy needs to be optimized; a token addition rate > 0.3 indicates that there is a severe illusion and is marked as a high-risk problem.

[0064] (2.4) Problem type identification and data transfer.

[0065] Based on the threshold results of token discovery rate and token addition rate, the problem types are classified as follows: Content omission-dominated type: Token discovery rate < 0.7 and token addition rate ≤ 0.1. This indicates that the evaluated system failed to fully extract the content of the source document, possibly due to reading path deviation or element detection failure.

[0066] Illusion-dominated type: Token discovery rate ≥ 0.7 and token addition rate > 0.3. This indicates that the evaluated system generates excessive or fabricated content, possibly due to model overfitting or contextual misunderstanding.

[0067] Hybrid type: Other cases (e.g., token discovery rate < 0.7 and token addition rate > 0.3). This indicates that the evaluated system suffers from both content loss and illusion, requiring comprehensive optimization.

[0068] Output results: Numerical score (specific values ​​for token discovery rate and token addition rate), issue type label (e.g., "content omission-dominated"), and risk level (low, medium, or high risk).

[0069] Data delivery: The diagnostic results are output to the subsequent "structural and semantic assessment" stage to guide specialized analysis (such as table assessment or structural hierarchy understanding). At the same time, the results are integrated into the final comprehensive assessment report, providing fine-grained improvement suggestions (e.g., "improving the completeness of table cell extraction" or "reducing the generation of duplicate content").

[0070] 3. Structural and semantic evaluation.

[0071] As the in-depth analysis stage of this framework, for the problem types of word-level diagnosis and location, the special evaluation system assesses the ability of the system in table structure understanding and document-level organization. This stage includes two parallel and complementary evaluation sub-modules: the table evaluation sub-module and the structural hierarchy understanding sub-module, which provide evaluation insights from the perspectives of local structure accuracy and global semantic consistency respectively. See Figure 4 , the detailed execution process of the structure and semantic evaluation stage is as follows: (3.1) Input and routing.

[0072] Receive the results and original data from word-level diagnosis, including problem type labels, token discovery rates, and token addition rate values.

[0073] If the problem type is "content omission-dominated" or "mixed", the "table evaluation sub-module" is preferentially activated to focus on checking the integrity of table content extraction. If the token addition rate > 0.3 (significant hallucination), the "structural hierarchy understanding sub-module" is also activated to check whether hallucination is caused by structural misunderstanding. By default, the two sub-modules execute in parallel to comprehensively cover structural semantic problems.

[0074] (3.2) Table evaluation sub-module.

[0075] The table evaluation sub-module is used to evaluate the table structure parsing ability, separating content recognition accuracy and spatial relationship understanding. Its execution steps are as follows: (3.2.1) Format-independent normalization.

[0076] Input: Map all formats (HTML, JSON, text, etc.) of the original output of the system to be evaluated into a unified semantic representation (a sequence of cell tuples).

[0077] Example: HTML: Header→(row=0,col=0,row_span=2,content="Header") The corresponding Chinese meaning is: The HTML content Header will be mapped to (row = 0, col = 0, rowspan = 2, content = "Header").

[0078] JSON: {"cell":[0,0],"content":"Data"}→(row=0,col=0,content="Data") The corresponding Chinese meaning is: The JSON content {"cell": [0,0], "content": "Data"} will be mapped to (row = 0, col = 0, content = "Data").

[0079] Text table: Parse it into tuple form through row and column alignment algorithms.

[0080] (3.2.2) Semantic-first table detection.

[0081] First, calculate the content similarity between the output tables of all evaluated systems and the ground truth tables. Content similarity is calculated through text comparison (such as Jaccard similarity or TF-IDF cosine similarity), ignoring structural differences. A table output by the evaluated system with a content similarity > 0.7 is considered a true positive (TP). A table output by the evaluated system but not in the ground truth table (i.e., content similarity ≤ 0.7) is considered a false positive (FP). A table in the ground truth table but not output by the evaluated system or with a content similarity ≤ 0.7 is considered a false negative (FN). Then, calculate TP, FP, and FN, and then calculate precision (P) and recall (R). Finally, substitute the results into the following formula to calculate the detection score (…). ), used as an evaluation metric for table detection capability, where is a weighting parameter used to balance the importance of precision and recall:

[0082] in, , .

[0083] (3.2.3) Cell-level alignment and space tolerance assessment.

[0084] To tolerate reasonable variations in table structure (such as merging / splitting cells, different interpretations of table headers), a position offset tolerance is introduced during the evaluation, and "content accuracy" and "index accuracy" are calculated separately.

[0085] Content accuracy measures whether the evaluated system correctly extracted the text content within each cell, regardless of its position. It is calculated by treating all predicted and actual cell content as two "documents" and calculating the similarity between them (e.g., an F1 score based on token overlap). This answers the question, "Did the system find all the correct text?"

[0086] Index accuracy measures whether a system correctly understands the spatial logical relationships (row and column structure) between cells. Its calculation method allows for an overall offset of the predicted cell position within a certain range (e.g., ±N rows / columns) to find the alignment that best matches the actual layout. Specifically, it iterates through all possible offset combinations. (For example, , For each offset, predict all cell positions. Unified translation ;in, These represent the amounts of translation made in the row and column directions, respectively, to shift the overall position of the prediction table. These represent the vertical and horizontal position numbers of the cell in the table, indicating the cell's row and column number. Then, the number of cell contents matching at the same position between the predicted and actual layouts after translation is calculated. Index accuracy is the maximum number of matches across all possible offsets.

[0087] (3.2.4) TEDS integration evaluation.

[0088] Tree edit distance similarity (TEDS) is used to assess the hierarchical integrity of the table as a supplement to the above assessment.

[0089] (3.3) Structural hierarchy understanding submodule.

[0090] The structural hierarchy understanding submodule is used to evaluate the overall logical organization of a document and detect the consistency of element tag assignment. Its execution steps are as follows: (3.3.1) Tag category mapping.

[0091] The input is the original set of labels of the system being evaluated. Heterogeneous labels are mapped to standard functional categories, and a unified label sequence is output for subsequent analysis.

[0092] The step of "mapping heterogeneous tags to standard functional categories" is performed based on a predefined tag normalization mapping table. This mapping table defines the correspondence between heterogeneous output tags from different parsing systems and a set of unified standard functional categories. For example, the mapping table includes, but is not limited to: mapping "Title", "Heading 1", "Heading", and "Header" to the standard category "TITLE"; mapping "Body Text", "Paragraph", "text", and "p" to the standard category "TEXT"; and mapping "Table", "table", and "tabular" to the standard category "TABLE". This mapping table is an extensible configuration file that supports updates based on newly emerging system tags.

[0093] The normalized mapping table used in this invention is as follows: {"title","header","sub-title"}→"TITLE" {"text","paragraph","body"}→"TEXT" {"table","tabular"}→"TABLE" {"figure","image","chart"}→"FIGURE" (3.3.2) Element matching and alignment.

[0094] A correspondence between predicted elements (pred) and true elements (gt) is established based on comprehensive similarity. Comprehensive similarity ( The formula for calculating ) is:

[0095] in, It is a text semantic similarity based on BERT embedding. It is based on the IoU similarity of elements in spatial location. When the combined similarity of a pair of elements (pred,gt) exceeds a predetermined threshold (e.g., ...), the similarity is calculated based on the spatial location of the elements. When the match is found to be valid, it is considered a valid match.

[0096] (3.3.3) Constructing the confusion matrix.

[0097] This invention uses a confusion matrix to quantify the consistency between the document element tags predicted by the device and the actual element tags.

[0098] Confusion Matrix The definition is as follows:

[0099] in, It is an extended tag set ( x represents a specific element in the set of true values ​​(gt), such as a text block labeled "heading," a "table" area, or a "caption" in a document. y represents a specific element predicted by the device. For example, a "heading" block or a section of "body" text identified by the device. i is the row index of the confusion matrix, representing the true category of the element. j is the column index of the confusion matrix, representing the category predicted by the device. It is a complete set of pairs, which contains all valid matching element pairs, and for each real element that is not matched... Each contains a pair For each predicted element that is not matched Each contains a pair This ensures that every real element and predicted elements All It appears once.

[0100] (3.3.4) Consistency score calculation.

[0101] Calculate the consistency score based on the confusion matrix. The process is as follows: Confusion matrix → (sum by rows / columns) → True positives (TP), false positives (FP), false negatives (FN) for each category → (substitute into precision and recall formulas) → Precision (P) and recall (R) for each category → (substitute into F1 formula) → F1 score for each category → (average) → Consistency score.

[0102] Let the confusion matrix C have a dimension of (1+K)×(1+K), where the first K rows / columns correspond to the standard category and the (K+1)th row / column corresponds to the NoMatch category. Then, the formulas involved in calculating the consistency score based on this confusion matrix are as follows (where, ): True positives (TPs) represent the number of true positives of class i that are correctly predicted as class i. The formula for calculating TPs is:

[0103] False positives (FPs) represent the number of cases where the predicted class is i but the true class is not i. The formula for calculating FPs is:

[0104] False negatives (FN) represent the number of true negatives of class i that were not predicted as class i. The formula for calculating FN is:

[0105] The formula for calculating precision (P) is:

[0106] The formula for calculating recall (R) is:

[0107] The formula for calculating the F1 score is:

[0108] The formula for calculating the consistency score is:

[0109] Where K is the number of standard categories, equal to the extended tag set. Decrease the number of elements by one; For the i-th category The score reflects the average performance of the evaluated system across all categories.

[0110] (3.4) Results integration and diagnostic output.

[0111] Table evaluation outputs: Content accuracy, Index accuracy, Detection score ( ), TEDS score.

[0112] Structural understanding outputs: consistency score, confusion matrix visualization, and major misclassification patterns.

[0113] Typical comprehensive assessment: If the content accuracy is high but the index accuracy is low, it indicates that the evaluated system can identify the content but lacks spatial understanding. If the consistency score is low but the token discovery rate is high, it indicates that the evaluated system extracts complete content but its organizational logic is chaotic. Based on the word-level diagnostic results, targeted improvement suggestions are generated.

[0114] The following example, using the diagnosis of "good structural understanding but with both illusions and omissions," illustrates the form of improvement suggestions.

[0115] Assuming the diagnostic results are: Word-level diagnosis TokensFound=0.72 (significant content omissions exist), TokensAdded=0.21 (moderate hallucination exists); Table evaluation content accuracy=0.68, index accuracy=0.82 (good spatial relationship understanding), TEDS=0.85 (table hierarchical structure is well maintained); Structural hierarchy understanding consistency score=0.78 (good consistency in element classification). An example of improvement suggestions generated by this device is as follows: Diagnostic Logic: The evaluated system exhibited good structural reasoning ability (high indexing accuracy, high TEDS, high consistency), but content fidelity was problematic (low TokensFound, high TokensAdded). This suggests that the system may over-rely on its internal knowledge or patterns for "reasoning generation" rather than strictly adhering to the source document image. Good structural understanding but low content accuracy, especially combined with high TokensAdded, strongly suggests that the system has a tendency to infer content based on incorrect structure or fill in missing information with common patterns; that is, "illusions" and "omissions" may originate from the same generation mechanism.

[0116] Enhanced visual-based constraint generation: It is recommended to introduce stronger visual signal supervision during the decoding process in the generation phase. For example, use attention-guided mechanisms to ensure that the generated text tokens have higher attention weight association with specific text lines or regions in the image, reducing "generating tokens out of thin air".

[0117] Implement confidence level calibration and threshold filtering: assign a confidence score to each content segment (such as a cell or paragraph) output by the system. For content with low confidence, trigger a review mechanism (such as secondary parsing or marking it as pending review) to balance recall and precision.

[0118] Optimize prompts and few-shot examples: For Visual Language Model (VLM) systems, the prompts more explicitly emphasize the instruction to "strictly follow the image content without adding any extra information" and provide few-shot examples with complex layouts and accurate extraction to guide the model's behavior.

[0119] 4. Optional implementation methods.

[0120] In other implementations, the following parameters can be adjusted: The space tolerance can be selected within the range of 1-5 cells; The similarity threshold can be adjusted between 0.6 and 0.9 depending on the document type; Tokenization can be replaced by alternatives such as BPE or SentencePiece; The weighting can be optimized according to specific application scenarios. For example, the weight of tables in financial documents can be increased to 1.5.

[0121] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating a generative document parsing system, characterized in that, Includes the following steps: (1) Obtain the output and true values ​​of the generative document parsing system to be evaluated, and calculate the semantic alignment edit distance between the output and the true values ​​to determine the semantic consistency of the content; (2) When the semantic alignment edit distance score is lower than the set threshold, fine-grained token analysis is initiated, and the problem is classified by calculating the token discovery rate and the token addition rate; (3) For each problem type, evaluate the generative document parsing system’s ability to understand table structure and organize document hierarchy; integrate the evaluation results and generate a comprehensive evaluation report.

2. The evaluation method for a generative document parsing system according to claim 1, characterized in that, Step (1) includes: The output of the generative document parsing system to be evaluated is preprocessed with the actual values, including text cleaning, tokenization, and sequence alignment; Calculate the standard normalized edit distance between the preprocessed output and the true value; Identify document elements and categorize them as tables, paragraphs, or figure captions; Calculate the weighted similarity of elements in each class; Based on the standard normalized edit distance and the adjusted score, the semantic alignment edit distance between the output and the true value is calculated. If the semantic alignment edit distance is not lower than the set threshold, a comprehensive evaluation report will be generated directly; otherwise, word-level diagnosis will be performed.

3. The evaluation method for generative document parsing systems according to claim 2, characterized in that, The adjusted score , where each type of element , Weighted similarity of elements The sum of weights; The semantic alignment edit distance ,in Indicates the output of the system being evaluated True value Standardized edit distance.

4. The evaluation method for a generative document parsing system according to claim 3, characterized in that, The semantic alignment edit distance introduces a structural alignment compensation mechanism; for table elements, a word-weighted fuzzy matching strategy is adopted, allowing semantically equivalent but structurally different representations; for paragraph text, the standard normalized edit distance calculation is retained; for caption elements, a hybrid evaluation combining the standard normalized edit distance and semantic similarity is used.

5. The evaluation method for a generative document parsing system according to claim 1, characterized in that, In step (2), the token discovery rate is Token addition rate ;in, Output multiple sets for the system being evaluated The frequency of occurrence of each token t in the array. For real values ​​of multiple sets The frequency of occurrence of each token t in the array; If the token discovery rate is <0.7 and the token addition rate is ≤0.1, the problem is classified as content omission-dominated; if the token discovery rate is ≥0.7 and the token addition rate is >0.3, the problem is classified as illusion-dominated; otherwise, the problem is classified as mixed.

6. The evaluation method for a generative document parsing system according to claim 5, characterized in that, If the problem type is content omission-dominant or mixed, the generative document parsing system's ability to understand table structure is evaluated first, with a focus on checking the completeness of table content extraction; if the token addition rate is >0.3, the generative document parsing system's ability to understand table structure and document hierarchy organization is evaluated, and it is checked whether the illusion is caused by structural misunderstanding.

7. The evaluation method for a generative document parsing system according to claim 1, characterized in that, In step (3), the ability of the generative document parsing system to understand table structures is evaluated, including: All formats of the raw output of the system being evaluated will be mapped to a unified semantic representation; By calculating the content similarity between all the output tables of the evaluated systems and the true value tables, the precision and recall rates are calculated based on the content similarity, and then the detection score is calculated as an evaluation indicator of the table detection capability. Calculate content accuracy and index accuracy; the content accuracy is used to measure whether the evaluated system correctly extracts the text content in each cell; the index accuracy is used to measure whether the evaluated system correctly understands the spatial logical relationship between cells; The hierarchical integrity of a table is assessed using tree edit distance similarity.

8. The evaluation method for a generative document parsing system according to claim 1, characterized in that, In step (3), the ability of the generative document parsing system to organize document hierarchy is evaluated, including: Based on the original tag set of the evaluated system, a semantic dictionary is used to map heterogeneous tags to standard functional categories, resulting in a unified tag sequence; Calculate the overall similarity to establish the output of the system being evaluated. True value The correspondence; Construct a confusion matrix and calculate a consistency score based on the confusion matrix; If the consistency score is low but the token discovery rate is high, it indicates that the evaluated system extracts complete content but the organizational logic is chaotic.

9. An evaluation apparatus for a generative document parsing system implementing the method as described in claim 1, characterized in that, include: The content fidelity assessment module is used to obtain the output and true values ​​of the generative document parsing system to be evaluated, and to calculate the semantic alignment edit distance between the output and the true values ​​to determine the semantic consistency of the content. The word-level diagnostic module is used to initiate fine-grained token analysis when the semantic alignment edit distance score is lower than a set threshold. It classifies the problem by calculating the token discovery rate and token addition rate. The structure and semantic evaluation module is used to evaluate the generative document parsing system’s ability to understand table structures and organize document hierarchies for different question types. The report generation module is used to integrate the various assessment results and generate a comprehensive assessment report.

10. The generative document parsing system evaluation apparatus according to claim 9, characterized in that, The structure and semantic evaluation module includes a table evaluation submodule and a structure hierarchy understanding submodule; The table evaluation submodule is used to evaluate the table structure parsing capability of the generative document parsing system, separating the accuracy of content recognition and spatial relationship understanding; The structural hierarchy understanding submodule is used to evaluate the overall logical organization capability of the generative document parsing system and detect the consistency of element tag assignment.

Images