A PDF document table data extraction and verification method and system

Abstract

This application discloses a method and system for extracting and validating tabular data from PDF documents. The method includes: performing differentiated preprocessing on PDF document data to obtain standardized text data; parsing the tabular data in the standardized text data according to custom parsing rules, dividing it into index items, and then performing text vectorization; using a weighted retrieval strategy to retrieve the vectorized index items and constructing an initial tabular data matrix; evaluating the initial tabular data matrix based on preset evaluation metrics and dynamically generating scheduling instructions; calling corresponding verification and repair function tools to perform verification and repair operations according to the scheduling instructions, and generating a repaired tabular data matrix; and re-triggering the evaluation process based on the repaired tabular data matrix until the evaluation score meets preset qualification conditions. This application can effectively adapt to the needs of large-scale, high-precision tabular data processing scenarios and significantly improve the accuracy of tabular data extraction.

Description

Technical Field

[0001] This application relates to the field of PDF document data processing technology, specifically to a method and system for extracting and verifying table data from PDF documents. Background Technology

[0002] With the deepening of digital office practices, PDF documents, due to their stable format and strong cross-platform compatibility, have become the mainstream carrier for data storage and transmission in fields such as government documents, corporate financial reports, scientific research reports, and medical records. In these PDF documents, tables, as a structured data presentation format, carry a large amount of key information, and the data they contain often has extremely high analytical and application value. Accurately and efficiently extracting data from tables in PDF documents, and verifying the accuracy and completeness of the extracted data, is a core prerequisite for realizing data resource utilization and supporting subsequent data analysis and decision-making. Related technologies have become a research hotspot and focus in the fields of data mining and information retrieval.

[0003] In the current technological system, PDF table data processing mainly encompasses core stages such as PDF document parsing, table recognition, data extraction, and basic data verification. PDF document parsing technology analyzes the underlying data structure of PDF files to initially extract elements such as text, graphics, and tables. Mainstream technologies include native parsing based on PDF specifications and OCR-based image recognition parsing. The former is suitable for PDF documents with copyable text, while the latter is for image-based PDF documents such as scanned documents. Table recognition technology relies on computer vision and image processing algorithms to locate table areas and identify row and column boundaries in PDF pages, defining the scope for subsequent data extraction. Data extraction technology, building on table recognition, extracts text, numbers, and other content from the table into structured data formats such as Excel and JSON for subsequent processing. Basic data verification technology uses preset rules to perform preliminary checks on the extracted data, such as format validity checks and basic value range judgments, to eliminate obviously erroneous data.

[0004] Although existing technologies have implemented the basic workflow for processing PDF table data, in practical applications, due to factors such as the diverse sources of PDF document generation, irregular table structures, and complex data presentation formats, existing technical solutions still have significant shortcomings in terms of processing accuracy, automation, and intelligence, making it difficult to meet the needs of large-scale, high-precision table data processing. Against this backdrop, various optimized technical solutions for extracting PDF table data have emerged in the industry.

[0005] For example, patent document CN119691163A discloses a method for parsing tables in PDF documents applicable to RAG scenarios. The core idea of ​​this method is to first parse the PDF document and extract tables page by page, saving them as Excel files along with location information. Then, the Excel files are converted to Markdown format tables. A large language model is used to summarize the Markdown table content to generate descriptive text. Simultaneously, contextual information is extracted from the table location information to generate related text. Finally, the descriptive text, related text, and summary text are vectorized and stored in a vector database, establishing a connection with the Markdown tables. This solution combines RAG technology with table parsing, attempting to improve the ability to obtain related information from table data through text vectorization retrieval. However, it does not optimize the RAG module for the characteristics of the table structure, nor does it solve core problems such as table data concatenation and semantic interference during retrieval. Furthermore, it lacks design considerations for data verification and repair, failing to achieve a closed-loop processing from extraction to verification.

[0006] Patent document CN117710996A discloses a method for data extraction, classification, and storage of unstructured tabular documents based on deep learning. It constructs a complete process including document preprocessing, table detection, table content recognition, table data classification, and table data storage. It leverages deep learning algorithms to improve the recognition and extraction capabilities of unstructured tables, aiming to reduce the workload and error rate of manual operations through automated processing. The core advantage of this solution lies in improving the accuracy of table detection and content recognition through deep learning. However, it lacks an effective design in the post-extraction verification stage, failing to establish a dynamic optimization mechanism based on state evaluation. When errors occur in the extracted data, it cannot autonomously correct them, and it does not optimize retrieval and matching strategies for the semantic features of the table data. Therefore, it still suffers from insufficient accuracy when handling complex table structures or data concatenation scenarios.

[0007] Patent document CN117475461A discloses a method for extracting irregular tables from PDF documents, focusing on irregular table scenarios such as cross-page tables and tables with missing borders. It obtains dual feature information by performing table recognition and text recognition on the target page, thereby determining whether the aforementioned irregular tables exist. Cross-page tables are merged, and tables with missing borders are filled in, ultimately extracting structured data. This solution focuses on the structural repair and data extraction of irregular tables, improving table processing capabilities in specific scenarios. However, it does not involve the application of search enhancement techniques, cannot improve the accuracy of data extraction through contextual information, and lacks a systematic verification and repair mechanism. It cannot effectively verify the completeness and accuracy of the extracted data, nor can it dynamically adjust the processing flow based on verification results.

[0008] Patent document CN114077830A discloses a location-based method for parsing PDF table documents. It leverages the feature of the official PDF specification that supports obtaining string location information. First, it extracts the location information of each string through text extraction. Then, it establishes a correspondence between the string and the table cell based on the location information, ultimately completing string filling to achieve table parsing. This solution relies on location information to achieve accurate matching between strings and cells, which has certain advantages in parsing regular tables. However, it is poorly adaptable to tables with unclear borders or ambiguous cell boundaries. Furthermore, it does not solve the problem of extracting data where numbers and text are intertwined. It also lacks semantic-level retrieval and optimization design, failing to utilize core semantic information such as table headers and field names to improve the accuracy of data extraction, and it does not address the end-to-end design of data verification and repair.

[0009] Patent document CN117454851A discloses a method for extracting table data from PDF documents. After parsing the initial table, the text list on the page containing the initial table is segmented to obtain a two-dimensional text list. Then, the table category is determined based on the number of rows and columns of the initial table and the number of columns in the two-dimensional text list. Finally, the initial table is reconstructed based on the table category and the text list to obtain structured data. This scheme optimizes the reconstruction of the initial table through text segmentation and table category determination, improving the structured nature of the extracted table data. However, it does not deeply mine the semantic features of the table data, making it impossible to locate core data through key semantic information. Furthermore, it lacks modules for data verification and repair, making it impossible to verify the accuracy of the reconstructed table data or to autonomously repair erroneous data, thus failing to guarantee the reliability of the output data.

[0010] The aforementioned existing technical solutions all focus on targeted optimizations for PDF table data extraction, improving the efficiency of table data processing in specific scenarios. However, they generally suffer from the following common shortcomings: the retrieval mechanism is not optimized for the characteristics of the table structure, failing to effectively address issues such as data concatenation and semantic interference; an intelligent closed-loop control process based on state assessment is lacking, preventing dynamic adjustment of processing strategies according to the data extraction status; and a standardized, tool-based verification and repair function system has not been built, failing to achieve fully automated processing from extraction to verification and repair. Specifically:

[0011] 1. Insufficient precision in parsing table data: PDF document tables inherently possess diverse structures and complex data presentation characteristics. Traditional parsing techniques, mostly built upon general text parsing logic, struggle to adapt to the unique presentation of table data. This leads to frequent technical flaws during processing, such as numbers and text overlapping and blurred cell boundaries. Consequently, the extracted raw data is mixed with a large amount of invalid noise, resulting in insufficient data purity and an inability to provide reliable support for subsequent verification and analysis. Simultaneously, existing retrieval technologies do not perform targeted preprocessing and block optimization for PDF table data. Retrieval units are often globally disorganized text rather than precise table structure units like rows or cells. This design easily introduces irrelevant text information during retrieval, causing severe semantic interference. This makes it difficult for the system to accurately locate core contexts containing key table headers and field names, not only reducing the accuracy of data extraction but also significantly increasing the cost of processing invalid data, directly impacting the accuracy and efficiency of data extraction.

[0012] 2. The process control mechanism has significant limitations, lacking an intelligent closed-loop design based on state assessment: Most technical solutions adopt a linear "extraction-verification" processing mode, which can only achieve unidirectional process advancement and cannot dynamically adjust subsequent processing strategies based on the actual state of the data extraction results. When errors or anomalies occur in the extracted data, the system lacks the ability to autonomously determine the error type and severity, and cannot trigger targeted remedial processes such as retrieval retry or data repair. This results in a large amount of erroneous data requiring manual intervention for correction, significantly reducing the automation level of data processing and greatly increasing labor costs and processing cycles. In addition, existing technologies lack a scientific problem prioritization mechanism, employing indiscriminate processing strategies for anomalies of different severity. This fails to simulate the decision-making logic of human experts, prioritizing critical issues such as missing core monitoring data, leading to unreasonable resource scheduling, low efficiency in core data processing, and further exacerbating the lag in the overall processing flow.

[0013] 3. The verification and repair functions suffer from modularity defects, lacking a standardized and tool-based encapsulation and scheduling system: Core verification and repair functions in the existing solution, such as value range checking, unit conversion, formula calculation, and cross-table joins, exist primarily as scattered functional modules. The lack of unified interface standards and tool-based encapsulation design makes it difficult for the system to call these functions flexibly and efficiently. Furthermore, there are significant gaps between these scattered functional modules and preceding stages such as data extraction and retrieval, hindering seamless collaboration and making it difficult to build a fully automated processing flow from data extraction and verification to repair. In practical applications, manual triggering of each functional module and manual configuration of relevant parameters and processing logic are required. This operational mode not only further increases labor costs but also easily introduces new errors due to human error, severely restricting data processing efficiency and result consistency, and failing to guarantee the reliability of output data.

[0014] In summary, existing technical solutions for extracting and validating tabular data from PDF documents have revealed many core technical shortcomings in practical applications. These shortcomings are interconnected and collectively restrict the accuracy, automation, and intelligence of data processing, making it difficult for existing solutions to meet the needs of large-scale, high-precision tabular data processing scenarios. Summary of the Invention

[0015] To address this issue, this application provides a method and system for extracting and validating tabular data from PDF documents, thereby solving the problem that existing technologies are ill-suited for large-scale, high-precision tabular data processing scenarios.

[0016] To achieve the above objectives, this application provides the following technical solution:

[0017] Firstly, a method for extracting and validating tabular data from PDF documents includes:

[0018] Step 1: Receive the PDF document data to be processed; the PDF document data is either a copyable text PDF or a scanned image PDF;

[0019] Step 2: Perform differential preprocessing on the PDF document data to obtain standardized text data;

[0020] Step 3: Parse the table data in the standardized text data according to the custom parsing rules pre-constructed based on the row and column distribution characteristics of the table, divide the parsed table into row or cell-level index items, and perform text vectorization processing on the index items;

[0021] Step 4: Use a weighted retrieval strategy to retrieve the vectorized index items and extract the structured table data from the retrieval results to construct an initial table data matrix; the weighted retrieval strategy prioritizes the retrieval of index items containing core marker words and combines global text similarity calculation to obtain the retrieval results;

[0022] Step 5: Evaluate the initial table data matrix based on preset evaluation indicators, and dynamically generate scheduling instructions according to the evaluation scores and problem priority ranking;

[0023] Step 6: According to the scheduling instruction, call the corresponding verification function tool and repair function tool to perform verification and repair operations, and generate the repaired tabular data matrix;

[0024] Step 7: Re-trigger the evaluation process of Step 5 based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions and then output the final structured tabular data.

[0025] Optionally, in step 2, the PDF document data undergoes differentiated preprocessing, specifically including: if the PDF document data is a copyable text PDF, then the text stream, position coordinates, and font information are directly extracted through the PDF parsing engine; if the PDF document data is a scanned image PDF, then the OCR engine is first called to convert the image into editable text, and then the text stream, position coordinates, and font information are extracted through the PDF parsing engine.

[0026] Optionally, in step 3, the custom parsing rules include: character type judgment rules, cell boundary recognition rules, and data adhesion splitting rules; wherein, the character type judgment rules distinguish between numbers, text, and special symbols by using the ASCII code range and font attributes of the characters; the cell boundary recognition rules combine the position coordinates of text lines and the spacing threshold between adjacent text to identify the row and column boundaries of the table, and for tables with missing borders, the boundary information is supplemented by the alignment features of text blocks; the data adhesion splitting rules are based on semantic features and format features, and use a bidirectional longest matching algorithm to split the adhered numbers and text.

[0027] Optionally, in step 4, the initial table data matrix includes: row index, column index, cell content, data type, and confidence score.

[0028] Optionally, in step 5, the evaluation indicators include three dimensions: confidence, completeness, and anomaly labeling; wherein, the confidence is based on the confidence score and a confidence threshold is set; the completeness is determined by comparing with a preset field template, statistically analyzing the number and proportion of missing fields, with a core field missing ratio exceeding 5% indicating incompleteness; and the anomaly labeling identifies the anomaly types in the data through a rule engine.

[0029] Optionally, in step 5, the problem priority ranking is determined based on the abnormal information in the evaluation report, combined with a pre-built problem priority rule base.

[0030] Optionally, the problem priority rule base is divided into three levels according to the severity of the problem: Level 1 priority includes missing core monitoring data, calculation errors in key formulas, and inconsistencies in core data across tables; Level 2 priority includes missing non-core fields, slightly out-of-range values, and inconsistent unit formats; Level 3 priority includes non-standard text formats and redundant spaces.

[0031] Optionally, in step 6, the verification function tools include: a value range checking tool, a unit conversion tool, a formula calculation tool, and a cross-table correlation verification tool.

[0032] Optionally, in step 7, the structured tabular data includes the problem type, processing result, repair log, and final data.

[0033] Secondly, a PDF document table data extraction and verification system includes:

[0034] The data input layer is used to receive PDF document data to be processed; the PDF document data is either a copyable text PDF or a scanned image PDF.

[0035] The core processing layer includes: a perception-retrieval module, a decision-planning module, and an execution-repair module;

[0036] The perception-retrieval module is used to perform differential preprocessing on the PDF document data to obtain standardized text data;

[0037] And according to the custom parsing rules pre-constructed based on the row and column distribution characteristics of the table, the table data in the standardized text data is parsed, the parsed table is divided into row or cell-level index items, and the index items are processed into text vectorization.

[0038] The weighted retrieval strategy is used to retrieve the vectorized index items and extract the structured table data from the retrieval results to construct an initial table data matrix. The weighted retrieval strategy prioritizes the retrieval of index items containing core marker words and combines global text similarity calculation to obtain the retrieval results.

[0039] The decision-planning module is used to evaluate the initial table data matrix based on preset evaluation indicators, and dynamically generate scheduling instructions according to the evaluation scores and problem priority ranking.

[0040] The execution-repair module is used to call the corresponding verification function tool and repair function tool to perform verification and repair operations according to the scheduling instruction, and generate the repaired table data matrix;

[0041] The data output layer is used to re-trigger the evaluation process based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions and then outputs the final structured tabular data.

[0042] Compared with the prior art, this application has at least the following beneficial effects:

[0043] This application provides a method for extracting and validating tabular data from PDF documents, comprising: performing differential preprocessing on PDF document data to obtain standardized text data; parsing the tabular data in the standardized text data according to a custom parsing rule, dividing the parsed tables into index items, and performing text vectorization processing on the index items; using a weighted retrieval strategy to retrieve the vectorized index items, and extracting structured tabular data from the retrieval results to construct an initial tabular data matrix; evaluating the initial tabular data matrix based on preset evaluation indicators, and dynamically generating scheduling instructions according to the evaluation score and issue priority; calling the corresponding verification and repair function tools to perform verification and repair operations according to the scheduling instructions, and generating a repaired tabular data matrix; and re-triggering the evaluation process based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions. This application can effectively adapt to the needs of large-scale, high-precision tabular data processing scenarios, significantly improving the accuracy of data extraction, the degree of process automation, and the end-to-end collaborative processing capabilities. Attached Figure Description

[0044] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0045] Figure 1 A flowchart of a method for extracting and verifying tabular data from a PDF document, provided in Embodiment 1 of this application;

[0046] Figure 2 This is a schematic diagram of a PDF document table data extraction and verification system provided in Embodiment 2 of this application;

[0047] Figure 3 This is a schematic diagram of the data processing flow structure of a PDF document table data extraction and verification system provided in Embodiment 2 of this application;

[0048] Figure 4 This is a schematic diagram of the perception-retrieval module structure provided in Embodiment 2 of this application;

[0049] Figure 5 This is a schematic diagram of the decision-planning module structure provided in Embodiment 2 of this application;

[0050] Figure 6 This is a schematic diagram of the execution-repair module structure provided in Embodiment 2 of this application. Detailed Implementation

[0051] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0053] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.

[0054] The purpose of this application is to provide a method and system for extracting and validating tabular data from PDF documents, overcoming existing technological bottlenecks and improving the accuracy, automation, and intelligence of PDF tabular data processing. This application aims to solve problems such as the adhesion of numbers and text and semantic interference in retrieval existing technologies by optimizing the table parsing and retrieval mechanism, achieving accurate extraction of tabular data and improving the purity and extraction efficiency of the original data. This application also aims to construct a multi-agent closed-loop decision-making system based on state evaluation, realizing dynamic process control of "extraction-evaluation-repair-re-evaluation," while establishing a scientific problem priority ranking mechanism to ensure that core problems are handled first, improving the intelligence level of process processing and the rationality of resource scheduling. Furthermore, this application aims to standardize and tool-encapsulate various verification and repair functions, forming an extensible function toolkit, achieving seamless connection between various functional modules and preceding steps, constructing a fully automated processing flow, reducing manual intervention, lowering labor costs, and ensuring data processing efficiency and result consistency. Through the above technical improvements, this application can effectively solve the core defects of existing technologies, providing reliable technical support for the large-scale, high-precision PDF tabular data processing needs, and enhancing the resource utilization value of tabular data.

[0055] Example 1

[0056] Please see Figure 1 This embodiment provides a method for extracting and validating tabular data from PDF documents, including:

[0057] S1: Receive PDF document data to be processed; the PDF document data can be a copyable text PDF or a scanned image PDF;

[0058] Specifically, this step receives the PDF document data to be processed through a file upload interface. The interface supports HTTP / HTTPS protocols, and the received file format is .pdf.

[0059] S2: Perform differential preprocessing on the PDF document data to obtain standardized text data;

[0060] Specifically, this step employs differentiated preprocessing strategies for different types of PDF document data. For PDFs containing copyable text, the PDF parsing engine extracts the document's text flow, font information, and position coordinates. For scanned image PDFs, an OCR engine is first used to convert the image into editable text, and then the PDF parsing engine extracts the document's text flow, position coordinates, and font information. Noise filtering is performed concurrently during preprocessing, removing irrelevant information such as headers, footers, and watermarks. The original reading order of the document is restored by sorting the text lines by their position coordinates, providing foundational data for subsequent table parsing.

[0061] S3: Parse the tabular data in the standardized text data according to the custom parsing rules pre-built based on the row and column distribution characteristics of the table, divide the parsed table into row or cell-level index items, and perform text vectorization processing on the index items;

[0062] Specifically, this step employs a custom-rule-based improved table text parsing method to achieve precise separation of numbers and text, resolving the problem of data concatenation in tables. The specific technical implementation involves constructing custom parsing rules based on the row and column distribution characteristics of the table and storing them in a rule base. These custom parsing rules include: character type judgment rules, cell boundary recognition rules, and data concatenation splitting rules. The character type judgment rules distinguish between numbers, text, and special symbols based on the ASCII code range and font attributes. The cell boundary recognition rules combine the position coordinates of text lines and the spacing threshold between adjacent text to identify the row and column boundaries of the table. For tables with missing borders, boundary information is supplemented through the alignment features of text blocks. The data concatenation splitting rules, based on semantic and format features, use a bidirectional longest matching algorithm to split concatenated numbers and text. For example, for concatenated data such as "2023 annual sales of 1200 units," by recognizing semantic markers such as "annual" and "units," combined with the format features of the numbers, it is precisely split into four independent data units: "2023," "annual sales," "1200," and "units." Meanwhile, the parsed table is segmented into blocks, dividing the table into independent index items by row or cell to ensure that each index item is a "clean" fragment of table data and avoid the mixing of messy text; then the index items are processed into text vectorization, using the Sentence-BERT model to convert the text into a fixed-dimensional vector representation and store it in the vector database.

[0063] S4: A weighted retrieval strategy is used to retrieve the vectorized index items and extract the structured table data from the retrieval results to construct an initial table data matrix; the weighted retrieval strategy prioritizes the retrieval of index items containing core marker words and combines global text similarity calculation to obtain the retrieval results;

[0064] Specifically, this step first employs a weighted retrieval strategy to enhance retrieval based on table semantics. It prioritizes the identification of location markers and combines this with global text similarity calculations to improve retrieval accuracy. Technically, the retrieval process is divided into two stages: the first stage is marker-priority retrieval, which extracts core markers such as table headers and field names from the retrieval units, constructs a marker dictionary, and prioritizes retrieval units containing these core markers through a combination of exact and fuzzy matching, assigning higher weights to the results of this stage; the second stage is global text similarity retrieval, which calculates the cosine similarity between the query text and the vectors of all retrieval units, assigning lower weights to the results of this stage. Finally, a weighted summation is used to obtain a comprehensive score for each retrieval unit, and the top N retrieval units with the highest comprehensive scores are selected as the retrieval results. N can be dynamically configured according to actual processing needs, with a default value of 5. This weighted retrieval strategy effectively reduces semantic interference, ensuring that the system prioritizes the location of core contextual information.

[0065] The output search results are then integrated, and the structured tabular data is extracted to construct an initial tabular data matrix, which includes information such as row index, column index, cell content, data type, and confidence score. The confidence score is determined by the overall search score and parsing accuracy and is used for subsequent status evaluation. The integrated initial tabular data matrix is ​​then transmitted to the next step through a standardized interface.

[0066] S5: Evaluate the initial table data matrix based on preset evaluation indicators, and dynamically generate scheduling instructions according to the evaluation scores and problem priority ranking;

[0067] Specifically, this step first conducts a comprehensive assessment of the data status based on the initial table data matrix and a pre-defined evaluation index system, generating an evaluation report. The evaluation index system includes three core dimensions: confidence, completeness, and anomaly labeling. Confidence is assessed using a confidence score (i.e., the confidence score in the initial table data matrix), with a set confidence threshold. Completeness is assessed by comparing the data against pre-defined field templates, statistically analyzing the number and proportion of missing fields; a missing core field ratio exceeding 5% is considered incomplete. Anomaly labeling uses a rule engine to identify anomalies in the data, including incorrect data format, out-of-range values, inconsistent units, and inconsistent cross-table joins.

[0068] This step quantifies the evaluation results of the three dimensions into an evaluation score, with a maximum score of 100. A score of ≥80 is "qualified", 60-79 is "needs repair", and <60 is "needs to be re-extracted". The evaluation report also includes information such as the anomaly type, anomaly location, and preliminary judgment of severity.

[0069] Then, based on the anomaly information in the assessment report, a combination of rule-driven and semantic reasoning methods is used to prioritize issues. Technically, an issue priority rule base is pre-stored, divided into three levels according to severity: Level 1 priority includes missing core monitoring data, errors in key formula calculations, and inconsistencies in core data across tables; these issues directly affect data availability. Level 2 priority includes missing non-core fields, slightly out-of-range values, and inconsistent unit formats. Level 3 priority includes non-standard text formatting and redundant spaces. Prioritization first uses rule matching to initially determine the issue level, then calls a large language model service for semantic reasoning to correct deviations in rule matching. For example, for boundary issues like "partially missing core fields," semantic analysis determines the impact of the missing fields on the overall data. Finally, a sorted list of issues is output, clearly defining the processing order and priority level of each issue.

[0070] Finally, based on the status assessment results and priority ranking results, a multi-round closed-loop control logic of "extraction-assessment-repair-reassessment" is constructed to dynamically generate scheduling instructions. When the assessment result is "qualified", an "output data" instruction is directly generated to output the initial table data matrix directly; when the assessment result is "needs repair", a tool call instruction is generated according to the priority-ranked problem list, specifying the function tools to be called for "execution-repair", parameter configuration, and processing order; when the assessment result is "needs re-extraction", a "re-retrieve" instruction is generated, along with information such as keywords and tag optimization suggestions for re-retrieval, and transmitted to step S3 to trigger the re-extraction process.

[0071] S6: According to the scheduling instructions, call the corresponding verification function tool and repair function tool to perform verification and repair operations, and generate the repaired tabular data matrix;

[0072] Specifically, this step first loads the corresponding verification function tool from the tool library according to the tool identifier in the scheduling instruction, passes in the relevant parameters, and performs the verification operation. The execution logic of different function tools differs, and the specific implementation is as follows: The value range check tool reads the value range rules in the rule management unit, compares the data to be verified with the preset value range, outputs the verification result of "compliant" or "non-compliant", and records the specific value that exceeds the range; The unit conversion tool supports automatic conversion between multiple units. By building a unit conversion dictionary, it stores the conversion coefficients of different units and converts the data to the target unit according to the instruction requirements. For example, converting "kilogram" to "gram" has a conversion coefficient of 1000; The formula calculation tool supports the calculation of preset formulas and custom formulas. Users can configure formula expressions through the rule management unit. After reading the formula expression, the tool extracts the relevant data in the table for calculation, compares the calculation result with the original data in the table, and determines whether there is a calculation error; The cross-table association verification tool extracts the related fields from multiple tables, establishes the association mapping relationship, verifies whether the values ​​corresponding to the related fields are consistent, and outputs the association consistency result.

[0073] Then, for the anomalies detected during verification, the corresponding repair function tool is invoked to perform the repair operation based on the priority ranking results and repair rules. Repair rules correspond one-to-one with problem types and are stored in the rule management unit. For example, for problems with value ranges exceeding the range, if the exceedance is small and conforms to the data trend, interpolation is used for repair; for problems with inconsistent units, a unit conversion tool is invoked to convert all data to the target unit; for problems with non-standard text formatting, a text format standardization tool is invoked to remove redundant spaces, unify capitalization, etc. The repair execution unit synchronously records a repair log during the repair process, including information such as data before repair, data after repair, repair tools, and repair time, ensuring the traceability of the repair process.

[0074] Finally, the verification and repair results are integrated to generate a detailed verification and repair report, which includes information such as the problem type, processing results, repair logs, and final data.

[0075] S7: Re-trigger the evaluation process of step S5 based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions and then output the final structured tabular data.

[0076] Specifically, this step triggers a re-evaluation based on the verification and repair report. At the same time, it updates the initial table data according to the repaired table data matrix and outputs the final structured table data. The structured table data includes the problem type, processing results, repair logs and final data, and supports exporting in multiple structured formats such as Excel and JSON.

[0077] This embodiment outputs the final result to the user through a file download interface. The data output interface supports batch export and single file export, and the user can select the output format and output content through the system interface.

[0078] This embodiment provides a method for extracting and validating table data from PDF documents. Addressing pain points such as low accuracy, insufficient automation, and severe semantic interference in PDF document table data extraction and validation, it constructs three key technical points based on a logical chain of "perception retrieval - decision scheduling - execution repair - closed-loop output." This not only solves the problems of insufficient parsing accuracy, excessive manual intervention, and poor retrieval targeting in existing technologies, but also provides a practical technical solution for intelligent PDF table data processing. The three key technical points are:

[0079] The RAG-enhanced perception-retrieval mechanism based on table structure optimization achieves accurate parsing and efficient retrieval of table data in PDF documents, adaptable to different types of PDF tables such as copyable text and scanned images. Through a table text parsing method improved by custom rules, it achieves accurate separation of numbers and text. Combined with specialized preprocessing and block-based strategies, it divides tables into clean retrieval units at the row / cell level. A weighted retrieval strategy combining priority markers and global text similarity calculation improves the accuracy of core context positioning and reduces semantic interference.

[0080] A multi-agent driven closed-loop decision-planning and scheduling mechanism: This mechanism constructs a decision-making and scheduling system centered on multiple agents. It can automatically identify the data extraction status of tables and break down processing tasks into sub-steps such as "extraction-evaluation-repair-re-evaluation." A three-dimensional quantitative evaluation system assesses data confidence, integrity, and anomaly marking, dynamically triggering data output, re-extraction, or repair scheduling. Combining rule matching and semantic reasoning, it prioritizes issues, simulating human expert decision-making logic and improving the rationality of resource scheduling.

[0081] The plug-in toolset supports a fully automated verification and repair mechanism: core capabilities such as value range checking, unit conversion, formula calculation, and cross-table correlation verification are encapsulated into standardized function tools to build an extensible plug-in tool library; under multi-agent scheduling, the tools are automatically invoked, executed, and the results are fed back, forming a seamless collaborative link from data extraction to verification and repair; differentiated repair strategies are used to handle issues of different priorities, and repair logs are recorded synchronously to ensure that the repair process is traceable.

[0082] The PDF document table data extraction and verification method provided in this embodiment has the following advantages:

[0083] 1. Enhanced structured parsing and retrieval of PDF tables through specialized methods, and the use of a custom rule base to split table data and create clean retrieval units; combined with a priority weighting strategy for marker words and global text similarity calculation to achieve accurate positioning of core context; forming a complete technical process from PDF table preprocessing, parsing, retrieval to initial data integration, improving the accuracy and efficiency of table data extraction.

[0084] 2. It includes three core functions: status assessment, decision scheduling, and priority ranking. It has the ability to perceive the status of tabular data processing and dynamically plan tasks. It can dynamically adjust the processing strategy based on the data assessment results and prioritize the scheduling of resources to handle core issues. The key technology lies in the control of the closed-loop logic of "extraction-assessment-repair-reassessment" by multiple agents to ensure that the processing process is efficient and intelligent.

[0085] 3. First, encapsulate various verification and repair capabilities into standardized function tools and build an extensible tool library; then, through multiple agents, enable the tools to be automatically invoked on demand, and match differentiated repair rules for different types of anomalies; achieve full automation and traceability of the verification and repair process, and solve the defects of fragmented and poorly coordinated verification and repair functions in existing technologies.

[0086] In summary, this embodiment achieves a technological breakthrough through the entire chain from table data parsing, retrieval enhancement, decision scheduling to verification and repair. It effectively solves the core problems of existing technologies, such as insufficient parsing accuracy, poor retrieval targeting, low degree of process automation, and modular defects in verification and repair. It significantly improves the accuracy, efficiency, and intelligence level of PDF table data processing, providing reliable technical support for the needs of large-scale, high-precision PDF table data processing, and can effectively adapt to the needs of large-scale, high-precision table data processing scenarios.

[0087] Example 2

[0088] Please see Figure 2 and Figure 3This embodiment provides a PDF document table data extraction and verification system. The overall architecture of the system adopts a layered modular design, which is divided into a data input layer, a core processing layer, a data output layer and a tool support layer. Data interaction and instruction transmission between each layer are realized through standardized interfaces. The overall architecture principle block diagram clearly shows the composition and relationship of each layer.

[0089] The data input layer is used to receive PDF document data to be processed; the PDF document data includes two types: copyable text PDFs and scanned image PDFs.

[0090] The core processing layer is the core part of the architecture. It includes three core functional modules: perception-retrieval module, decision-planning module, and execution-repair module, which are used to realize data extraction, decision scheduling, verification and repair.

[0091] The perception-retrieval module is used to perform differential preprocessing on PDF document data to obtain standardized text data.

[0092] And based on custom parsing rules pre-built based on the row and column distribution characteristics of the table, the table data in the standardized text data is parsed, the parsed table is divided into row or cell-level index items, and the index items are processed into text vectorization.

[0093] The algorithm employs a weighted retrieval strategy to search the vectorized index terms and extracts the structured table data from the search results to construct an initial table data matrix. The weighted retrieval strategy prioritizes the search of index terms containing core marker words and combines global text similarity calculations to obtain the search results.

[0094] The decision-making and planning module is used to evaluate the initial tabular data matrix based on preset evaluation indicators and dynamically generate scheduling instructions according to the evaluation scores and problem priority.

[0095] The execution-repair module is used to call the corresponding verification function tool and repair function tool to perform verification and repair operations according to the scheduling instructions, and generate the repaired tabular data matrix.

[0096] The data output layer is used to re-trigger the evaluation process based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions, and then outputs the final structured tabular data, verification report and repair log. It supports exporting multiple structured formats such as Excel and JSON.

[0097] The tool support layer provides fundamental tools for the core processing layer, including a vector database, a PDF parsing engine, a large language model service, and a rule engine. The vector database uses Milvus to store text vector data for the "perception-retrieval" module, supporting efficient vector similarity retrieval. The PDF parsing engine integrates Poppler and Tesseract OCR engines to parse and extract text from different types of PDFs. The large language model service uses the open-source Qwen model to support semantic reasoning in the "decision-planning" module and text summarization in the "perception-retrieval" module. The rule engine uses the Drools rule engine to achieve efficient execution and matching of evaluation rules, priority rules, and repair rules. All service components are deployed in a cluster to ensure high system availability and processing performance.

[0098] In the overall architecture diagram, each layer is connected via bidirectional data links. PDF document data from the data input layer is converted and then input to the "Perception-Retrieval" module of the core processing layer. Within the core processing layer, there is bidirectional data interaction between the "Perception-Retrieval" module, the "Decision-Planning" module, and the "Execution-Repair" module. The "Decision-Planning" module sends tool invocation commands to the "Execution-Repair" module via an instruction link. The tool support layer provides real-time service support to the three modules of the core processing layer through service interfaces. The processing results of the core processing layer are transmitted to the data output layer to complete the data output. The core design advantage of this architecture lies in achieving decoupling and collaboration among modules, improving system scalability through standardized interface design, and ensuring intelligent and automated processing flows by leveraging the decision-making and scheduling capabilities of multi-agent systems.

[0099] Specifically, the three modules of the core processing layer are the core innovations of the technical solution. Each module has its own independent technical implementation logic and achieves functional complementarity through a collaborative mechanism. The following section, combined with the internal principle block diagram of each module, elaborates on its technical implementation scheme.

[0100] The "Perception-Retrieval" module is a retrieval enhancement module based on table structure optimization. Unlike traditional RAG modules, its core technological innovation lies in its specialized optimization design for PDF table structures, enabling accurate parsing and efficient retrieval of table data. The module's internal principle diagram includes four core units: a PDF preprocessing unit, a table perception and parsing unit, a retrieval unit, and a data integration unit. Figure 4 As shown.

[0101] More specifically, after receiving PDF document data, the "Perception-Retrieval" module first performs differentiated preprocessing by the PDF preprocessing unit. Then, the table perception and parsing unit calls a custom rule base to accurately parse the table data through character type judgment, cell boundary recognition, and data adhesion splitting, dividing the parsed table into row / cell-level index items. Next, the retrieval unit performs text vectorization processing on the index items, generating vector data and storing it in the vector database. At the same time, a weighted retrieval strategy is executed, prioritizing the retrieval of retrieval units containing core marker words, and combining global text similarity calculation to obtain the retrieval results. Finally, the data integration unit integrates the retrieval results, constructs an initial table data matrix, and transmits it to the "Decision-Planning" module through the inter-module interface.

[0102] The "Decision-Planning" module serves as the system's "brain," corely implementing multi-round closed-loop control and problem prioritization. Its internal principle diagram includes four core units: a state evaluation unit, a decision scheduling unit, a priority ranking unit, and a rule management unit. Figure 5 As shown, through multi-agent logical reasoning and instruction scheduling, the collaborative control of the "perception-retrieval" module and the "execution-repair" module is achieved.

[0103] More specifically, after receiving the initial table data matrix, the "Decision-Planning" module performs an evaluation based on three dimensions: confidence, completeness, and anomaly marking, generating an evaluation report and score. The priority ranking unit prioritizes the issues based on the anomaly information in the evaluation report and the priority rule base, outputting a ranked list of issues. The decision scheduling unit determines the processing strategy based on the evaluation score. If the score is ≥80, it generates an "output data" instruction; if the score is 60-79, it generates a tool call instruction, specifying the function tools to be called and the processing order; if the score is <60, it generates a "re-retrieve" instruction and transmits it to the "Perception-Retrieval" module.

[0104] In this embodiment, the decision scheduling unit receives the processing results of the "execution-repair" module or the "perception-retrieval" module in real time through the status feedback mechanism, and re-triggers the status evaluation unit to perform evaluation until the evaluation result is "qualified", thus completing the closed-loop control.

[0105] The rule management unit is responsible for storing, updating, and retrieving priority rules, evaluation rules, and scheduling rules. It employs a scalable rule configuration mechanism, allowing users to add and modify rule parameters through a visual interface. Rules are stored in JSON format to ensure readability and maintainability. The rule management unit interacts in real-time with the status evaluation unit, priority ranking unit, and decision scheduling unit through interfaces, providing rule support to each unit.

[0106] The "Execution-Repair" module encapsulates various verification and repair capabilities into tools, serving as an extensible toolset scheduled by the "Decision-Planning" module. Its internal principle diagram includes four core units: a tool management unit, a verification execution unit, a repair execution unit, and a result feedback unit. This enables automated execution and result feedback of verification and repair functions. Figure 6 As shown. The specific technical implementation scheme is as follows:

[0107] The Tool Management Unit employs a plug-in architecture, encapsulating various validation and calculation capabilities into standard function tools. Each function tool has a unified API interface, including input parameters, output parameters, execution logic, and error handling mechanisms. Encapsulated function tools include value range checking tools, unit conversion tools, formula calculation tools, cross-table join validation tools, and text format standardization tools. The Tool Management Unit is responsible for registering, loading, updating, and uninstalling function tools, and supports users in extending and customizing function tools as needed. The function tool registration process is as follows: users submit the tool's interface definition, execution logic code, and test cases. The Tool Management Unit performs compatibility and security verification on the tool. Upon successful verification, registration is completed, a unique tool identifier is assigned, and the tool is stored in the tool library.

[0108] In this example, if the decision scheduling unit generates a tool invocation command, the tool management unit of the "Execution-Repair" module loads the corresponding function tool according to the command, the verification execution unit calls the tool to perform verification operations, and identifies abnormal issues; the repair execution unit calls the tool to perform repair operations according to the priority sorting results and repair rules, and records the repair log; the result feedback unit integrates the verification and repair results, generates a verification and repair report and a repaired tabular data matrix, and transmits it to the "Decision-Planning" module to trigger a re-evaluation. If the decision scheduling unit generates a "Re-Search" command, the "Perception-Search" module adjusts the search keywords and tags according to the optimization suggestions in the command, re-executes the parsing and retrieval process, generates a new initial tabular data matrix, and transmits it to the "Decision-Planning" module.

[0109] In this implementation, the modules interact with each other in real time through standardized interfaces to exchange data and transmit commands. The tool support layer provides real-time service support for each stage, ensuring the continuity and efficiency of data processing. During the data flow, all intermediate data and final results are encrypted and stored to ensure data security and privacy. At the same time, a complete processing log is recorded to support the traceability and auditing of the data processing process.

[0110] This embodiment provides a PDF document table data extraction and verification system that adopts a layered and modular system architecture and a full-link data flow mechanism: it employs a layered architecture consisting of a data input layer, a core processing layer, a tool support layer, and a data output layer. Each layer achieves low-coupling and high-cohesion collaborative interaction through standardized interfaces. The tool support layer integrates basic services such as vector databases, PDF parsing engines, and large language model services to provide stable support for core processing. The data output layer supports exporting to various structured formats such as Excel and JSON, adapting to different scenarios such as data archiving and subsequent analysis, thereby improving system adaptability and deployment convenience.

[0111] This embodiment integrates four levels: data input, core processing, tool support, and data output. Each level achieves data interaction and command transmission through standardized interfaces. The core processing layer integrates three modules: perception-retrieval, decision-making-planning, and execution-repair. The tool support layer provides comprehensive basic service support. At the system level, it realizes full-link automated processing from PDF document input to structured data output, ensuring the system's maintainability, scalability, and high availability.

[0112] For details on the implementation of each module in a PDF document table data extraction and verification system, please refer to the above description of the limitations of a PDF document table data extraction and verification method, which will not be repeated here.

[0113] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A method for PDF document table data extraction and verification, characterized in that, include: Step 1: Receive the PDF document data to be processed; the PDF document data is either a copyable text PDF or a scanned image PDF; Step 2: Perform differential preprocessing on the PDF document data to obtain standardized text data; Step 3: Parse the table data in the standardized text data according to the custom parsing rules pre-constructed based on the row and column distribution characteristics of the table, divide the parsed table into row or cell-level index items, and perform text vectorization processing on the index items; Step 4: Use a weighted retrieval strategy to retrieve the vectorized index items and extract the structured table data from the retrieval results to construct an initial table data matrix; the weighted retrieval strategy prioritizes the retrieval of index items containing core marker words and combines global text similarity calculation to obtain the retrieval results; Step 5: Evaluate the initial table data matrix based on preset evaluation indicators, and dynamically generate scheduling instructions according to the evaluation scores and problem priority ranking; Step 6: According to the scheduling instruction, call the corresponding verification function tool and repair function tool to perform verification and repair operations, and generate the repaired tabular data matrix; Step 7: Re-trigger the evaluation process of Step 5 based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions and then output the final structured tabular data.

2. The PDF document table data extraction and verification method of claim 1, wherein, In step 2, the PDF document data undergoes differentiated preprocessing, specifically including: if the PDF document data is a copyable text PDF, the text stream, position coordinates, and font information are directly extracted through the PDF parsing engine; if the PDF document data is a scanned image PDF, the OCR engine is first called to convert the image into editable text, and then the text stream, position coordinates, and font information are extracted through the PDF parsing engine.

3. The PDF document table data extraction and verification method of claim 1, wherein, In step 3, the custom parsing rules include: character type judgment rules, cell boundary recognition rules, and data sticking and splitting rules; wherein, the character type judgment rules distinguish between numbers, text, and special symbols by using the ASCII code range and font attributes of the characters; the cell boundary recognition rules combine the position coordinates of text lines and the spacing threshold between adjacent text to identify the row and column boundaries of the table, and for tables with missing borders, the boundary information is supplemented by the alignment features of text blocks; the data sticking and splitting rules are based on semantic features and format features, and use a bidirectional longest matching algorithm to split sticking numbers and text.

4. The method of claim 1, wherein, In step 4, the initial table data matrix includes: row index, column index, cell content, data type, and confidence score.

5. The PDF document table data extraction and verification method of claim 4, wherein, In step 5, the evaluation indicators include three dimensions: confidence, completeness, and anomaly labeling. The confidence is achieved by using a confidence score and setting a confidence threshold. The completeness is determined by comparing a preset field template and counting the number and proportion of missing fields. A missing core field ratio exceeding 5% is considered incomplete. The anomaly labeling identifies the types of anomalies in the data using a rule engine.

6. The method of claim 1, wherein, In step 5, the problem priority ranking is determined based on the abnormal information in the evaluation report, combined with a pre-built problem priority rule base.

7. The PDF document table data extraction and verification method of claim 6, wherein, The problem priority rule base is divided into three levels according to the severity of the problem: Level 1 priority includes missing core monitoring data, calculation errors in key formulas, and inconsistencies in core data across tables; Secondary priority includes missing non-core fields, slightly out-of-range values, and inconsistent unit formats; Level 3 priority includes non-standard text formatting and redundant spaces.

8. The method of claim 1, wherein, In step 6, the verification function tools include: a value range checking tool, a unit conversion tool, a formula calculation tool, and a cross-table association verification tool.

9. The method of claim 1, wherein, In step 7, the structured table data includes the problem type, processing result, repair log, and final data.

10. A PDF document table data extraction and verification system, characterized in that, include: The data input layer is used to receive PDF document data to be processed. The PDF document data is either a copyable text PDF or a scanned image PDF; The core processing layer includes: a perception-retrieval module, a decision-planning module, and an execution-repair module; The perception-retrieval module is used to perform differential preprocessing on the PDF document data to obtain standardized text data; And according to the custom parsing rules pre-constructed based on the row and column distribution characteristics of the table, the table data in the standardized text data is parsed, the parsed table is divided into row or cell-level index items, and the index items are processed into text vectorization. The weighted retrieval strategy is used to retrieve the vectorized index items and extract the structured table data from the retrieval results to construct an initial table data matrix. The weighted retrieval strategy prioritizes the retrieval of index items containing core marker words and combines global text similarity calculation to obtain the retrieval results. The decision-planning module is used to evaluate the initial table data matrix based on preset evaluation indicators, and dynamically generate scheduling instructions according to the evaluation scores and problem priority ranking. The execution-repair module is used to call the corresponding verification function tool and repair function tool to perform verification and repair operations according to the scheduling instruction, and generate the repaired table data matrix; The data output layer is used to re-trigger the evaluation process based on the repaired tabular data matrix until the evaluation score meets the preset qualification conditions and then outputs the final structured tabular data.

Images