Bidding response file-oriented multi-type document structured analysis method and system

Abstract

The invention provides a multi-type document structured analysis method for a bidding response file. The method comprises the following steps: S1, analyzing a PDF (Portable Document Format) document; s2, document type identification; s3, field extraction and JSON generation are carried out; s4, mapping to a structured two-dimensional table; s5, the full text content is written into a PDF-Content table; and S6, DuckDB loading and SQL (Structured Query Language) query are performed. The method aims at overcoming the defects that an existing bidding response file analysis technology is poor in adaptability, complex in deployment, lack of traceability and global analysis ability and the like, automatic structured conversion of multiple types of PDF proof materials is achieved, an off-line efficient query and analysis system is constructed, traceability verification of field information is ensured, cross-document intelligent question answering and bidding assistance are supported, and the method has the advantages of being high in practicability and high in efficiency. And finally, the automation degree, the accuracy and the compliance of bidding document compilation and review are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross-technical field of document intelligent processing, structured data generation and database query technology, in particular to a multi-type document structured analysis method and system for bidding response files. BACKGROUND

[0002] The proof materials (contracts, invoices, certificates, etc.) in the bidding response files are mostly in the form of PDF or scanned copies, with the characteristics of large size, large quantity, significant format difference, and high difficulty of digital processing. The existing technology mainly relies on OCR text extraction, fixed template field extraction, relational database storage and field-level retrieval, and has five core defects: It cannot adapt to heterogeneous proof materials of multiple sources and formats, and the template reusability is poor; Only structured fields are retained, and the full-text content is not stored, resulting in the breaking of the fields and the original text, and the lack of traceability and cross-page semantic understanding ability; It is difficult to realize global correlation analysis across documents and years (such as matching the contract and invoice amounts); The traditional relational database deployment is complex and resource-intensive, and is not suitable for temporary batch processing and offline deployment requirements in the bidding scenario; It cannot support knowledge graph construction and intelligent bidding assistance, and the bidding preparation and review efficiency is low. SUMMARY

[0003] The present application provides a multi-type document structured analysis method and system for bidding response files, aiming to overcome the defects of poor adaptability, complex deployment, lack of traceability and global analysis ability of the existing bidding response file analysis technology, realize the automatic structured conversion of multi-type PDF proof materials, build an offline efficient query and analysis system, ensure the traceability of field information, support cross-document intelligent question and answer and bidding assistance, and ultimately improve the automation degree, accuracy and compliance of bidding file preparation and review.

[0004] To achieve the above purpose, the present application adopts the following technical solutions: The multi-type document structured analysis method for bidding response files comprises the following steps: S1: obtaining a set of multi-type PDF proof materials in the bidding response file, performing OCR text recognition and layout analysis on each PDF proof material, extracting page-level text, pictures, tables and position-related information, and obtaining unstructured content; S2: automatically identifying the document type of each PDF proof material based on the text features and format features of the unstructured content; S3: According to the document type, a corresponding extraction strategy is called to extract key business fields and generate semi-structured JSON data, which contains field name, field value, belonging page number and confidence related information, and the fields appearing across pages or repeatedly are merged and optimized; S4: The semi-structured JSON data is mapped to a structured data table, and all structured data tables are associated by unique identification; S5: The full text content of each PDF proof material is converted into a standardized text format, and the original text, page number and position index information in the unstructured content are combined and stored in a full text content table; S6: After converting the structured data table and the full text content table into a specified data format, load them into an embedded columnar database to form a queryable data set, perform statistical analysis, cross-document correlation comparison based on the data set, and realize intelligent question and answer, data verification and generative summary output combined with a large language model.

[0005] In this specification, the document types in S2 include contract, invoice, patent, certificate and narrative document, and the recognition method is realized by keyword matching, format feature analysis or large language model reasoning.

[0006] In this specification, in S3, when calling a large language model to extract fields, input includes preset prompt information containing document type description, list of fields to be extracted, JSON output format specification and field value verification rules to ensure the standardization of JSON data.

[0007] In this specification, the mapping method in S4 includes at least one of direct mapping, dynamic expansion mapping and one-to-many association mapping, and non-standard fields are stored in an extended data table.

[0008] In this specification, the standardized text format in S5 is Markdown format, which is realized by converting the title, paragraph, table and picture area in the PDF proof material into the corresponding Markdown format.

[0009] In this specification, the embedded columnar database is DuckDB, which adopts a service-free deployment mode and supports direct operation in local devices or offline environments without the need for a separate database server.

[0010] In this specification, the JSON data generated in S3 also contains position box information corresponding to the fields, and in S6, the position box information and the position index in the full text content table are used to accurately locate the specific position of the field in the original PDF proof material.

[0011] In the specification, the cross-document association comparison in S6 includes contract amount and invoice amount matching verification, certificate validity period verification, and the generated summary includes contract summary, patent list and performance certificate related materials.

[0012] In the specification, the position-related information in S1 includes in-page position coordinates, and the full-text content table in S5 further stores document type, picture placeholder and table placeholder information.

[0013] The multi-type document structured analysis system for a bid response document comprises: A document analysis module is configured to obtain a multi-type PDF proof material set in the bid response document, perform OCR text recognition and layout analysis, and extract unstructured content. A document classification module is configured to automatically identify the document type of the PDF proof material based on the text features and format features of the unstructured content. A field extraction module is configured to call a corresponding extraction strategy according to the document type, extract key business fields and generate semi-structured JSON data, and merge and optimize cross-page or repeated fields. A structured data conversion module is configured to map the semi-structured JSON data to a structured data table to realize standardized storage of data. A PDF-Content full-text storage module is configured to convert the full-text content of the PDF proof material into a standardized text format, and store the original text, page number and position index information in the full-text content table. An embedded columnar database module is configured to load the structured data table and the full-text content table, form a queryable data set, and provide data query and analysis support. An intelligent question answering and verification module is configured to perform statistical analysis and cross-document association comparison based on the data set, and realize intelligent question answering, data verification and generated summary output based on a large language model.

[0014] In summary, the present application has at least the following advantages: Improved adaptability: compatible with multi-type and multi-format proof materials in the bidding scenario, breaking through the adaptation limitations of fixed templates; Complete data link: multi-layer modeling of "unstructured PDF→ semi-structured JSON→ structured table→ full-text Markdown" is realized, solving the problem of structured data and original text breakage; Efficient deployment and query: no need for independent database services, supporting offline environment and batch processing, greatly reducing deployment complexity and resource occupation; Enhanced analysis capability: supporting cross-document association verification, global question answering and semantic analysis, improving the mining and utilization efficiency of bidding-related data; Source tracing and audit compliance: Achieve precise association between fields and original page numbers and text fragments to meet the authenticity and audit requirements of bidding scenarios; Bidding assistance: Automatically generates performance certificates, patent lists and other materials required for bidding, simplifying the preparation process and improving the accuracy of the review. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a multi-type document structured parsing system for bid response documents involved in this invention.

[0017] Figure 2 This is a schematic diagram of the multi-type document structure parsing method for bid response documents involved in this invention.

[0018] Figure 3 This is a schematic diagram of the dual-layer storage structure of the structured field table and the PDF-Content table involved in this invention.

[0019] Figure 4 This is a schematic diagram of the timing diagram for question-answering and verification based on DuckDB and PDF-Content involved in this invention.

[0020] Figure 5 This is a schematic diagram of the data processing pipeline components involved in this invention.

[0021] Figure 6 This is a schematic diagram of the deployment architecture involved in this invention. Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] like Figure 1 As shown, this embodiment provides a method for structured parsing of multiple document types for bid response documents, including the following steps: S1: Obtain a collection of various types of PDF supporting documents from the bid response file, perform OCR text recognition and layout analysis on each PDF supporting document, extract page-level text, images, tables and location-related information to obtain unstructured content; S2: Based on the text and format features of the unstructured content, automatically identify the document type of each PDF supporting material; S3: Based on the document type, call the corresponding extraction strategy to extract key business fields and generate semi-structured JSON data. The JSON data includes field names, field values, page numbers, and confidence information. Fields that cross pages or appear repeatedly are merged and optimized. S4: Map the semi-structured JSON data to a structured data table, and associate all structured data tables with a unique identifier; S5: Convert the full text of each PDF supporting document into a standardized text format, and store it in the full text content table by combining the original text, page number and position index information in the unstructured content; S6: After converting the structured data table and the full-text content table into a specified data format, load them into an embedded columnar database to form a queryable dataset. Perform statistical analysis and cross-document association comparison based on the dataset, and combine it with a large language model to realize intelligent question answering, data verification and generative summary output.

[0026] In some embodiments, the document types mentioned in S2 include contract types, invoice types, patent types, certificate types, and narrative document types, and the identification method is achieved through keyword matching, layout feature analysis, or large language model inference.

[0027] In some embodiments, when the large language model is called in S3 to extract fields, a preset prompt message is input containing a document type description, a list of fields to be extracted, JSON output format conventions, and field value validation rules to ensure the standardization of JSON data.

[0028] In some embodiments, the mapping method described in S4 includes at least one of direct mapping, dynamic extended mapping, and one-to-many association mapping, and non-standard fields are stored in an extended data table.

[0029] In some embodiments, the standardized text format mentioned in S5 is Markdown format, which is achieved by converting the titles, paragraphs, tables, and image areas in the PDF supporting materials into the corresponding Markdown format.

[0030] In some embodiments, the embedded columnar database is DuckDB, which adopts a serverless deployment mode and supports direct operation on local devices or offline environments without the need for a separate database server.

[0031] In some embodiments, the JSON data generated in S3 also includes location box information corresponding to the field. When tracing the field in S6, the specific location of the field in the original PDF supporting material is accurately located by using the location box information and the location index in the full-text content table.

[0032] In some embodiments, the cross-document association comparison in S6 includes contract amount and invoice amount matching verification and certificate validity period verification, and the generative summary includes contract summary, patent list and performance proof related materials.

[0033] In some embodiments, the location-related information in S1 includes page location coordinates, and the full-text content table in S5 also stores document type, image placeholders, and table placeholders.

[0034] A multi-type document structure parsing system for bid response documents, including: The document parsing module is used to obtain a collection of various types of PDF supporting materials from the bid response file, perform OCR text recognition and layout analysis, and extract unstructured content; The document classification module is used to automatically identify the document type of PDF supporting materials based on the text and format features of unstructured content; The field extraction module is used to call the corresponding extraction strategy according to the document type, extract key business fields and generate semi-structured JSON data, and merge and optimize cross-page or duplicate fields; The structured data conversion module is used to map semi-structured JSON data into structured data tables, thereby achieving standardized data storage. The PDF-Content full-text storage module is used to convert the full-text content of PDF supporting materials into a standardized text format and store it in a full-text content table, combining the original text, page numbers, and location index information. Embedded columnar database module: Employs an embedded columnar database to load structured data tables and full-text content tables, forming a queryable dataset and providing data query and analysis support; The intelligent question answering and verification module is used to perform statistical analysis and cross-document correlation comparison based on the dataset, and to achieve intelligent question answering, data verification and generative summary output by combining a large language model.

[0035] The technical concept of this invention is as follows: To overcome the problems of difficulty in parsing bid response documents, low level of structure, difficulty in querying cross-document information, lack of full-text traceability capabilities, and complex deployment of traditional databases in existing technologies, the present invention aims to provide a method and system for structured parsing of multiple document types for bid response documents. This method converts unstructured document content into semi-structured JSON data by performing text recognition, automatic document type identification, and field extraction on PDF documents, and further maps it into a structured two-dimensional data table. Simultaneously, it generates a corresponding full-text content table, realizing a two-layer storage structure of structured fields and full-text content.

[0036] Another objective of this invention is to provide a data processing mechanism that can efficiently execute analytical queries without the need for a separate database service. By employing an embedded columnar database to load and query structured tables and full-text content, it supports field-level retrieval, cross-page semantic analysis, cross-document association queries, and global question answering for various types of supporting documents such as contracts, invoices, patent certificates, and qualification certificates, thereby improving the automation and accuracy of bid document processing.

[0037] Another objective of this invention is to achieve reliable traceability and verification of structured fields, ensuring that the field information extracted from the PDF document corresponds to its original location, thereby meeting the stringent requirements for authenticity, traceability and auditability in bidding scenarios.

[0038] In summary, the present invention aims to provide a unified document structure parsing technology solution that can adapt to various types of PDF supporting materials, has structured extraction capabilities, supports full-text storage and global query, and can be seamlessly integrated into the bidding preparation and review process.

[0039] This invention provides an integrated technical solution combining a system and a method, with four key design elements: System architecture: It is divided into an input layer, a parsing and extraction layer (document parsing, classification, field extraction, structured transformation, and full-text storage modules), a data storage and query layer (structured field table, PDF-Content full-text table, and DuckDB embedded database), and an application and service layer (SQL analysis, intelligent question answering, and validation modules). Core process: Parse PDFs using OCR and layout analysis, automatically identify document types, extract key fields using LLM or rules to generate JSON, map it to a structured two-dimensional table, synchronously store the full-text Markdown to the PDF-Content table, and finally execute SQL queries and LLM-based question answering using DuckDB; Key technologies: The combination of "multi-layer modeling + dual-layer storage + embedded columnar database" is used to solve the core defects of existing technologies; Deployment method: Supports local client and offline environment deployment, adapting to the temporary batch processing needs of bidding scenarios.

[0040] like Figure 1 As shown, the structured parsing system for bid response documents of the present invention mainly includes the following modules: 1. The document parsing module is used to perform OCR, page layout extraction, and image content extraction on PDF documents to obtain page-level text fragments, image fragments, and position coordinate information.

[0041] 2. Document classification module, used to automatically identify document types based on parsing results, including contracts, invoices, certificates, patents, test reports, and narrative documents.

[0042] 3. Field extraction module, used to call the corresponding extraction model or large language model (LLM) to generate semi-structured JSON data based on the identified document type, including field name, field value, page number position, confidence score, etc.

[0043] 4. The structured data conversion module is used to map the JSON generated by the field extraction module to predefined structured two-dimensional tables, such as contract tables, invoice tables, certificate tables, and patent tables, for field-level querying and statistical analysis.

[0044] 5. PDF-Content full-text storage module, used to store the full-text Markdown content, OCR line text and page structure of documents, for cross-page analysis, context understanding and source tracing.

[0045] 6. The embedded columnar database module (DuckDB) is used to load structured tables and full-text content tables into an in-memory dataset and execute SQL queries without deploying a separate server.

[0046] 7. Intelligent question answering and validation module, used to realize cross-document question answering, field validation, contract and invoice amount matching validation and generative summary based on structured fields and full text content.

[0047] like Figure 2 As shown, the document structure parsing method of the present invention mainly includes the following steps: Step S1: PDF document parsing Using an OCR engine and layout analysis model, PDF documents are processed as follows: 1. Text recognition; 2. Image detection; 3. Table area recognition; 4. Extract headers and footers; 5. Extraction of text hierarchical structure; Obtain the unstructured content of the document.

[0048] Step S2: Document Type Recognition The document type is automatically identified based on OCR text, format features, and keyword patterns, which is used to select subsequent extraction strategies.

[0049] Step S3: Field Extraction and JSON Generation The specific process of extracting fields based on document type and generating semi-structured JSON data is as follows: First, the field extraction module loads the corresponding field definition set and extraction strategy based on the document type output in step S2; then, it takes the aforementioned OCR text, page structure information and document type as input to generate a semi-structured JSON result containing field names, field values, page numbers and optional position information.

[0050] S3.1 Main Process Description Specifically, the field extraction module extracts key business fields for each type of document, such as: Contract documents include: contract number, name of Party A, name of Party B, contract amount, and signing date. Invoice documents include: invoice number, invoice date, buyer, seller, tax amount, and detailed items; Patent documents include: patent number, patent name, inventor, and date of authorization announcement. Certificate documents include: certificate number, certificate holder, issuing authority, and validity period.

[0051] The extracted results are organized into a JSON structure, which includes at least the field name (key), field value (value), and the page number of the field (page). Additional information such as field confidence and location box (bbox) can be added.

[0052] S3.2 Extraction method based on large language model The field extraction module calls the large language model and automatically generates the values ​​of each field based on preset extraction prompts. These prompts include: Document type description; List of fields to be extracted; Key-value structure conventions for outputting JSON; Valid format and validation rules for field values, such as date format, currency format, etc.

[0053] Based on the above prompts, the large language model performs semantic understanding of the OCR text and outputs strictly structured JSON data in accordance with the agreed format, thereby improving the consistency and controllability of field extraction.

[0054] S3.3 Cross-page field merging and conflict resolution For fields such as contract amount, certificate number, and patent number that may appear multiple times in the document or be distributed across multiple pages, the system further performs the following steps: Cross-page fragment merging: Based on the semantic similarity or numerical consistency of fields, candidate fields appearing on multiple pages are merged into a unified field; Confidence-weighted selection: Determine the optimal field value based on model confidence or rule hit rate; Page range record: Record the page range involved in the field in JSON (e.g., "pages":[3,4]).

[0055] S3.4 JSON Output Structure Example The JSON generated by the field extraction module may include the following formats: { "document_type":"contract", "fields":{ "contract_no":{"value":"HT-2024-001","page":1,"confidence":0.98}, "buyer":{"value":"XXX Technology Co., Ltd.","page":1}, "seller":{"value":"XXX Equipment Co., Ltd.","page":2}, "amount":{"value":"¥2,560,000","pages":[2,3],"confidence":0.92} } }

[0056] Step S4: Mapping to a structured two-dimensional table Step S4 is used to map the semi-structured JSON data generated in step S3 into a structured two-dimensional data table so that field-level queries, statistical analysis and cross-document comparison can be performed in subsequent steps.

[0057] S4.1 Main Process Description The JSON data output by the field extraction module includes field names, field values, page numbers, confidence levels, and optional location coordinates. In step S4, the system selects the corresponding target table structure based on the document type, mapping the key-value pairs in the JSON to field records in the structured table.

[0058] For example: Contract documents are mapped to the ContractHeader table. Invoice documents are mapped to the InvoiceHeader table and the InvoiceItems table. Patent documents are mapped to the PatentInfo table; Certificate documents are mapped to the CertificateInfo table; All records are associated with a unique doc_id for subsequent tracing.

[0059] S4.2 Direct mapping from field names to table fields The system reads field names from the JSON, writes them into the corresponding database fields according to the field mapping dictionary, and generates a two-dimensional table record with the following structure: (contract_no→contract_header.contract_no); (buyer→contract_header.buyer_name); (issuer→certificate_info.issuer); (invoice_no→invoice_header.invoice_no); This approach is suitable for document types with stable structured field definitions (such as invoices and certificates).

[0060] S4.3 Template-based dynamic mapping method In addition, the system supports dynamically expanding field structures: 1. Load field templates (schema) based on document type.

[0061] 2. All fields appearing in the JSON are dynamically created or expanded to correspond to the field columns of the structured table according to the template.

[0062] 3. For records with fields not in the template, enter the extended field table (ExtFields) to enhance the adaptability of document types.

[0063] S4.4 Implementation Method 3: One-to-Many Structure Processing (e.g., Invoice Details) For fields in JSON containing array structures (such as invoice details or contract terms), the system executes: Create a separate record for each item in the array; Establish a one-to-many relationship using a primary key field (such as invoice_id); Record the source page number for subsequent tracing and verification.

[0064] Step S5: Write the full text content into the PDF-Content table. Step S5 is used to generate a PDF-Content table that supports full-text search, cross-page semantic analysis, and field tracing. This table serves as a semantic supplement and tracing basis for structured data.

[0065] S5.1 Main Process Description The system organizes the parsed OCR text, Markdown text, page layout information, and image content according to the document page number order, forming a full-text content record and writing it into the PDF-Content table. This table includes at least: Document ID (doc_id); Document type; Page number information; Raw OCR text (raw_text); Markdown format full text (md_text); Image placeholders and table placeholder information; Text position index (optional); S5.2 Markdown Content Generation In one implementation, the system converts the page layout (headings, paragraphs, tables, images) into Markdown representation, including: Table → Markdown table; Image area → ; Heading levels → #, ##, etc.; Lists and items → -, 1, etc.; Markdown format is used to enhance the accuracy of LLM in understanding the entire text.

[0066] S5.3 OCR Text and Layout Position Blending In another implementation, the system retains the in-page position coordinates (e.g., bbox) of the OCR text for: When tracing the origin of a field, locate its specific position in the original PDF. Supports merging multiple candidates for certain fields; Improve LLM's ability to determine context structure.

[0067] S5.4 Minimize the slices of the full text To support cross-page semantic question answering, the system can divide a document into multiple "semantic fragments" and store them in the PDF-Content table: Each segment includes the paragraph text, the page number, and its relationship to the preceding and following paragraphs; Provides support for fragment-level query and vector retrieval; Reduce the size of large model inputs and improve inference efficiency.

[0068] Step S6: Loading DuckDB and SQL Queries Step S6 is used to load all structured tables and PDF-Content full-text tables into the embedded columnar database DuckDB to support subsequent field queries, analysis and processing, and intelligent question answering applications.

[0069] S6.1 Main Process Description The system converts the data tables generated by S4 and S5 into DataFrame or Arrow format, and loads them into a queryable dataset using DuckDB's in-memory table registration function. duckdb.register("contract_header",df_contract); duckdb.register("pdf_content",df_content); Then, an SQL query is executed to perform: Field filtering; Key field validation; Cross-document statistical analysis; Document correlation and comparison (e.g., contract ↔ invoice); Document deduplication, merging, and aggregation; S6.2 Columnar Storage and Vectorized Execution DuckDB uses columnar storage and vectorized execution, offering the following technical advantages: Load only the query fields to reduce I / O pressure; Batch vector execution improves the speed of aggregate queries; It is suitable for large-volume, low-frequency queries and high-throughput statistical tasks in bidding scenarios.

[0070] S6.3 Serverless Deployment DuckDB runs as an embedded library and does not require a separate database process. It can run directly in local bidding tools, desktop, and offline network environments; Structured tables and full-text content tables do not need to be written to disk again after loading (memory computation); It facilitates the execution of temporary queries, local processing, and rapid iteration.

[0071] S6.4SQL+LLM Joint Q&A The system can leverage DuckDB's fast query capabilities to pass SQL results to the LLM for: Automatically generate contract summary; Automatically calculates historical performance; Automatically matches bidding requirements; Automatically verify whether the invoice amount corresponds to the contract; Comprehensive cross-document analysis and Q&A; DuckDB provides accurate data for the structured portion, while LLM provides flexible semantic judgment for the full-text portion. The combination of the two forms an intelligent bidding analysis framework that is "computable and understandable".

[0072] Comparison with existing technologies 1. Upgrade from "simple OCR output" to a multi-layered modeling approach of "OCR + JSON + structured tables + full-text content". Existing technology: OCR provides plain text, which cannot be structured later.

[0073] Template-based methods are poorly suited for complex layouts.

[0074] This invention: Semi-structured JSON is generated by extracting fields; Then map it back to a structured two-dimensional table; Simultaneously stores the full-text Markdown; To form a complete information chain.

[0075] 2. Introduce a two-tiered storage system: structured tables and full-text PDF content tables. Existing technology: The extracted fields were not fully documented, making it impossible to trace the source.

[0076] Full-text search and field search cannot be unified.

[0077] This invention: Structured tables are used for field-level calculations and validations. The PDF-Content table is used for full-text search, cross-page semantics, and source tracing. Resolve the issues of "field breakage" and "inability to cross pages".

[0078] 3. Replace traditional databases with DuckDB embedded columnar database. Existing technology: MySQL / PG requires complex processes such as server-side development, table creation, importing, and permissions. Not suitable for ad-hoc bid analysis tasks; This invention: DuckDB requires no service and no deployment. Supports direct loading of JSON / DataFrame; Columnar and vector execution is suitable for analyzing queries; It greatly enhances the data processing capabilities after document parsing.

[0079] 4. Supports cross-document global Q&A and automatic verification in bidding scenarios. Existing technology: Unable to automatically generate a summary of supporting documents; It is not possible to link contracts, invoices, and certificates; Lack of cross-page semantic understanding ability; This invention: Comparison of executable contract amount with invoice; Automatically generate patent lists and contract summaries; Joint reasoning based on full-text semantics and table fields; Significantly improve the efficiency of bid preparation and review.

[0080] 5. Enable field-level traceability and automated auditing capabilities. Existing technology: The field cannot be traced back to the original PDF. Audit requirements are difficult to meet; This invention: Each field can be located to a specific page number or a specific text fragment; Form a structured chain of evidence that is verifiable, traceable, and auditable.

[0081] The specific implementation process is as follows: 6.1 System Overall Structure like Figure 1 As shown, the structured parsing system for bid response documents of the present invention includes: 1. The document parsing module is used to perform OCR, layout analysis, and image extraction to obtain information such as text fragments, table fragments, and image areas for each page.

[0082] 2. Document classification module, used to automatically determine the document category based on the parsed content, including types such as contracts, invoices, certificates, patents, test reports, and personnel qualification certificates.

[0083] 3. Field extraction module, used to perform field extraction based on document type and generate semi-structured JSON containing field names, field values ​​and page number information.

[0084] 4. The structured data conversion module is used to map JSON data into structured two-dimensional tables for field-level retrieval and correlation analysis.

[0085] 5. The PDF-Content full-text storage module is used to uniformly store information such as full-text Markdown, OCR line text, and page number positions of documents for RAG retrieval, field tracing, and cross-page semantic analysis.

[0086] 6. Embedded columnar database module (DuckDB), used to load structured tables and full-text content tables, and execute analytical queries based on SQL.

[0087] 7. Intelligent Question Answering and Validation Module, used to perform cross-document question answering, field validation, content comparison and automatic summary generation based on SQL results and PDF-Content table content.

[0088] The above modules can be deployed on the same device, or distributed across servers, local clients, or offline environments. The deployment architecture is as follows: Figure 6 As shown.

[0089] 6.2 PDF document parsing process like Figure 2 As shown, the document parsing module first performs the following steps on the input PDF file: 1. OCR Text Extraction: Uses an OCR engine to recognize the text on each page and outputs the recognition confidence score, paragraph structure, and line position coordinates.

[0090] 2. Page Layout Analysis: Identify page elements in the document, such as titles, paragraphs, lists, tables, stamp areas, and image areas.

[0091] 3. Text sequence and hierarchy restoration: Restore the natural reading order of the text, including in-page sorting and cross-page paragraph splicing.

[0092] 4. Identification of table regions and candidate field regions: The table structure is extracted through methods such as layout inference, connected component analysis, and row and column segmentation to support subsequent field alignment.

[0093] The parsing results include single-page text, layout tags, structural blocks, image information, etc., which will be used as input for subsequent document classification and field extraction.

[0094] 6.3 Document Type Recognition Mechanism The document type recognition module performs classification based on OCR text and layout features. Examples include: Fields including "Contract Number", "Party A", and "Party B" are identified as contract type. Including fields such as "invoice code", "tax rate", and "amount" → identified as an invoice type The layout of the text area conforms to the certificate cover format → Recognized as a certificate type Includes fields such as "Grant Announcement Date" and "Patent Number" → Identified as a patent type Long, continuous paragraphs without strongly structured fields are identified as narrative text (such as after-sales commitment letters). Document type identification can be achieved through rules, keyword matching, layout templates, or large language models. This invention does not limit the implementation method of the classifier.

[0095] Document type serves as an important basis for selecting templates and extraction methods in subsequent field extraction steps.

[0096] 6.4 Field Extraction and JSON Generation The field extraction module performs structured field extraction based on document type. The process includes: 1. Use a large language model or rule template to extract key fields such as contract number, amount, invoice date, patent number, certificate number, and validity period; 2. Organize the field name, field value, page number, location coordinates, confidence level, and other information into a JSON data structure; 3. Perform merging, deduplication, and confidence score integration on cross-page or duplicate fields; 4. Output strictly formatted JSON fields as input to the data conversion module.

[0097] Field extraction methods can employ large models, rules, regular expressions, keyword positioning, template positioning, or a combination of multiple techniques; this invention does not impose any limitations on these methods.

[0098] 6.5 Structured Table Construction and Field Mapping like Figure 3 As shown, the structured data conversion module maps JSON data into a predefined structured two-dimensional table. Implementation methods include: 1. Contract Header field mapping: including contract number, names of Party A and Party B, amount, signing date, etc.; 2. InvoiceHeader / InvoiceItems Mapping: Perform one-to-many record expansion on fields containing detailed entries; 3. Certificate Information Table (CertificateInfo) Mapping: Includes fields such as certificate number, issuing authority, and validity period; 4. Patent Information Table (PatentInfo) Mapping: Includes fields such as patent number, inventor, and authorization date; 5. Extended Fields (ExtFields): Stores fields not defined in the standard table structure to enhance extensibility.

[0099] All structured tables include a doc_id field that is associated with the PDF-Content table, ensuring that the field can be traced back to the original text.

[0100] 6.6 PDF Content Table Construction While the structured table is being built, the system writes the OCR text, Markdown content, and page layout into the PDF-Content table. This process includes: 1. Markdown text generation: Generate more readable Markdown text based on heading levels, paragraph structure, table content, and image placement; 2. OCR line text and position records: Store the page number and (optional) position coordinates of each text segment for easy field location and original text tracing; 3. Cross-page concatenation: Perform cross-page concatenation on the split paragraphs to restore semantic continuity; 4. Semantic Fragment Segmentation (Optional): Slices the document into semantic units for use in vector retrieval and context enhancement of LLM.

[0101] PDF-Content tables are an important basis for structured analysis, full-text question answering, and field tracing.

[0102] 6.7 DuckDB Loading and SQL Query Execution The system loads the structured table and the PDF-Content table into the embedded columnar database DuckDB and executes multiple types of queries via SQL.

[0103] The loading steps include: 1. Convert the structured table to DataFrame or Arrow format; 2. Use DuckDB's in-memory table registration capability to directly load data; 3. Perform field queries, conditional filtering, aggregation calculations, sorting, and JOIN operations on the table; 4. The query results can be passed to the intelligent question-and-answer module, or used for functions such as contract amount verification, invoice and contract matching check, and certificate validity verification.

[0104] DuckDB offers columnar storage, vectorized execution, and a zero-deployment mode, enabling this invention to run efficiently in local tools, bidding systems, or offline environments.

[0105] 6.8 Intelligent Question Answering, Validation, and Result Output After the above data loading and SQL query are completed, the intelligent question answering module executes: 1. Field-level Q&A: such as "What is the amount of a certain contract?" 2. Cross-document Q&A: such as "Who is the customer with the highest contract amount in the past three years?" 3. Validation-type reasoning: such as "Does the invoice amount equal the contract amount?" 4. Locating clauses throughout the document: For example, "On which page is the clause regarding liability for breach of contract?" 5. Generative summary output: Automatically generates materials required for bidding, such as "contract summary" and "patent list".

[0106] The system can output the results as JSON, tables, Markdown, or bid template documents for manual review or automatic generation of bid response forms.

[0107] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0108] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0109] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0110] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0111] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0112] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0113] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0114] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0115] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0116] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A method for structured parsing of multiple document types for bid response documents, characterized in that, include: S1: Obtain a collection of various types of PDF supporting documents from the bid response file, perform OCR text recognition and layout analysis on each PDF supporting document, extract page-level text, images, tables and location-related information to obtain unstructured content; S2: Based on the text and format features of the unstructured content, automatically identify the document type of each PDF supporting material; S3: Based on the document type, call the corresponding extraction strategy to extract key business fields and generate semi-structured JSON data. The semi-structured JSON data includes field names, field values, page numbers, and confidence information. Fields that cross pages or appear repeatedly are merged and optimized. S4: Map the semi-structured JSON data to a structured data table, and associate all structured data tables with a unique identifier; S5: Convert the full text of each PDF supporting document into a standardized text format, and store it in the full text content table by combining the original text, page number and position index information in the unstructured content; S6: After converting the structured data table and the full-text content table into a specified data format, load them into an embedded columnar database to form a queryable dataset. Perform statistical analysis and cross-document association comparison based on the queryable dataset, and combine it with a large language model to realize intelligent question answering, data verification and generative summary output.

2. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The document types mentioned in S2 include contracts, invoices, patents, certificates, and narrative documents. The identification method is achieved through keyword matching, layout feature analysis, or large language model inference.

3. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, In S3, the large language model is called to extract fields. The input includes preset prompts containing document type descriptions, a list of fields to be extracted, JSON output format conventions, and field value validation rules to ensure the standardization of JSON data.

4. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The mapping method described in S4 includes at least one of direct mapping, dynamic extended mapping, and one-to-many association mapping, and non-standard fields are stored in the extended data table.

5. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The standardized text format described in S5 is Markdown format, which is achieved by converting the titles, paragraphs, tables, and image areas in the PDF supporting materials into the corresponding Markdown format.

6. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The embedded columnar database is DuckDB, which adopts a serverless deployment mode and supports direct operation on local devices or offline environments.

7. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The semi-structured JSON data generated in S3 also includes the location box information corresponding to the fields. In S6, the field is traced back to its source. By using the location box information and the location index in the full-text content table, the specific location of the field in the original PDF proof material is accurately located.

8. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The cross-document association comparison mentioned in S6 includes contract amount and invoice amount matching verification and certificate validity period verification. The generative summary includes contract summary, patent list and performance proof related materials.

9. The method for structured parsing of multiple document types for bid response documents according to claim 1, characterized in that, The location-related information mentioned in S1 includes the page location coordinates, and the full-text content table mentioned in S5 also stores document type, image placeholders, and table placeholders.

10. A multi-type document structure parsing system for bid response documents, characterized in that, The method for structuring and parsing multi-type documents for bid response documents according to any one of claims 1 to 9, wherein the multi-type document structure parsing system for bid response documents comprises: The document parsing module is used to obtain a collection of various types of PDF supporting materials from the bid response file, perform OCR text recognition and layout analysis, and extract unstructured content; The document classification module is used to automatically identify the document type of PDF supporting materials based on the text and format features of unstructured content; The field extraction module is used to call the corresponding extraction strategy according to the document type, extract key business fields and generate semi-structured JSON data, and merge and optimize cross-page or duplicate fields; The structured data conversion module is used to map semi-structured JSON data into structured data tables, thereby achieving standardized data storage. The PDF-Content full-text storage module is used to convert the full-text content of PDF supporting materials into a standardized text format and store it in a full-text content table, combining the original text, page numbers, and location index information. Embedded columnar database module: Employs an embedded columnar database to load structured data tables and full-text content tables, forming a queryable dataset and providing data query and analysis support; The intelligent question answering and verification module is used to perform statistical analysis and cross-document correlation comparison based on the dataset, and to achieve intelligent question answering, data verification and generative summary output by combining a large language model.

Images